Laya Deployment

Introduction

Laya is a multilingual, non-autoregressive System 1 decision model. You only need to input state information (text, emails, tickets, or JSON) along with a typed question; the model returns mathematically calibrated typed answers with a probability distribution in a single forward pass (taking about 33 milliseconds), natively supporting more than 100 languages.

The model is trained with Strictly Proper Scoring Rules through reinforcement learning (RLCD), so outputting a true and accurate probability is the only way for the model to obtain the maximum reward. The model never generates free text, completely eliminating the overhead of text parsing and the possibility of hallucinations.

This chapter demonstrates the deployment, loading, and recognition workflow of laya on edge devices. The following deployment method is provided:

• AidLite Python API

In this case, model inference runs on the device-side NPU computing unit. The code calls relevant interfaces to receive user input and return results.

• Device: Rhino Pi-X1
• OS: Ubuntu 22.04
• Source Model: laya
• Quantized Precision: INT8 quantization
• Model Farm Reference: laya

Supported Platforms

Platform	Runtime Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Prerequisites

1. Rhino Pi-X1 Hardware
2. Ubuntu 22.04 OS or AidLux OS

Download laya-FP16 Model Resources

mms list laya

#------------------------ Available laya model ------------------------
Model  Precision  Chipset  SOC               Backend
-----  ---------  -------  ----------------  -------
laya   FP16       qcs8550  Qualcomm QCS8550  QNN2.48


# Download laya FP16
mms get -m laya -p fp16 -c qcs8550 -b qnn2.48 -d /home/aidlux/laya
cd /home/aidlux/laya
# Extract
unzip laya_qcs8550_fp16.zip

💡Note

Developers can also search for and download via the Model Farm web page.

AidLite SDK Installation

Developers can also refer to the README.md in the model folder for SDK installation.

• Ensure the QNN backend version is ≥ 2.48.
• Ensure the versions of aidlite-sdk and aidlite-qnnxxx are 2.5.x.

# AidLite & QNN version check
dpkg -l | grep aidlite
#------------------------ Example output ------------------------
ii  aidlite-qnn248       2.5.0.284         arm64        aidlux aidlite qnn248 backend plugin
ii  aidlite-sdk          2.5.0.284         arm64        aidlux inference module sdk

Updating QNN & AidLite versions:

# Install AidLite SDK
sudo aid-pkg update
sudo aid-pkg install aidlite-sdk
sudo aid-pkg install aidlite-qnn48

# aidlite sdk python check
python3 -c "import aidlite; print(aidlite.get_py_library_version())"

AidLite Python API Deployment

Run Python API example

cd /home/aidlux/laya/

# Install dependencies
pip3 install -r code/python/requirements.txt

python3 code/python/run_test.py \
  --encoder_model ./models/QCS8550/FP16/Encoder.serialized.bin.aidem \
  --head_model ./models/QCS8550/FP16/Head_k16.serialized.bin.aidem \
  --question ./code/python/assets/question.json \
  --input ./code/python/assets/sample.jsonl --index 0 \
  --invoke_nums 10

You can view the model inference latency (in ms) and the inference result in the terminal:

====================================
Encoder.serialized.bin.aidem
QNN invoke 10 times:
 --mean_invoke_time is 81.78284168243408
 --max_invoke_time is 82.71455764770508
 --min_invoke_time is 81.55393600463867
 --var_invoketime is 0.10294212927419721
====================================
====================================
Head_k16.serialized.bin.aidem
QNN invoke 10 times:
 --mean_invoke_time is 4.773068428039551
 --max_invoke_time is 6.082057952880859
 --min_invoke_time is 4.555463790893555
 --var_invoketime is 0.1934741140985352
====================================
===================
Text: Fears for T N pension after talks Unions representing workers at Turner   Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul.
Sequence length: 98  markers: [14, 23, 33, 43]
0 logit= -3.3281 prob=0.017267  World: international news, politics, conflict
1 logit= -4.0039 prob=0.011762  Sports: games, athletes, matches, tournaments
2 logit=  3.7031 prob=0.937741  Business: companies, markets, economy, finance
3 logit= -2.1758 prob=0.033231  Sci/Tech: science, technology, computing, research
Answer: 2 (Business: companies, markets, economy, finance)
Confidence: 0.786663  act_probability: 1.000000
Result saved to ./code/python/result.json