端侧部署 Qwen3 系列
介绍
Qwen3 是 Qwen 系列最新一代的大语言模型，提供了完整的稠密模型和专家混合（MoE）模型套件。基于大规模训练，Qwen3 在推理、指令跟随、智能体能力以及多语言支持方面实现了突破性进展
本章节将演示如何在边缘设备上完成 Qwen3 系列模型部署、加载与对话流程。提供以下两种方式部署：
- AidGen C++ API
- AidGenSE OpenAI API
在本案例中，大语言模型推理运行在设备端，通过代码调用相关接口接收用户输入并实时返回对话结果。
- 设备：Rhino Pi-X1
- 系统：Ubuntu 22.04
- 模型：Qwen3-1.7B
支持平台
| 平台 | 运行方式 |
|---|---|
| Rhino Pi-X1 | Ubuntu 22.04, AidLux |
准备工作
Rhino Pi-X1 硬件
Ubuntu 22.04 系统或 AidLux 系统
AidGen 案例部署
步骤一：安装 AidGen SDK
bash
# 安装 AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248
# 拷贝测试代码
cd /home/aidlux/aidllm
cp -r /usr/local/share/aidgen/examples/ ./步骤二：下载模型资源
由于 Qwen3-1.7B 目前位于 Model Farm 预览板块中，需要通过
mms命令获取
bash
# 登录
mms login
# 查找模型
mms list qwen3
# 下载模型
mms get -m Qwen3-1.7B -p w4a16 -c qcs8550 -b qnn2.36 -d /home/aidlux/aidllm/
cd /home/aidlux/aidllm/
unzip qnn236_qcs8550_cl4096.zip步骤三：资源文件确认
文件分布如下：
bash
/home/aidlux/aidllm/qnn236_qcs8550_cl4096
├── tokenizer.json
├── qwen3-1.7b_qnn236_qcs8550_c14096_3_of_3.serialized.bin
├── qwen3-1.7b_qnn236_qcs8550_c14096_2_of_3.serialized.bin
├── qwen3-1.7b_qnn236_qcs8550_c14096_1_of_3.serialized.bin
├── qwen3-1.7b-tokenizer.json
├── qwen3-1.7b-htp.json
├── prompt.conf
├── htp_backend_ext_config.json
├── config_linux.json
├── chat-think.txt
├── chat-nothink.txt
├── aidgen_config.json
├── prefix-kvcache步骤四：编译运行
bash
cd /home/aidlux/aidllm/examples
# 编译
mkdir build && cd build
cmake .. && make
cp ./test_t2t ../../qnn236_qcs8550_cl4096
cd ../../qnn236_qcs8550_cl4096
./test_t2t aidgen_config.json "hi"- 模型运行成功后，有如下日志回显
bash
============================================================
Turn 1: Single-turn inference
============================================================
User: hi
Assistant: [API] Generator::run(prompt, callback)
[BOS]<think>
Okay, the user said "hi". I should respond politely. Let me check if there's any specific context I need to consider. Since it's a general greeting, I'll keep it friendly. Maybe mention being here to help. Let me make sure the response is concise and welcoming. Alright, "Hi there! I'm here to help you with anything you need. How can I assist you today?" That sounds good.
</think>
Hi there! I'm here to help you with anything you need. How can I assist you today?<|im_end|>[EOS]
--- Turn 1 Performance (Generator::get_profiler) ---
Init Time (us): 2824242
Prompt Token Count: 30
Time-to-First-Token (us): 72845
Prompt TPS (tok/s): 1757.16
Generated Token Count: 111
Generate Time (us): 2785727
Generate TPS (tok/s): 39.846AidGenSE 案例部署
步骤一：安装 AidGenSE
bash
sudo aid-pkg update
# 确保 aidgense 是最新版本
sudo aid-pkg remove aidgense
sudo aid-pkg -i aidgense
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248步骤二：模型查询 & 获取
bash
# 查看模型
aidllm remote-list api | grep Qwen3
#------------------------可以看到 Qwen3 模型------------------------
Current Soc : 8550
Name Url CreateTime
----- --------- ---------
qwen3-0.6b-qnn2.36-w4a16-qcs8550 aplux/qwen3-0.6b-qnn2.36-w4a16-qcs8550 2026-05-15 10:59:35
qwen3-1.7b-qnn2.36-w4a16-qcs8550 aplux/qwen3-1.7b-qnn2.36-w4a16-qcs8550 2026-05-15 10:57:37
qwen3-4b-basic-quant-qnn2.36-w4a16-qcs8550 aplux/qwen3-4b-basic-quant-qnn2.36-w4a16-qcs8550 2026-05-15 10:59:47
qwen3-4b-instruct-2507-qnn2.36-w4a16-qcs8550 aplux/qwen3-4b-instruct-2507-qnn2.36-w4a16-qcs8550 2026-05-15 10:57:45
...
# 下载 qwen3-1.7B-8550
aidllm pull api aplux/qwen3-1.7b-qnn2.36-w4a16-qcs8550步骤三：启动 HTTP 服务
bash
# 启动对应模型的 openai api 服务
aidllm start api -m qwen3-1.7b-qnn2.36-w4a16-qcs8550
# 查看状态
aidllm status api
# 停止服务: aidllm stop api
# 重启服务: aidllm restart api💡注意
默认端口号是 8888
步骤四：对话测试
使用 Web UI 对话测试
bash
# 安装 UI 前端服务
sudo aidllm install ui
# 启动 UI 服务
aidllm start ui
# 查看 UI 服务状态: aidllm status ui
# 停止 UI 服务: aidllm stop uiUI 服务启动后访问 http://ip:51104
使用 Python 对话测试
python
import os
import requests
import json
def stream_chat_completion(messages, model="qwen3-1.7b-qnn2.36-w4a16-qcs8550"):
url = "http://127.0.0.1:8888/v1/chat/completions"
headers = {
"Content-Type": "application/json"
}
payload = {
"model": model,
"messages": messages,
"stream": True # 打开流式
}
# 发起带 stream=True 的请求
response = requests.post(url, headers=headers, json=payload, stream=True)
response.raise_for_status()
# 逐行读取并解析 SSE 格式
for line in response.iter_lines():
if not line:
continue
# print(line)
line_data = line.decode('utf-8')
# SSE 每一行以 "data: " 前缀开头
if line_data.startswith("data: "):
data = line_data[len("data: "):]
# 结束标志
if data.strip() == "[DONE]":
break
try:
chunk = json.loads(data)
except json.JSONDecodeError:
# 解析出错时打印并跳过
print("无法解析JSON：", data)
continue
# 取出模型输出的 token
content = chunk["choices"][0]["delta"].get("content")
if content:
print(content, end="", flush=True)
if __name__ == "__main__":
# 示例对话
messages = [
{"role": "system", "content": ""},
{"role": "user", "content": "Give me a short introduction to large language model."}
]
print("Assistant:", end=" ")
stream_chat_completion(messages)
print() # 换行