Deploying LLM with AidGen

Introduction

Deploying a Large Language Model (LLM) on edge devices refers to compressing, quantizing, and deploying large models that originally run in the cloud onto local devices, enabling offline, low-latency natural language understanding and generation. This chapter is based on the AidGen inference engine and demonstrates how to perform LLM deployment, loading, and conversation on edge devices.

In this case, the large language model inference runs on the device side, and the relevant interfaces are called through C++ code to receive user input and return conversation results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: Qwen2.5-0.5B-Instruct

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware
2. Ubuntu 22.04 system or AidLux system
3. Prepare the model files

Visit Model Farm: Qwen2.5-0.5B-Instruct to download the model resource files.

💡Note

Select the QCS8550 chip.

Case Deployment

Step 1: Install the AidGen SDK

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

# Copy test code
cd /home/aidlux/aidllm

cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Obtain Model Resources

# Log in
mms login

# Search for the model
mms list Qwen2.5-0.5B-Instruct

# Download the model
mms get -m Qwen2.5-0.5B-Instruct -p w4a16 -c qcs8550 -b qnn2.29 -d /home/aidlux/aidllm/

cd /home/aidlux/aidllm/
unzip qnn229_qcs8550_cl4096.zip

Step 3: Confirm Resource Files

The file distribution is as follows:

/home/aidlux/aidllm/qnn229_qcs8550_cl4096
├── tokenizer_config.json
├── tokenizer.json
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_2_of_2.serialized.bin
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_1_of_2.serialized.bin
├── qwen2.5-0.5b-instruct-tokenizer.json
├── qwen2.5-0.5b-instruct-htp.json
├── prompt.conf
├── metadata.json
├── htp_backend_ext_config.json
├── genie_config.json
├── config_linux.json
├── chat.txt
├── aidgen_config.json
├── aidgen_chat_template.txt

Step 4: Set the Conversation Template

💡Note

For the conversation template, refer to the chat.txt file in the model resource package.

Modify the test_aidgen_t2t.cpp file according to the large model's template:

    // ========================================================================
    // 5. Build the prompt template (Qwen2 format)
    // ========================================================================
    std::string system_prompt =
        "<|im_start|>system\n"
        "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n";

    auto make_user_turn = [](const std::string& text) -> std::string {
        return "<|im_start|>user\n" + text + "<|im_end|>\n<|im_start|>assistant\n";
    };

Step 5: Compile and Run

cd /home/aidlux/aidllm/examples

# Compile
mkdir build && cd build
cmake .. && make

cp test_t2t ../../qnn229_qcs8550_cl4096

cd /home/aidlux/aidllm/qnn229_qcs8550_cl4096
./test_t2t aidgen_config.json 'hi'

• After the model runs successfully, the following log output is displayed:

============================================================
  Turn 1: Single-turn inference
============================================================
User: hi
Assistant: [API] Generator::run(prompt, callback)
[BOS]Hello! How can I assist you today? Please feel free to ask me any questions.<|im_end|>[EOS]

--- Turn 1 Performance (Generator::get_profiler) ---
  Init Time (us):           573878
  Prompt Token Count:       30
  Time-to-First-Token (us): 32204
  Prompt TPS (tok/s):       1987.33
  Generated Token Count:    18
  Generate Time (us):       187135
  Generate TPS (tok/s):     96.1872