Deploying VLM with AidGen

Introduction

Deploying a vision language model (VLM) on edge devices refers to compressing, quantizing, and deploying large models that originally run in the cloud onto local devices, enabling offline, low-latency natural language understanding and generation. This chapter is based on the AidGen inference engine and demonstrates how to perform deployment, loading, and conversation of multimodal large models on edge devices.

In this case, the multimodal large model inference runs on the device side, and the relevant interfaces are called through C++ code to receive user input and return conversation results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: Qwen2.5-VL-3B (392x392)

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware
2. Ubuntu 22.04 system or AidLux system

Case Deployment

Step 1: Install the AidGen SDK

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

# Copy test code
mkdir /home/aidlux/aidmlm
cd /home/aidlux/aidmlm
cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Model Acquisition

Since Qwen2.5-VL-3B (392x392) is currently in the Model Farm preview section, it must be retrieved via the mms command.

# Log in
mms login

# Search for the model
mms list Qwen2.5-VL-3B

# Download the model
mms get -m 'Qwen2.5-VL-3B-Instruct (392x392)' -p w4a16 -c qcs8550 -b qnn2.36 -d /home/aidlux/aidmlm/

# Extract the model
cd /home/aidlux/aidmlm/
unzip qnn236_qcs8550_cl2048.zip

Step 3: Confirm Resource Files

The file distribution is as follows:

/home/aidlux/aidmlm/qnn236_qcs8550_cl2048
├── window_attention_mask.raw
├── veg.serialized.bin.aidem
├── qwen2p5-v1-3b_qnn236_qcs8550_c12048_1_of_1.serialized.bin.aidem
├── position_ids_sin.raw
├── position_ids_cos.raw
├── full_attention_mask.raw
├── embedding_weights_151936x2048.raw
├── demo.jpg
├── aidgen_config.json
├── examples

Step 4: Compile and Run

cd /home/aidlux/aidmlm/examples

# Replace example files
cp ../qnn236_qcs8550_cl2048/examples/* ./

# Compile
mkdir build && cd build
cmake .. && make

# Place the compiled example files into the model directory
cp ./test_qwen25vl ../../qnn236_qcs8550_cl2048
cd /home/aidlux/aidmlm/qnn236_qcs8550_cl2048

# Run the example program
./test_qwen25vl qnn240

• After the model runs successfully, the following log output is displayed:

#################################################
Image-text Embedding Input
This image shows a woman and her dog on a beach. The background is a vast ocean, with waves gently lapping at the shore. The sky looks clear and sunny, adding a warm and peaceful atmosphere to the scene.

The woman sits on the sand, dressed in casual clothes, seemingly interacting with her dog. Her dog is wearing a blue collar and looks friendly and loyal. They appear to be sharing a special moment, perhaps giving each other high fives or encouraging each other.

The overall atmosphere is relaxed and pleasant, full of intimacy between humans and pets. The beach is a perfect place for such activities, as it provides a natural environment for people to relax, enjoy nature, and spend quality time with their pets.<|im_end|>
#################################################