Deploying VLM with AidGen

Introduction

Deploying a Vision Language Model (VLM) on edge devices refers to compressing, quantizing, and deploying large models that originally run in the cloud onto local devices, enabling offline, low-latency natural language understanding and generation. This chapter is based on the AidGen inference engine and demonstrates how to complete the deployment, loading, and conversation process of multimodal large models on edge devices.

In this case, the multimodal large model inference runs on the device side, and the relevant interfaces are called through C++ code to receive user input and return conversation results in real time.

• Device: IQ9075
• System: Ubuntu 24.04
• Model: Qwen2.5-VL-3B (504x336)

Supported Platforms

Platform	Running Method
IQ9075	Ubuntu 24.04

Prerequisites

1. IQ9075 hardware

2. Ubuntu 24.04 system

System Dependency Configuration

Configure the AidLux Repository

# Download the correct public key
sudo wget -O- https://archive.aidlux.com/ubuntu24/public.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/private-aidlux.gpg > /dev/null

# Edit the source file
sudo vim /etc/apt/sources.list.d/private-aidlux.list

# Add the private key provided by AidLux to the source file
deb [arch=arm64 signed-by=/etc/apt/trusted.gpg.d/private-aidlux.gpg] https://archive.aidlux.com/ubuntu24 noble main

# Update the cache
sudo apt update

After the update, you can obtain the official AidLux SDK dependencies with the following command:

sudo apt list | grep aid | grep unknown

# Install software
# Must be installed first; not included with the system
sudo apt install python3 python3-pip libopencv-dev python3-opencv  net-tools
# Required before installing aidlite
sudo apt install aidlux-aistack-base aidrtcm

# Install aidlite and its dependencies
sudo apt install aid-lms aidlms-sdk aidlite-sdk cmake
sudo apt-get install libfmt-dev nlohmann-json3-dev
sudo apt install aidlite-*

# Enable DSP support
sudo apt-get install qcom-fastrpc1
sudo apt-get install qcom-fastrpc-dev

# Install the aidgen SDK
sudo apt install aidgen-sdk
sudo apt install aidgen-qnn*

# Install the mms service
sudo apt install aid-mms

# Enable GPU support
sudo apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa
sudo apt install qcom-adreno-cl1
sudo ln -s /usr/lib/aarch64-linux-gnu/libOpenCL.so.1 /usr/lib/aarch64-linux-gnu/libOpenCL.so

After installation, check that the aidlite and aidgen directories have been added under /usr/local/share.

Device Authorization

Obtain the Device SN Code

cat  /sys/devices/soc0/serial_number

Obtain the License File

Provide the SN code to the Aplux technical staff to generate a device-specific License file, and place it under /etc/opt/aidlux/license/AidLuxLics.

Activate Authorization

sudo /opt/aidlux/cpf/aid-lms/manager.sh restart

Case Deployment

Step 1: Copy the AidGen SDK Code Examples

# Copy the test code
cd /home/ubuntu/aidmlm
cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Obtain the Model

Since Qwen2.5-VL-3B (504x336) is currently in the Model Farm preview section, it must be retrieved via the mms command.

Using mms requires logging in with a Model Farm account. Please visit Model Farm account registration

# Log in
mms login

# Search for the model
mms list Qwen2.5-VL-3B

# Download the model
mms get -m 'Qwen2.5-VL-3B-Instruct (504x336) ' -p w4a16 -c qcs9075 -b qnn2.40 -d /home/ubuntu/aidmlm/

cd /home/ubuntu/aidmlm/
unzip qnn236_qcs9075_cl2048.zip

Step 3: Confirm the Model Directory

The file distribution is as follows:

/home/ubuntu/aidmlm/qnn236_qcs9075_cl2048
├── window_attention_mask.raw
├── veg.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn236-qcs9075-cl2048_3_of_3.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn236-qcs9075-cl2048 2_of_3.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn236-qcs9075-cl2048_1_of_3.serialized.bin.aidem
├── position_ids_sin.raw
├── position_ids_cos.raw
├── full_attention_mask.raw
├── embedding_weights_151936x2048.raw
├── demo512x342.jpeg
├── aidgen_config.json
├── examples

Step 4: Compile and Run

cd /home/ubuntu/aidmlm/

# Replace the files in the aidgen-sdk example
cp qnn236_qcs9075_cl2048/examples/* ./examples/
cd ./examples

mkdir build && cd build
cmake .. && make

# Copy the compiled example program test_qwen25vl to the model directory
cp test_qwen25vl ../../qnn236_qcs9075_cl2048
cd /home/ubuntu/aidmlm/qnn236_qcs9075_cl2048

./test_qwen25vl qnn240

• After the model runs successfully, the following log output is displayed:

#################################################
Image-Text Embedding Input
This image shows a warm scene of a woman and her dog playing on the beach. The woman is sitting on the sand, facing the dog, seemingly interacting or playing with the dog. Her dog is wearing a blue collar and appears to be a Golden Retriever. The background is an open sea and a distant sky, with bright sunshine, giving a warm and peaceful feeling. The whole scene is full of joy and companionship between people and pets.<|im_end|>
#################################################