Edge Deployment of the Qwen3-VL Series

Introduction

Qwen3-VL is the vision-language model series of the Qwen family. This generation delivers comprehensive upgrades: stronger text understanding and generation, deeper visual perception and reasoning, longer context lengths, and significantly enhanced spatial understanding, video motion perception, and agentic interaction capabilities. The models come in both Dense and MoE architectures, scaling from edge to cloud, and are available in Instruct versions and Thinking versions with enhanced reasoning, meeting flexible, on-demand deployment requirements.

This chapter demonstrates how to perform deployment, loading, and conversation with the Qwen3-VL series models on edge devices. Two deployment methods are provided:

• AidGen C++ API
• AidGenSE OpenAI API

In this case, the large language model inference runs on the device side, and the relevant interfaces are called through code to receive user input and return conversation results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: Qwen3-VL-4B-Instruct (448x448)

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware

2. Ubuntu 22.04 system or AidLux system

AidGen Case Deployment

Step 1: Install the AidGen SDK

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

# Copy test code
cd /home/aidlux/aidmlm

cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Download Model Resources

Since Qwen3-VL-4B-Instruct (448x448) is currently in the Model Farm preview section, it must be retrieved via the mms command.

# Log in
mms login

# Search for the model
mms list qwen3-vl

# Download the model
mms get -m "Qwen3-VL-4B-Instruct (448x448)" -p w4a16 -c qcs8550 -b qnn2.40 -d /home/aidlux/aidmlm/

cd /home/aidlux/aidmlm/
unzip qnn240_qcs8550_cl2048.zip

Step 3: Confirm Resource Files

The file distribution is as follows:

/home/aidlux/aidmlm/qnn240_qcs8550_cl2048
├── qwen3-v1-4b_qnn240_qcs8550_c12048_3_of_3.serialized.bin.aidem
├── qwen3-v1-4b_qnn240_qcs8550_cl2048_2_of_3.serialized.bin.aidem
├── qwen3-v1-4b_qnn240_qcs8550_cl2048_1_of_3.serialized.bin.aidem
├── qwen3-v1-4b-vit.serialized.bin.aidem
├── position_ids_sin.raw
├── position_ids_cos.raw
├── ingredients.png
├── embedding_fp32.bin
├── aidgen_config.json
├── examples

Step 4: Compile and Run

cd /home/aidlux/aidmlm/examples

# Replace the example files
cp ../qnn240_qcs8550_cl2048/examples/* ./

# Compile
mkdir build && cd build
cmake .. && make

# Copy the compiled example program to the model directory
cp ./test_qwen3vl ../../qnn240_qcs8550_cl2048
cd /home/aidlux/aidmlm/qnn240_qcs8550_cl2048

# Run the example program
./test_qwen3vl qnn240

• After the model runs successfully, the following log output is displayed:

Image-text embedding input
This image shows a colorful and varied collection of fresh vegetables and fruits, along with two slices of raw fish, arranged on a dark background with a clean, layered composition.

The following can be seen in the picture:

1.  **Vegetables**:
   - **Cucumbers**: Several emerald-green cucumbers with natural fine lines on the surface, looking fresh and plump.
   - **Tomatoes**: Both ripe red tomatoes and green unripe ones, with a sharp color contrast showing fruits at different ripening stages.
   - **Carrots**: Several orange-red carrots with thick roots and leafy tops, bright and full of vitality.
   - **Celery/parsley**: Several green celery stalks with lush stems and spread-out leaves.
   - **Onions/garlic**: The picture also includes several onions and garlic, common seasoning vegetables usually paired with spices.
   - **Green peppers/zucchini**: There are also several green zucchini, similar in shape and color to cucumbers but more rounded.
   - **Ginger**: A few pieces of ginger with light-brown skin and firm texture, commonly used for cooking and seasoning.
   - **Broccoli**: A few florets of broccoli also appear, with plump buds in white or light green, looking very fresh.

2.  **Fruits**:
   - **Lemons**: Several yellow lemons with smooth, bright skin, often used for flavoring or juicing.
   - **Tangerines/oranges**: There are also several tangerines with orange-yellow peel and plump flesh.
   - **Pomegranates**: A few pomegranates also appear, with reddish-brown rinds and rich flesh, often used for making wine or jam.
   - **Avocados**: Several avocados appear as well, with dark-green skin and orange flesh, commonly used in salads or desserts.
   - **Apples**: A few apples are also present, with reddish-brown skin and white flesh, often used for making jam or jelly.

3.  **Protein**:
   - **Raw fish (sashimi)**: Two slices of raw fish with tender, pinkish flesh and a little fat on the outside, a common seafood ingredient.

Through its rich colors, varied shapes, and layered composition, the image presents the natural beauty of all kinds of fresh ingredients. These ingredients are not only brightly colored and distinct in shape but also arranged in a pleasing, staggered layout, giving a sense of health, nature, and abundance. The picture not only shows the diversity of the ingredients but also enhances its visual appeal through contrasts in color and shape, conveying the gifts of nature and the freshness and richness of the food.

This image is not just a display of food, but a picture full of vitality and natural beauty, as if placing the viewer in a vibrant kitchen or market.<|im_end|>

AidGenSE Case Deployment

Step 1: Install AidGenSE

sudo aid-pkg update

# Ensure aidgense is the latest version
sudo aid-pkg remove aidgense
sudo aid-pkg -i aidgense
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

Step 2: Model Query & Retrieval

# View models
aidllm remote-list api | grep qwen3-vl

#------------------------ You can see the Qwen3-VL models ------------------------

qwen3-vl-4b-instruct-llamacpp-llamacpp-q4_0-qcs8550            VLM     2026-09-10 19:44:12
qwen3-vl-2b-instruct-llamacpp-llamacpp-q4_0-qcs8550            VLM     2026-09-10 19:44:08
qwen3-vl-4b-instruct-448x448-qnn2.40-w4a16-qcs8550             VLM     2026-08-13 17:55:59
qwen3-vl-8b-instruct-qnn2.48-w4a16-qcs8550                     VLM     2026-08-13 17:55:43
...

# Download the model

aidllm pull api qwen3-vl-4b-instruct-448x448-qnn2.40-w4a16-qcs8550

Step 3: Start the HTTP Service

# Start the OpenAI API service for the corresponding model
aidllm start api -m qwen3-vl-4b-instruct-448x448-qnn2.40-w4a16-qcs8550

# Check status
aidllm status api

# Stop service: aidllm stop api

# Restart service: aidllm restart api

💡Note

The default port is 8888.

Step 4: Chat Test

Chat Test via Web UI

# Install UI frontend service
sudo aidllm install ui

# Start UI service
aidllm start ui

# Check UI service status: aidllm status ui

# Stop UI service: aidllm stop ui

After the UI service starts, visit http://ip:51104.

Chat Test via Python

import os
import sys
import requests
import json
import base64


def stream_chat_completion(messages, model="qwen3-vl-4b-instruct-448x448-qnn2.40-w4a16-qcs8550"):

    url = "http://127.0.0.1:8888/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "stream": True    # Enable streaming
    }

    # Make request with stream=True
    response = requests.post(url, headers=headers, json=payload, stream=True)
    try:
        response.raise_for_status()
    except requests.HTTPError as e:
        print(f"\nHTTP error: {e}", file=sys.stderr)
        if response.text:
            print(response.text, file=sys.stderr)
        raise

    # Read line by line and parse SSE format
    for line in response.iter_lines():
        if not line:
            continue
        # print(line)
        line_data = line.decode('utf-8')
        # Each SSE line starts with the "data: " prefix
        if line_data.startswith("data: "):
            data = line_data[len("data: "):]
            # End marker
            if data.strip() == "[DONE]":
                break
            try:
                chunk = json.loads(data)
            except json.JSONDecodeError:
                # Print and skip when parsing fails
                print("Unable to parse JSON:", data)
                continue

            # Extract the model output token
            content = chunk["choices"][0]["delta"].get("content")
            if content:
                print(content, end="", flush=True)

def image_to_base64(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def image_to_data_url(image_path):
    """Read the image and convert it to a data URL, compatible with the OpenAI-style image_url field"""
    ext = os.path.splitext(image_path)[1].lower()
    mime_map = {
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".webp": "image/webp",
    }
    mime_type = mime_map.get(ext, "image/jpeg")
    b64_str = image_to_base64(image_path)
    return f"data:{mime_type};base64,{b64_str}"


if __name__ == "__main__":
    image_url = image_to_data_url("/path/to/your/image.jpg")
    # Example conversation
    messages = [
        {
            "role": "system", 
            "content": "You are a helpful assistant."
        },
        {
            "role": "user", 
            "content":[
                        {
                            "type": "text",
                            "text": "这张图片描述了什么内容?"
                        },
                        {
                            "type": "image_url",
                            "image_url": 
                            {
                                "url": image_url
                            }
                        }
                    ]   
        },
    ]
    print("Assistant:", end=" ")
    stream_chat_completion(messages)
    print()  # New line