Deploy an LLM HTTP Server with AidGenSE

Introduction

Edge deployment of a large language model (LLM) refers to compressing, quantizing, and deploying a large model that originally runs in the cloud onto a local device, enabling offline, low-latency natural language understanding and generation. This chapter is based on the AidGenSE inference engine and demonstrates how to complete the deployment of a large language model HTTP service (compatible with the OpenAI API) on an edge device.

In this case, large language model inference runs on the device, and the relevant interfaces are called through the HTTP API to receive user input and return conversation results in real time.

• Device: IQ8275
• System: Ubuntu 24.04
• Model: Qwen2.5-0.5B-Instruct

Supported Platforms

Platform	Execution Method
IQ8275	Ubuntu 24.04

Prerequisites

1. IQ8275 hardware

2. Ubuntu 24.04 system

System Dependency Configuration

Configure the AidLux Package Source

# Download the correct public key
sudo wget -O- https://archive.aidlux.com/ubuntu24/public.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/private-aidlux.gpg > /dev/null

# Edit the source list file
sudo vim /etc/apt/sources.list.d/private-aidlux.list

# Add the repository provided by AidLux to the source file
deb [arch=arm64 signed-by=/etc/apt/trusted.gpg.d/private-aidlux.gpg] https://archive.aidlux.com/ubuntu24 noble main

# Update the package cache
sudo apt update

After the update is complete, you can use the following command to list the SDK dependencies officially provided by AidLux:

sudo apt list | grep aid | grep unknown

# Install software
# Must be installed first because they are not included in the system by default
sudo apt install python3 python3-pip libopencv-dev python3-opencv  net-tools
# Must be installed before aidlite
sudo apt install aidlux-aistack-base aidrtcm

# Install aidlite and its dependencies
sudo apt install aid-lms aidlms-sdk aidlite-sdk cmake
sudo apt-get install libfmt-dev nlohmann-json3-dev
sudo apt install aidlite-*

# Enable DSP support
sudo apt-get install qcom-fastrpc1
sudo apt-get install qcom-fastrpc-dev

# Install aidgen-sdk
sudo apt install aidgen-sdk
sudo apt install aidgen-qnn*

# Install the mms service
sudo apt install aid-mms

# Enable GPU support
sudo apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa
sudo apt install qcom-adreno-cl1
sudo ln -s /usr/lib/aarch64-linux-gnu/libOpenCL.so.1 /usr/lib/aarch64-linux-gnu/libOpenCL.so

After the installation is complete, check that the aidlite and aidgen directories have been added under /usr/local/share.

Device Authorization

Get the Device SN

cat  /sys/devices/soc0/serial_number

Get the License File

Provide the SN to APLUX technical support so that they can generate the device-specific license file. Place the generated file under /etc/opt/aidlux/license/AidLuxLics.

Activate the License

sudo /opt/aidlux/cpf/aid-lms/manager.sh restart

Case Deployment

Step 1: Install AidGenSE

# Configure the virtual runtime environment
sudo apt install -y python3-pip python3-venv > /dev/null 2>&1
sudo python3 -m venv /opt/aidlux/aid-python3

# Create the aid-python3 command
echo '#!/bin/bash
exec /opt/aidlux/aid-python3/bin/python3 "$@"' | sudo tee /usr/bin/aid-python3 > /dev/null
sudo chmod +x /usr/bin/aid-python3

# Create the aid-pip3 command
echo '#!/bin/bash
exec /opt/aidlux/aid-python3/bin/python3 -m pip "$@"' | sudo tee /usr/bin/aid-pip3 > /dev/null
sudo chmod +x /usr/bin/aid-pip3

# Install aidgense
sudo apt install aidgense
# aidllm does not yet support 8275; using 8550 as a substitute
sudo aidllm system --sys linux --soc 8550
sudo apt install aid-pkg
sudo aidllm install ui

Step 2: Query & Retrieve the Model

• View the supported models

# View the supported models
aidllm remote-list api

#------------------------ Example output ------------------------

Current Soc : 8550

Name                                 Url                                          CreateTime
-----                                ---------                                    ---------
qwen2.5-0.5B-Instruct-8550           aplux/qwen2.5-0.5B-Instruct-8550             2025-03-05 14:52:23
qwen2.5-3B-Instruct-8550             aplux/qwen2.5-3B-Instruct-8550               2025-03-05 14:52:37
...

• Download Qwen2.5-0.5B-Instruct

# Download the model
aidllm pull api aplux/qwen2.5-0.5b-instruct-qnn2.29-w4a16-qcs8550

# View the downloaded models
aidllm list api

Step 3: Start the HTTP Service

# Start the OpenAI API service for the corresponding model
aidllm start api -m qwen2.5-0.5b-instruct-qnn2.29-w4a16-qcs8550

# View the status
aidllm status api

# Stop the service: aidllm stop api

# Restart the service: aidllm restart api

💡Note

The default port number is 8888.

Step 4: Conversation Testing

Conversation Test Using the Web UI

# Install the UI frontend service
sudo aidllm install ui

# Start the UI service
aidllm start ui

# View the UI service status: aidllm status ui

# Stop the UI service: aidllm stop ui

After the UI service starts, visit http://ip:51104

Conversation Test Using Python

import os
import requests
import json

def stream_chat_completion(messages, model="qwen2.5-0.5b-instruct-qnn2.29-w4a16-qcs8550"):

    url = "http://127.0.0.1:8888/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "stream": True    # Enable streaming
    }

    # Send a request with stream=True
    response = requests.post(url, headers=headers, json=payload, stream=True)
    response.raise_for_status()

    # Read line by line and parse the SSE format
    for line in response.iter_lines():
        if not line:
            continue
        # print(line)
        line_data = line.decode('utf-8')
        # Each SSE line starts with a "data: " prefix
        if line_data.startswith("data: "):
            data = line_data[len("data: "):]
            # End marker
            if data.strip() == "[DONE]":
                break
            try:
                chunk = json.loads(data)
            except json.JSONDecodeError:
                # Print and skip if parsing fails
                print("Unable to parse JSON:", data)
                continue

            # Get the token output by the model
            content = chunk["choices"][0]["delta"].get("content")
            if content:
                print(content, end="", flush=True)

if __name__ == "__main__":
    # Example conversation
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello."}
    ]
    print("Assistant:", end=" ")
    stream_chat_completion(messages)
    print()  # Newline