Deploying GGUF Models with AidGenSE

Introduction

GGUF (GPT-Generated Unified Format) is a model file format designed for edge-side inference. It packages model weights, the tokenizer, and metadata into a single file and supports various quantization schemes, making it easy to load and run efficiently on resource-constrained edge devices. Edge deployment of GGUF models refers to converting large models that originally run in the cloud into the GGUF format with quantization compression and deploying them on local devices to provide offline, low-latency inference services. This chapter is based on the AidGenSE inference engine and demonstrates how to deploy a GGUF model HTTP service (compatible with the OpenAI API) on edge devices.

In this case, the GGUF model inference runs on the device side, and the relevant interfaces are called through code to receive user input and return conversation results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: qwen3-4b

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware

2. Ubuntu 22.04 system or AidLux system

Case Deployment

Step 1: Install AidGenSE

sudo aid-pkg update
sudo aid-pkg -i aidgense
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

Step 2: Model Query & Retrieval

• View supported models

# View supported models
aidllm remote-list api | grep llamacpp

#------------------------Example output------------------------
ministral-3-3b-instruct-2512-llamacpp-llamacpp-q4_0-qcs8550    VLM     2026-09-10 19:44:28
granite-4.0-micro-llamacpp-llamacpp-q4_0-qcs8550               LLM     2026-09-10 19:44:24
gemma-4-e2b-it-llamacpp-llamacpp-q4_0-qcs8550                  LLM     2026-09-10 19:44:16
qwen3-vl-4b-instruct-llamacpp-llamacpp-q4_0-qcs8550            VLM     2026-09-10 19:44:12
qwen3-vl-2b-instruct-llamacpp-llamacpp-q4_0-qcs8550            VLM     2026-09-10 19:44:08
qwen3.5-2b-llamacpp-llamacpp-q4_0-qcs8550                      VLM     2026-09-10 19:44:04
qwen3.5-0.8b-llamacpp-llamacpp-q4_0-qcs8550                    VLM     2026-09-10 19:44:00
qwen3-4b-instruct-2507-llamacpp-llamacpp-q4_0-qcs8550          LLM     2026-09-10 19:43:56
qwen3-4b-llamacpp-llamacpp-q4_0-qcs8550                        LLM     2026-09-10 19:43:52
qwen3-1.7b-llamacpp-llamacpp-q4_0-qcs8550                      LLM     2026-09-10 19:43:48
qwen3-0.6b-llamacpp-llamacpp-q4_0-qcs8550                      LLM     2026-09-03 14:59:45

• Download qwen3-4b-llamacpp-llamacpp-q4_0-qcs8550

# Download the model
aidllm pull api qwen3-4b-llamacpp-llamacpp-q4_0-qcs8550

# View downloaded models
aidllm list api

Step 3: Start the HTTP Service

# Start the OpenAI API service for the corresponding model
aidllm start api -m qwen3-4b-llamacpp-llamacpp-q4_0-qcs8550 --device dsp

# Check status
aidllm status api

# Stop service: aidllm stop api

# Restart service: aidllm restart api

💡Note

The default port is 8888.

Step 4: Chat Test

Chat Test via Web UI

# Install UI frontend service
sudo aidllm install ui

# Start UI service
aidllm start ui

# Check UI service status: aidllm status ui

# Stop UI service: aidllm stop ui

After the UI service starts, visit http://ip:51104.

Chat Test via Python

import os
import requests
import json

def stream_chat_completion(messages, model="qwen3-4b-llamacpp-llamacpp-q4_0-qcs8550"):

    url = "http://127.0.0.1:8888/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "stream": True    # Enable streaming
    }

    # Make request with stream=True
    response = requests.post(url, headers=headers, json=payload, stream=True)
    response.raise_for_status()

    # Read line by line and parse SSE format
    for line in response.iter_lines():
        if not line:
            continue
        # print(line)
        line_data = line.decode('utf-8')
        # Each SSE line starts with the "data: " prefix
        if line_data.startswith("data: "):
            data = line_data[len("data: "):]
            # End marker
            if data.strip() == "[DONE]":
                break
            try:
                chunk = json.loads(data)
            except json.JSONDecodeError:
                # Print and skip when parsing fails
                print("Unable to parse JSON:", data)
                continue

            # Extract the model output token
            content = chunk["choices"][0]["delta"].get("content")
            if content:
                print(content, end="", flush=True)

if __name__ == "__main__":
    # Example conversation
    messages = [
        {"role": "system", "content": ""},
        {"role": "user", "content": "Give me a short introduction to large language model."}
    ]
    print("Assistant:", end=" ")
    stream_chat_completion(messages)
    print()  # New line