Edge Deployment of HY-MT2-1.8B

Introduction

HY-MT2 is a series of "fast-thinking" multilingual translation models designed for complex real-world scenarios. The series comes in three model sizes: 1.8B, 7B, and 30B-A3B (MoE), all supporting translation between 33 languages and effectively following translation instructions in multiple languages.

For edge deployment, with AngelSlim 1.25-bit extreme quantization, the storage requirement of the 1.8B model is reduced to only 440 MB, while inference speed is improved by 1.5x.

Multi-dimensional evaluations show that HY-MT2 delivers outstanding performance across general-domain, real business scenario, domain-specific, and instruction-following translation tasks. In fast-thinking mode, its 7B and 30B-A3B models surpass open-source models such as DeepSeek-V4-Pro and Kimi K2.6, while the lightweight 1.8B model also outperforms mainstream commercial APIs such as those from Microsoft and Doubao in overall performance.

This chapter demonstrates how to perform HY-MT2-1.8B deployment, loading, and translation on edge devices. Two deployment methods are provided:

• AidGen C++ API
• AidGenSE OpenAI API

In this case, the large language model inference runs on the device side, and the relevant interfaces are called through code to receive user input and return conversation results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: HY-MT2-1.8B

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware
2. Ubuntu 22.04 system or AidLux system

AidGen Case Deployment

Step 1: Install the AidGen SDK

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn240
sudo aid-pkg -i aidgen-qnn248

# Copy test code
cd /home/aidlux/aidllm

cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Download Model Resources

Since HY-MT2-1.8B is currently in the Model Farm Pro section, it must be retrieved via the mms command.

# Log in
mms login

# Search for the model
mms list HY

# Download the model
mms get -m HY-MT2-1.8B -p w4a16 -c qcs8550 -b qnn2.48 -d /home/aidlux/aidllm/

cd /home/aidlux/aidllm/
unzip qnn236_qcs8550_cl4096.zip

Step 3: Confirm Resource Files

The file distribution is as follows:

/home/aidlux/aidllm/qnn236_qcs8550_cl4096
├── hy-mt2-1.8b_qnn236_qcs8550_cl4096_3_of_3.serialized.bin.aidem
├── hy-mt2-1.8b_qnn236_qcs8550_cl4096_2_of_3.serialized.bin.aidem
├── hy-mt2-1.8b_qnn236_qcs8550_cl4096_1_of_3.serialized.bin.aidem
├── chat.txt
├── aidgen_config.json
├── prefix-kvcache

Step 4: Set the Conversation Template

💡Note

For the conversation template, refer to the chat.txt file in the model resource package.

Modify the test_aidgen_t2t.cpp file according to the large model's template:

std::string system_prompt = 
        "<｜hy_begin▁of▁sentence｜><｜hy_User｜>Translate the following text into Chinese.\n"
        "Only output the translated result without any additional explanation:\n";
auto make_user_turn = [](const std::string& text) -> std::string {
    return "<|im_start|>user\n" + text + "<|im_end|>\n" + "<｜hy_Assistant｜>";
};

Step 5: Compile and Run

cd /home/aidlux/aidllm/examples

# Compile
mkdir build && cd build
cmake .. && make

cp test_t2t ../../qnn236_qcs8550_cl4096

cd /home/aidlux/aidllm/qnn236_qcs8550_cl4096
./test_t2t aidgen_config.json "Success is not final, failure is not fatal: it is the courage to continue that counts. Believe in yourself and all that you are, for greatness lives within your soul."

• After the model runs successfully, the following log output is displayed:

============================================================
  Turn 1: Single-turn inference
============================================================
User: Success is not final, failure is not fatal: it is the courage to continue that counts. Believe in yourself and all that you are, for greatness lives within your soul.
Assistant: [API] Generator::run(prompt, callback)
[BOS]Success is not final, and failure is not fatal: what matters is the courage to keep going. Believe in yourself and all that you are, for greatness lives within your soul.<｜hy_place▁holder▁no▁2｜>[EOS]

--- Turn 1 Performance (Generator::get_profiler) ---
  Init Time (us):           1982206
  Prompt Token Count:       65
  Time-to-First-Token (us): 53478
  Prompt TPS (tok/s):       2393.51
  Generated Token Count:    27
  Generate Time (us):       708170
  Generate TPS (tok/s):     38.1264

============================================================

AidGenSE Case Deployment

Step 1: Install AidGenSE

sudo aid-pkg update

# Ensure aidgense is the latest version
sudo aid-pkg remove aidgense
sudo aid-pkg -i aidgense
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn*

Step 2: Model Query & Retrieval

# View models
aidllm remote-list api | grep hy

#------------------------ You can see the HY-MT2-1.8B model ------------------------

hy-mt2-1.8b-qnn2.48-w4a16-qcs8550                              LLM     2026-08-13 17:46:43
hy-mt1.5-1.8b-qnn2.36-w4a16-qcs8550                            LLM     2026-08-13 17:46:43
...


# Download HY-MT2-1.8B

aidllm pull api hy-mt2-1.8b-qnn2.48-w4a16-qcs8550

Step 3: Start the HTTP Service

# Start the OpenAI API service for the corresponding model
aidllm start api -m hy-mt2-1.8b-qnn2.48-w4a16-qcs8550 --qnn_ver 248

# Check status
aidllm status api

# Stop service: aidllm stop api

# Restart service: aidllm restart api

💡Note

The default port is 8888.

Step 4: Chat Test

Chat Test via Web UI

# Install UI frontend service
sudo aidllm install ui

# Start UI service
aidllm start ui

# Check UI service status: aidllm status ui

# Stop UI service: aidllm stop ui

After the UI service starts, visit http://ip:51104.

Enter the following conversation template to perform translation:

Translate the following segment into Chinese, without additional explanation.

Success is not final, failure is not fatal: it is the courage to continue that counts. Believe in yourself and all that you are, for greatness lives within your soul.

Chat Test via Python

import os
import requests
import json

def stream_chat_completion(messages, model="hy-mt2-1.8b-qnn2.48-w4a16-qcs8550"):

    url = "http://127.0.0.1:8888/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "stream": True    # Enable streaming
    }

    # Make request with stream=True
    response = requests.post(url, headers=headers, json=payload, stream=True)
    response.raise_for_status()

    # Read line by line and parse SSE format
    for line in response.iter_lines():
        if not line:
            continue
        # print(line)
        line_data = line.decode('utf-8')
        # Each SSE line starts with the "data: " prefix
        if line_data.startswith("data: "):
            data = line_data[len("data: "):]
            # End marker
            if data.strip() == "[DONE]":
                break
            try:
                chunk = json.loads(data)
            except json.JSONDecodeError:
                # Print and skip when parsing fails
                print("Unable to parse JSON:", data)
                continue

            # Extract the model output token
            content = chunk["choices"][0]["delta"].get("content")
            if content:
                print(content, end="", flush=True)

if __name__ == "__main__":
    # Example conversation
    messages = [
        {"role": "system", "content": ""},
        {"role": "user", "content": "Translate the following segment into Chinese, without additional explanation.\n\nSuccess is not final, failure is not fatal: it is the courage to continue that counts. Believe in yourself and all that you are, for greatness lives within your soul."}
    ]
    print("Assistant:", end=" ")
    stream_chat_completion(messages)
    print()  # New line