AidLite C++ API Reference

💡Note: When developing with Aidlite-SDK C++, you need to be aware of the following:

• Include the header file during compilation, located at: /usr/local/include/aidlux/aidlite/aidlite.hpp
• Link the library file during linking, located at: /usr/local/lib/libaidlite.so

Model Data Type enum DataType

For AidliteSDK, it handles different models from different frameworks, and each model has different input data types and output data types. When setting the input and output data types of the model in the usage workflow mentioned above, this data type enumeration is needed.

Member Variable	Type	Value	Description
TYPE_DEFAULT	uint8_t	0	Invalid DataType
TYPE_UINT8	uint8_t	1	Unsigned byte data
TYPE_INT8	uint8_t	2	Byte data
TYPE_UINT32	uint8_t	3	Uint32 data
TYPE_FLOAT32	uint8_t	4	Float32 data
TYPE_INT32	uint8_t	5	Int32 data
TYPE_INT64	uint8_t	6	Int64 data
TYPE_UINT64	uint8_t	7	Uint64 data
TYPE_INT16	uint8_t	8	Int16 data
TYPE_UINT16	uint8_t	9	Uint16 data
TYPE_FLOAT16	uint8_t	10	Float16 data
TYPE_BOOL	uint8_t	11	Bool data

Inference Implementation Type enum ImplementType

Member Variable	Type	Value	Description
TYPE_DEFAULT	uint8_t	0	Invalid ImplementType
TYPE_MMKV	uint8_t	1	Implementation via MMKV (no current implementation, may be removed later)
TYPE_REMOTE	uint8_t	2	Backend implementation via IPC
TYPE_FAST	uint8_t	2	Compatibility alias of TYPE_REMOTE; deprecated and not recommended for new code
TYPE_LOCAL	uint8_t	3	Backend implementation via local calls

💡Note

Special Notes:

The difference between TYPE_LOCAL and TYPE_REMOTE is that, for the Aplux integrated system, the former completes inference locally in the Linux environment without data transfer, while the latter exchanges data with the Android environment through IPC (inter-process communication), and the actual inference process is completed in Android, so data transfer is involved. Therefore, TYPE_LOCAL is the preferred mode whenever it can satisfy the inference workflow.

REMOTE compatibility and recovery

This section applies to clients with the REMOTE compatibility fix. A source merge does not update the package on a device.

TYPE_REMOTE uses inter-process communication (IPC) to call the Android backend. Check both the client and the Android service. A C++ enum value does not establish support in an old service.

• Tensor information: Use information from the service first. An old QNN service can return no tensor information. In that case, use explicit single-graph FP32 shapes for the ordinary route. Set both inputs and outputs with Model::set_model_properties(). The client does not infer native dtypes, names, or quantization parameters.
• Native data: is_native=true requires actual tensor information. The legacy protocol client supports native FP16, UINT8, and FP32 transfer. The Android backend must also support the selected type. An unsupported type causes an error. Do not pass an INT8 buffer as FP32.
• Timeout recovery: A timeout does not prove that the remote operation did not execute. Stop using the current object. Release it. Create a new object and load the model again. Set all inputs again. Do not retry on the old object. A late reply can still arrive.
• Output lifetime: A C++ output pointer is a shared-memory view. Copy the data if you must keep it. Do not use the old pointer after the next model operation or object release.
• Thread use: Individual calls are serialized. Use an external lock for the complete input, invoke, and output sequence. Use separate objects for independent tasks.
• Protocol capacity: A legacy control message can contain at most 1008 payload bytes. An oversized message causes an error. Shorten the request or use a matching service that supports larger messages.

The following behavior requires the Android service update for model slot management.

• Model capacity: If the service returns All remote model slots are in use, release unused models. Then create a new object. Running models retain their slots.
• Client disconnect: Before the first observed heartbeat, the idle grace period is 15 seconds. Later, it is 2 seconds. A running operation finishes before its worker returns the slot. A timeout does not cancel NPU work immediately.
• Declared shapes: Each input and output keeps its own declared shape. Different tensors can have different ranks. An empty shape declares a scalar. Check that the selected backend supports the shape.
• Thread count: For TFLite, set a positive Config::number_of_threads to request a thread count. The updated service forwards this value. The default -1 retains the SDK default.

The Python high-level AutoModel and InferenceSession APIs still require LOCAL. This transport fix does not remove that restriction. It does not establish support for every backend, dynamic shape, or native dtype.

Use the same model and runtime for a performance comparison. Check the results first. Then compare complete inference latency, tail latency, and CPU use. Process scheduling and communication still have a cost. Not every model can run at near-local inference speed.

Set inputs, execute, and read outputs in one call

Use Interpreter::run() with matching SDK headers and libraries that contain this interface. The existing functions remain available. Existing call sequences do not change automatically.

std::vector<Aidlux::Aidlite::RunInput> inputs{
    {0, input.data(), input.size() * sizeof(float), false}
};
std::vector<Aidlux::Aidlite::RunOutput> outputs{{0, false}};
if (interpreter->run(inputs, outputs) != 0) {
    // Report the failure. Do not retry automatically.
    return 1;
}
// outputs[0].data is a borrowed view; outputs[0].byte_size is its length.

The example assumes a contiguous FP32 buffer required by the current backend input contract. Use the required data type for other models. byte_size counts bytes, not elements. Each input can select is_native independently. Native data must match the backend representation.

Supply every input in one call. Inputs bind by index and can appear in a different order. Select the required outputs, or use an empty output list. Duplicate indices, incorrect lengths, and models that are not loaded cause errors.

run() is serialized with data and lifecycle operations on the same interpreter. Returned outputs are borrowed views. They expire at the next model operation or object release. Copy results before you keep them. This interface does not extend the lifetime of the C++ object itself.

The REMOTE client uses one inter-process communication (IPC) request only when the service declares FUSED_RUN. Old services use the existing input, execution, and output steps. Old plugins without the optional interface also use these steps. Capability checks do not execute a model.

An error after the call starts can occur after model execution. The client does not retry automatically. It does not run the legacy route after an execution error. After a timeout, create a new object as specified in the recovery procedure above.

The current complete REMOTE call requires fixed, verifiable tensor lengths. It does not switch runtime graphs. Types that the Android SDK does not support cause explicit errors. They are not interpreted as FP32. This limit does not require every model to use FP32. The high-level Python REMOTE restriction remains.

REMOTE shared-memory resources

Reuse a loaded interpreter for repeated inference. When you finish with a model, stop using its output pointers. Then call destroy() and check the return value.

The IPC shared-memory fix requires matching packages. Update the Linux client IPC library. Update the Android service IPC library and JNI components. An update on one side cannot fix a resource leak on the other side. A source merge does not update device packages.

The updated IPC library releases the object's memory mapping and descriptor. A failed connection also closes its temporary descriptor. When you stop a legacy descriptor server, the library waits for its worker to exit. Independent mappings held by other objects remain valid.

This fix adds no model-format or dtype support. Use matching clients, services, and runtimes for a performance comparison. Check inference results first. Then compare complete-call latency.

REMOTE model-information negotiation

The updated client and Android service select a protocol when they create a model. If the peer does not declare the new feature, the client uses the legacy protocol. The C++ call sequence stays the same.

1. Create and load the model. The new load reply contains the operation status only.
2. The client reads complete information through get_model_info and caches it. This read occurs during model loading.
3. Later input, execution, and output calls use shared memory. They do not query model information again.

One reply from the new information interface has a 4 MiB limit. It preserves graph groups, tensor names, indices, element counts, runtime data types, and shapes. The legacy control payload still has a 1008-byte limit. Old clients cannot read multiple graphs or information that exceeds this payload.

The backend SDK supplies these fields. They describe the converted runtime tensors. They do not replace the original model input and output contract in AIMO. The service does not infer original layouts, quantization scales, or offsets. The current Java SDK interface does not supply these fields. Prepare data according to the original model contract. Do not change every input to FP32. The legacy QNN FP32 fallback applies only to the old-service route described above.

The model worker reads the information. Other threads read an immutable cache. If the backend cannot supply a category, the reply marks it unavailable. Do not infer a data type or quantization parameters from an empty list. Information queries do not use the inference control area.

The new protocol assigns an identity token to each model. After release, the old token cannot query or call a new model in the same slot. This token prevents accidental model reuse. It is not an access-authorization mechanism.

The current C++ tensor-information API uses unsigned shapes. A negative dynamic dimension causes an explicit unsupported-shape error. The client does not convert it to a large positive value. Transport of multiple graph descriptions does not enable graph switching in REMOTE. The high-level Python interfaces still require LOCAL.

Select a separate backend directory

Use a core library that supports AIDLITE_BACKEND_LIBRARY_DIR to select a separate backend directory for a process. This setting selects Linux client plugins. Keep the installed plugins when you test a new backend.

1. Put the matching backend libraries in a separate directory. Use an absolute path.

2. Use matching SDK headers and the core library. Keep backend dependencies in the dynamic linker's search path.

3. Start the test program. For example:

   env AIDLITE_BACKEND_LIBRARY_DIR="$PWD/test-backends" ./infer

Set the variable before you start the process. If the variable is absent, the core uses the existing default directory. An empty value, relative path, or missing plugin causes initialization to fail. The core reports an error if it cannot load the plugin from the specified directory.

This variable selects the client plugin only. For REMOTE, check the Android service version and features. Use separate processes to compare different backend versions.

Model Framework Type enum FrameworkType

As mentioned earlier, AidliteSDK integrates multiple deep learning inference frameworks, so in the usage workflow described above, you need to set which framework's model is currently being used, which requires this framework type enumeration.

Member Variable	Type	Value	Description
TYPE_DEFAULT	uint8_t	0	Invalid FrameworkType
TYPE_SNPE	uint8_t	1	SNPE1.x(dlc) model
TYPE_TFLITE	uint8_t	2	TFLite model
TYPE_RKNN	uint8_t	3	RKNN model
TYPE_QNN	uint8_t	4	QNN model
TYPE_SNPE2	uint8_t	5	SNPE2.x(dlc) model
TYPE_NCNN	uint8_t	6	NCNN model
TYPE_MNN	uint8_t	7	MNN model
TYPE_TNN	uint8_t	8	TNN model
TYPE_PADDLE	uint8_t	9	Paddle model
TYPE_MS	uint8_t	10	MindSpore model
TYPE_ONNX	uint8_t	11	ONNX model
TYPE_QNN216	uint8_t	101	QNN2.16 model
TYPE_SNPE216	uint8_t	102	SNPE2.16 model
TYPE_QNN223	uint8_t	103	QNN2.23 model
TYPE_SNPE223	uint8_t	104	SNPE2.23 model
TYPE_QNN229	uint8_t	105	QNN2.29 model
TYPE_SNPE229	uint8_t	106	SNPE2.29 model
TYPE_QNN231	uint8_t	107	QNN2.31 model
TYPE_QNN236	uint8_t	108	QNN2.36 model
TYPE_QNN240	uint8_t	109	QNN2.40 model

💡Note

Special Notes:

• If the user environment has multiple backend implementation packages installed at the same time, such as AidLite-QNN216, AidLite-QNN223, and AidLite-QNN231, and FrameworkType in the code is set to TYPE_QNN (that is, no specific QNN backend version is explicitly selected), then the backend that actually runs should be the highest available QNN version. In other words, AidLite-QNN231 is preferred for inference. SNPE2 follows the same rule.
• The same program cannot use backend packages of different QNN versions at the same time. In other words, if the program first specifies TYPE_QNN216 for inference, it cannot then use TYPE_QNN223 or TYPE_QNN231 for model inference. SNPE2 follows the same rule.

Acceleration Hardware Type enum AccelerateType

For each deep learning inference framework, it may support running on different acceleration devices (such as SNPE/QNN models running on DSP devices, RKNN models running on NPU devices). Therefore, in the usage workflow described above, you need to set which device the current model is expected to run on, which requires this acceleration hardware enumeration.

Member Variable	Type	Value	Description
TYPE_DEFAULT	uint8_t	0	Invalid AccelerateType
TYPE_CPU	uint8_t	1	CPU acceleration
TYPE_GPU	uint8_t	2	GPU acceleration
TYPE_DSP	uint8_t	3	DSP acceleration
TYPE_NPU	uint8_t	4	NPU acceleration

Log Level enum LogLevel

AidliteSDK provides interfaces for setting logs (which will be introduced later). You need to provide the interface with the current log level to use, so this log level enumeration is needed.

Member Variable	Type	Value	Description
INFO	uint8_t	0	Information
WARNING	uint8_t	1	Warning
ERROR	uint8_t	2	Error
FATAL	uint8_t	3	Fatal error

Device Information Class class DeviceInfo

Starting from version 2.3.1, Aidlite uses DeviceInfo to describe the available underlying inference devices. It is currently mainly used for device queries and custom device configuration in QNN DSP scenarios.

Member Variable List

Member Variable	Type	Default Value	Description
version	uint32_t	0X00020301	Structure version number of DeviceInfo
id	uint32_t	0	Device ID
type	uint32_t	0	Device type number interpreted by the backend
cores_num	uint32_t	1	Number of cores included in the device
cores_id	std::vector<uint32_t>	Empty	List of device core IDs

Input/Output Information Class class TensorInfo

Each model has its own inputs and outputs. For models that developers are not familiar with, they may need to query the input and output tensor information of the model during development to facilitate pre- and post-processing operations for model inference. Aidlite provides interfaces for developers to query detailed information about model inputs and outputs.

Member Variable List

The TensorInfo object is used to store detailed information about model input and output tensors, including the following parameters:

Member Variable	version
Type	uint32_t
Default Value	0X00020205
Description	This member is used to identify the TensorInfo version. If more information needs to be extended in the future, the version number will be updated

Member Variable	index
Type	uint32_t
Default Value	No default value
Description	Index value of the Tensor

Member Variable	name
Type	std::string
Default Value	No default value
Description	Name string of the Tensor

Member Variable	element_count
Type	uint64_t
Default Value	0
Description	Number of data elements in the Tensor

Member Variable	element_type
Type	enum DataType
Default Value	DataType::TYPE_DEFAULT
Description	Type of data elements in the Tensor

Member Variable	dimensions
Type	uint32_t
Default Value	0
Description	Dimension value of data elements in the Tensor

Member Variable	shape
Type	std::vector< uint64_t >
Default Value	No default value
Description	Shape description of data elements in the Tensor

Model Class .class Model

As mentioned earlier, before creating an inference interpreter, you need to set detailed parameters related to the specific model. The Model class is mainly used to record model file information, structure information, and model-related content during runtime.

Create Empty Instance Object .create_instance()

The current header also provides a no-argument overload that creates an empty Model instance first, allowing the caller to fill in the model path or property information later in the workflow.

API	create_instance
Description	Construct a Model instance object that does not carry a model file path
Parameters	void
Return Value	If the value is nullptr, the function failed to construct the object; otherwise, it is a pointer to the Model object

Create Instance Object .create_instance()

To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as snpe, tflite, and so on) requires only one model file, call this interface.

API	create_instance
Description	Construct a Model type instance object by passing the path name of the model file
Parameters	model_path: Path name of the model file
Return Value	If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object

// Create model object using inceptionv3_float32.dlc file in current path, error if return value is null
Model* model = Model::create_instance("./inceptionv3_float32.dlc");
if(model == nullptr){
    printf("Create model failed !\n");
    return EXIT_FAILURE;
}

Create Instance Object create_instance()

To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as ncnn, tnn, etc.) involves two model files, call this interface.

API	create_instance
Description	Construct a Model type instance object by passing the path name of the model file
Parameters	model_struct_path: Path name of the model structure file
	model_weight_path: Path name of the model weight parameter file
Return Value	If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object

// Create model object using two ncnn model files in current path, error if return value is null
Model* model = Model::create_instance("./mobilenet_ssd_voc_ncnn.param", 
     "./mobilenet_ssd_voc_ncnn.bin");
if(model == nullptr){
    printf("Create model failed !\n");
    return EXIT_FAILURE;
}

Set Model Properties set_model_properties()

After the model instance object is successfully created, you need to set the input and output data types of the model and the shape information of the input and output tensor data.

API	set_model_properties
Description	Set model properties, input and output data shapes and data types
Parameters	input_shapes: Shape array of input tensors, two-dimensional array structure
	input_data_type: Data type of input tensors, DataType enumeration type
	output_shapes: Shape array of output tensors, two-dimensional array structure
	output_data_type: Data type of output tensors, DataType enumeration type
Return Value	0 indicates success; a non-zero value indicates failure
Note	Declare input and output information for the ordinary REMOTE route. The ordinary SNPE/QNN route uses DataType::TYPE_FLOAT32. Other backends use the declared model dtype. Native QNN data requires actual tensor information. See REMOTE compatibility and recovery.

// Data types use the aforementioned DataType enum, input/output shapes are 2D arrays
std::vector<std::vector<uint32_t>> input_shapes = {{1,299,299,3}};
std::vector<std::vector<uint32_t>> output_shapes = {{1,1001}};
model->set_model_properties(input_shapes, DataType::TYPE_FLOAT32, 
    output_shapes, DataType::TYPE_FLOAT32);

Get Model Path get_model_absolute_path()

If the model framework (such as SNPE, TFLite, etc.) involves only one model file, and the Model object construction is successful, the input model file parameter will be converted to an absolute path form.

API	get_model_absolute_path
Description	Used to get the absolute path of the model file
Parameters	void
Return Value	Model file path string object corresponding to the Model object

Get Model Path get_model_struct_absolute_path()

If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model structure file parameter will be converted to an absolute path form.

API	get_model_struct_absolute_path
Description	Used to get the absolute path of the model structure file
Parameters	void
Return Value	Model structure file path string object corresponding to the Model object

Get Model Path get_model_weight_absolute_path()

If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model weight parameter file parameter will be converted to an absolute path form.

API	get_model_weight_absolute_path
Description	Used to get the absolute path of the model weight parameter file
Parameters	void
Return Value	String object representing the model weight parameter file path corresponding to the Model object

Configuration Class class Config

Before creating an inference interpreter, in addition to setting the specific Model information, you also need to set some runtime configuration information for inference. The Config class is used to record configuration options that need to be preset, and these configuration items will be used at runtime.

Create Instance Object.create_instance()

To set runtime configuration information, you first need a configuration instance object. This function is used to create a Config instance.

API	create_instance
Description	Used to construct an instance object of the Config class
Parameters	void
Return Value	If the value is nullptr, the function failed to construct the object; otherwise, it is a pointer to the Config object

// Create a configuration instance object; report an error if the return value is null
Config* config = Config::create_instance();
if(config == nullptr){
    printf("Create config failed.\n");
    return EXIT_FAILURE;
}

Member Variable List

The Config object is used to set runtime configuration information, which includes the following parameters:

Member	accelerate_type
Type	enum AccelerateType
Default	AccelerateType.TYPE_CPU
Description	Type of acceleration hardware

Member	implement_type
Type	enum ImplementType
Default	ImplementType.TYPE_DEFAULT
Description	Used to distinguish different environments for backend inference implementation
Note	Developers do not need to explicitly set this configuration item. AidliteSDK will adaptively determine a more suitable ImplementType according to factors such as the current system environment and model framework

Member	framework_type
Type	enum FrameworkType
Default	FrameworkType.TYPE_DEFAULT
Description	Type of the underlying inference framework

Member	number_of_threads
Type	int8_t
Default	-1
Description	Number of threads; valid when greater than 0
Note	This configuration item can be set optionally if and only if framework_type is TFLITE

Member	snpe_out_names
Type	string array
Default	Empty
Description	List of model output node names
Note	This configuration item must be set if and only if framework_type is SNPE

Member	is_quantify_model
Type	int32_t
Default	0
Description	Whether the model is quantized; 1 indicates a quantized model
Note	This configuration item must be set if and only if implement_type is REMOTE

Member	remote_timeout
Type	int32_t
Default	-1
Description	Interface timeout duration, in milliseconds; valid when greater than 0
Note	This configuration item can be set optionally if and only if implement_type is REMOTE

Member	qnn_shared_buffer
Type	int32_t
Default	0
Description	Whether to enable the QNN shared buffer; currently mainly used in QNN DSP scenarios

Member	batch_invoke_size
Type	uint32_t
Default	1
Description	Backend batch inference configuration parameter; currently mainly used in QNN DSP scenarios

Member	backend_extension_config
Type	std::string
Default	Empty string
Description	Extension configuration file of the backend inference framework; currently mainly used in QNN DSP scenarios

Member	custom_device_info
Type	std::vector<DeviceInfo>
Default	Empty
Description	Custom device information list used to specify the actual device set used during backend QNN DSP inference

Member	reserved1
Type	int32_t
Default	0
Description	Reserved field. It currently has no business semantics, and callers do not need to set it explicitly

💡Note

remote_timeout in Config shares the same union memory with the historical field fast_timeout. fast_timeout has already been marked as deprecated in the header file and is retained only for compatibility. New code should consistently use remote_timeout.

config->framework_type = FrameworkType::TYPE_QNN;
config->accelerate_type = AccelerateType::TYPE_DSP;
// Set other configuration items as needed
// ... ...

Context Class class Context

Used to store runtime context involved in the execution process, including Model objects and Config objects. More runtime data may be extended in the future.

Constructor Context()

API	Context
Description	Construct Context instance object
Parameters	model: Model instance object pointer
	config: Config instance object pointer
Return Value	Normal: Context instance object

💡Note

Special note: The memory space pointed to by the model and config pointer parameters must be allocated on the heap. After Context is successfully constructed, their lifecycle will be managed by Context (Context will automatically release model and config objects when released), so developers do not need to manually release the space of both, otherwise it will report an error of duplicate object release.

Get Model Member Variable get_model()

API	get_model
Description	Get the Model object pointer managed by context
Parameters	void
Return Value	Pointer to Model object

Get Config Member Variable get_config()

API	get_config
Description	Get the Config object pointer managed by context
Parameters	void
Return Value	Pointer to Config object

Interpreter Class class Interpreter

The Interpreter class object instance is the main body for performing inference operations and is used to execute specific inference processes. In the inference workflow mentioned in the previous sections, after creating the interpreter object, all operations are based on the interpreter object, making it the absolute core content of AidliteSDK.

Create Instance Object create_instance()

To perform inference-related operations, an inference interpreter is essential. This function is used to construct an inference interpreter instance object.

API	create_instance
Description	Construct an Interpreter type object using various data managed by the Context object
Parameters	context: Pointer to Context instance object
Return Value	If nullptr, it indicates the function failed to construct the object; otherwise, it is a pointer to the Interpreter object

// Use Context object pointer to create interpreter object, error if return value is null
Interpreter* current_interpreter = Interpreter::create_instance(context);
if(current_interpreter == nullptr){
    printf("Create Interpreter failed !\n");
    return EXIT_FAILURE;
}

Initialize init()

After the interpreter object is created, some initialization operations need to be performed (such as environment checks, resource construction, etc.).

API	init
Description	Complete the initialization work required for inference
Parameters	void
Return Value	A value of 0 indicates successful initialization; otherwise, a non-zero value indicates failure

// Initialize interpreter, error if return value is non-zero
int result = aidlite_interpreter->init();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->init() failed !\n");
    return EXIT_FAILURE;
}

Get Device Information get_device_info()

The current header file adds get_device_info(std::vector<DeviceInfo>& device_info) for querying the device information actually visible to the backend. This interface is supported starting from version 2.3.1 and is currently mainly used in QNN DSP scenarios.

API	get_device_info
Description	Get the list of backend device details
Parameters	device_info: Array used to store device details
Return Value	A value of 0 indicates success; otherwise, a non-zero value indicates failure

Load Model load_model()

After the interpreter object completes initialization operations, the required model files can be loaded for the interpreter to complete model loading work. The subsequent inference process uses the loaded model resources.

API	load_model
Description	Complete the model loading related work. Since the model file path has been set in the Model object as mentioned above, the model loading operation can be executed directly
Parameters	void
Return Value	A value of 0 indicates successful model loading operation; otherwise, a non-zero value indicates failure

// Load model for interpreter, error if return value is non-zero
int result = aidlite_interpreter->load_model();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->load_model() failed !\n");
    return EXIT_FAILURE;
}

Get Input Tensor Details get_input_tensor_info()

If you want to get detailed information about the model's input tensors, you can use this interface to obtain it.

API	get_input_tensor_info
Description	Get detailed information of input Tensor in the model
Parameters	in_tensor_info: Array storing detailed information of input tensor
Return Value	A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note	This interface is supported starting from version 2.2.5

// Get model input details data, error if return value is non-zero
std::vector<std::vector<TensorInfo>> in_tensor_info;
result = interpreter->get_input_tensor_info(in_tensor_info);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_input_tensor_info() failed !\n");
    return EXIT_FAILURE;
}
for(int gi = 0; gi < in_tensor_info.size(); ++gi){
    for(int ti = 0; ti < in_tensor_info[gi].size(); ++ti){

        std::string tensor_shape_str = std::to_string(in_tensor_info[gi][ti].shape[0]);
        for(int si = 1; si < in_tensor_info[gi][ti].dimensions; ++si){
            tensor_shape_str += " ";
            tensor_shape_str += std::to_string(in_tensor_info[gi][ti].shape[si]);
        }

        printf("Input  tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n",
                gi, ti, in_tensor_info[gi][ti].name.c_str(), in_tensor_info[gi][ti].element_count, 
                (int)in_tensor_info[gi][ti].element_type, in_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str());
    }
}

Get Output Tensor Details get_output_tensor_info()

If you want to get detailed information about the model's output tensors, you can use this interface to obtain it.

API	get_output_tensor_info
Description	Get detailed information of output Tensor in the model
Parameters	out_tensor_info: Array storing detailed information of output tensor
Return Value	A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note	This interface is supported starting from version 2.2.5

// Get model output details data, error if return value is non-zero
std::vector<std::vector<TensorInfo>> out_tensor_info;
result = aidlite_interpreter->get_output_tensor_info(out_tensor_info);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_output_tensor_info() failed !\n");
    return EXIT_FAILURE;
}
for(int gi = 0; gi < out_tensor_info.size(); ++gi){
    for(int ti = 0; ti < out_tensor_info[gi].size(); ++ti){

        std::string tensor_shape_str = std::to_string(out_tensor_info[gi][ti].shape[0]);
        for(int si = 1; si < out_tensor_info[gi][ti].dimensions; ++si){
            tensor_shape_str += " ";
            tensor_shape_str += std::to_string(out_tensor_info[gi][ti].shape[si]);
        }

        printf("Output tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n",
                gi, ti, out_tensor_info[gi][ti].name.c_str(), out_tensor_info[gi][ti].element_count, 
                (int)out_tensor_info[gi][ti].element_type, out_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str());
    }
}

Set Input Data set_input_tensor()

As mentioned in the workflow introduction earlier, before setting input data, different preprocessing operations need to be completed for different models to adapt to specific models.

API	set_input_tensor
Description	Set the data of input tensor required for inference
Parameters	in_tensor_idx/in_tensor_name: Index value/name string of input tensor
	input_data: Pointer to the data storage address of input tensor
Optional Parameters	is_native: Whether to treat the input data as native backend data format; when `false`, the SDK handles quantization/dequantization according to the default logic, and when `true`, the data is passed directly in the original format
Return Value	A value of 0 indicates successful setting of input Tensor; otherwise, a non-zero value indicates failure
Special Note	1. Parameter `in_tensor_name` is supported starting from version 2.2.5 2. The current header file provides two groups of overloads: one with the `is_native` parameter and one without it 3. For SNPE/QNN frameworks, input data is normally handled as float32; when using the native-format interface, users must ensure the data format matches the backend requirements

// Set input data for inference, error if return value is non-zero
// int result = aidlite_interpreter->set_input_tensor(0, input_tensor_data);
int result = aidlite_interpreter->set_input_tensor("in_tensor_name", input_tensor_data);
if(result != EXIT_SUCCESS){
    printf("interpreter->set_input_tensor() failed !\n");
    return EXIT_FAILURE;
}

Execute Inference .invoke()

As mentioned in the previous sections about the workflow, after setting up the input data, the next step is naturally to execute the inference process on the input data.

API	invoke
Description	Execute the inference computation process
Parameters	void
Return Value	A value of 0 indicates successful inference execution, otherwise a non-zero value indicates execution failure

// Execute inference operation, non-zero return value indicates error
int result = aidlite_interpreter->invoke();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->invoke() failed !\n");
    return EXIT_FAILURE;
}

Get Output Data .get_output_tensor()

After inference is completed, the result data obtained from inference needs to be extracted. As mentioned in the previous sections about the workflow, after extracting the result data, these result data can be processed to determine whether the results are correct.

API	get_output_tensor
Description	Get inference result data after successful inference
Parameters	out_tensor_idx/out_tensor_name: Index value/name string of the output result tensor
	output_data: This value is the address of a pointer variable, the function internally resets the value of this pointer variable
	output_length: The data size (in bytes) of this result output tensor
Optional Parameters	is_native: Whether to directly return output in the native backend format; when `false`, the SDK processes it according to the default logic, and when `true`, it directly returns a view of the backend output buffer in the original model data format
Return Value	A value of 0 indicates successful output tensor retrieval, otherwise a non-zero value indicates execution failure
Special Note	1. Parameter `out_tensor_name` is supported starting from version 2.2.5 2. `output_length` indicates the byte size of the output Tensor data 3. The current header file provides two groups of overloads: one with the `is_native` parameter and one without it 4. For SNPE/QNN, the default interface usually returns data with float32 semantics; when using the native-format interface, users need to parse the output according to the actual backend output format

// Get inference result data, non-zero return value indicates error
float* out_data = nullptr;
uint32_t output_tensor_length = 0;
// int result = aidlite_interpreter->get_output_tensor(0, (void**)&out_data, &output_tensor_length);
int result = aidlite_interpreter->get_output_tensor("out_tensor_name", (void**)&out_data, &output_tensor_length);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_output_tensor() failed !\n");
    return EXIT_FAILURE;
}

Resource Cleanup .destroy()

Call destroy() when you no longer need the model. Check the return value.

If destroy() fails, stop inference on this interpreter. Keep the interpreter object. For LOCAL, correct the problem reported by the backend. Then call destroy() again to retry cleanup. Normal model operations fail after cleanup starts. This rule also applies if initialization fails and resources still need cleanup.

If a REMOTE service reports failed native cleanup, stop requests to that context. The updated service retains its model slot because native resources can still be in use. Stop other model jobs. Check the backend error. Restart the backend service and create a new interpreter. Retrying destroy() on the closed remote context cannot reclaim that server slot. The same rule applies if cleanup fails before the model worker starts.

Do not treat failed cleanup as successful cleanup. The destructor cannot guarantee resource recovery if the backend continues to refuse release. The existing low-level interface still returns failure if you call destroy() before initialization or after successful cleanup.

API	destroy
Description	Perform necessary cleanup operations
Parameters	void
Return Value	A value of 0 indicates successful cleanup operation, otherwise a non-zero value indicates execution failure

// Execute interpreter cleanup process, non-zero return value indicates error
int result = aidlite_interpreter->destroy();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter-> destroy() failed !\n");
    return EXIT_FAILURE;
}

Interpreter Creation Class .class InterpreterBuilder

Unified interpreter creation function for Interpreter objects. Use this class to create the required interpreter objects.

Build Interpreter .build_interpreter_from_path()

Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as SNPE, TFLite, etc.) only requires one model file, call this interface.

API	build_interpreter_from_path
Description	Create corresponding interpreter object directly through the path name of the model file
Parameters	path: Path name of the model file
Return Value	Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

Build Interpreter .build_interpreter_from_path()

Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as NCNN, TNN, etc.) involves two model files, call this interface.

API	build_interpreter_from_path
Description	Create corresponding interpreter object directly through the path name of the model file
Parameters	model_struct_path: Path name of the model structure file
	model_weight_path: Path name of the model weight parameter file
Return Value	Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

Build Interpreter .build_interpreter_from_model()

Build inference interpreter object. In addition to providing model file paths, Model objects can also be provided, which allows setting not only the model file path and name, but also the input/output data types and input/output data shape information of the model.

API	build_interpreter_from_model
Description	Create corresponding interpreter object by passing in Model object. All Config-related parameters use default values
Parameters	model: Model type object containing model-related data
Return Value	Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

💡Note

Special note: The memory space pointed to by the model parameter must be allocated on the heap. After successful Interpreter construction, its lifecycle will be managed by the Interpreter (when the Interpreter is released, it will automatically release the model object), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.

Build Interpreter .build_interpreter_from_model_and_config()

Build inference interpreter object. In addition to the aforementioned methods, Model objects and Config objects can also be provided simultaneously, which allows providing not only model-related information but also more runtime configuration parameters.

API	build_interpreter_from_model_and_config
Description	Create corresponding interpreter object by passing in Model object and Config
Parameters	model: Model type object containing model-related data
	config: Config type object containing configuration parameters
Return Value	Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

💡Note

Special note: The memory space pointed to by both model and config parameters must be allocated on the heap. After successful Interpreter construction, their lifecycles will be managed by the Interpreter (when the Interpreter is released, it will automatically release both model and config objects), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.

// Build interpreter by passing in Model and Config objects, null return value indicates error
std::unique_ptr<Interpreter>&& aidlite_interpreter = 
 InterpreterBuilder::build_interpreter_from_model_and_config(model, config);
if(aidlite_interpreter == nullptr){
    printf("build_interpreter failed !\n");
    return EXIT_FAILURE;
}

Other Methods

In addition to the aforementioned inference-related interfaces, Aidlite-SDK also provides the following auxiliary interfaces.

Get SDK Version Information .get_library_version()

API	get_library_version
Description	Used to get version-related information of the current aidlite-sdk.
Parameters	void
Return Value	Returns the version information string of the current aidlite-sdk

Set Log Level .set_log_level()

API	set_log_level
Description	Set the current minimum log level, output log data greater than or equal to this log level. By default, prints WARNING level and above logs
Parameters	log_level: Enumeration type LogLevel value
Return Value	Returns 0 by default

Log Output to Standard Terminal .log_to_stderr()

API	log_to_stderr
Description	Set log information output to standard error terminal
Parameters	void
Return Value	Returns 0 by default

Log Output to Text File .log_to_file()

API	log_to_file
Description	Set log information output to specified text file
Parameters	path_and_prefix: Storage path and name prefix of the log file
	also_to_stderr: Indicates whether to simultaneously output logs to stderr terminal, default value is false
Return Value	A value of 0 indicates successful execution, otherwise a non-zero value indicates execution failure

Get Recent Log Information .last_log_msg()

API	last_log_msg
Description	Get the latest log information for a specific log level, usually to get the latest error information
Parameters	log_level: Enumeration type LogLevel value
Return Value	Storage address pointer to the latest log information