> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akool.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Streaming Avatar API Overview
> Comprehensive guide to the Streaming Avatar API
Both the avatar\_id and voice\_id can be easily obtained by copying them directly from the web interface. You can also create and manage your streaming avatars using our intuitive web platform.
Create and manage your avatars at: [https://akool.com/apps/upload/avatar?from=%2Fapps%2Fstreaming-avatar%2Fedit](https://akool.com/apps/upload/avatar?from=%2Fapps%2Fstreaming-avatar%2Fedit)
The resources (image, video, voice) generated by our API are valid for 7 days.
Please save the relevant resources as soon as possible to prevent expiration.
To experience our live avatar streaming feature in action, explore our demo built on the Agora streaming service: [AKool Streaming Avatar React Demo](https://github.com/AKOOL-Official/akool-streaming-avatar-react-demo).
**Knowledge Base Integration:** You can enhance your streaming avatar with contextual AI responses by integrating a [Knowledge Base](/ai-tools-suite/knowledge-base). When creating a session, provide a `knowledge_id` parameter to enable the AI to use documents and URLs from your knowledge base for more accurate and relevant responses.
### API Endpoints
#### Avatar Management
* [Upload Streaming Avatar](/ai-tools-suite/live-avatar/upload) - Create a new streaming avatar from a video URL
* [Get Avatar List](/ai-tools-suite/live-avatar/list) - Retrieve a list of all streaming avatars
* [Get Avatar Detail](/ai-tools-suite/live-avatar/detail) - Get detailed information about a specific avatar
#### Session Management
* [Create Session](/ai-tools-suite/live-avatar/create-session) - Create a new streaming avatar session
* [Get Session Detail](/ai-tools-suite/live-avatar/session-detail) - Retrieve detailed information about a specific session
* [Close Session](/ai-tools-suite/live-avatar/close-session) - Close an active streaming avatar session
* [Get Session List](/ai-tools-suite/live-avatar/list-sessions) - Retrieve a list of all streaming avatar sessions
### Live Avatar Stream Message
```ts theme={null}
IAgoraRTCClient.on(event: "stream-message", listener: (uid: UID, pld: Uint8Array) => void)
IAgoraRTCClient.sendStreamMessage(msg: Uint8Array | string, flag: boolean): Promise;
```
**Send Data**
**Chat Type Parameters**
| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| ------------- | -------- | ------------ | --------- | --------------------------------------------------- |
| v | Number | Yes | 2 | Version of the message |
| type | String | Yes | chat | Message type for chat interactions |
| mid | String | Yes | | Unique message identifier for conversation tracking |
| idx | Number | Yes | | Sequential index of the message, start from 0 |
| fin | Boolean | Yes | | Indicates if this is the final part of the message |
| pld | Object | Yes | | Container for message payload |
| pld.text | String | Yes | | Text content to send to avatar (e.g. "Hello") |
**Command Type Parameters**
| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| ---------------- | -------- | ------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| v | Number | Yes | 2 | Protocol version number |
| type | String | Yes | command | Specifies this is a system command message |
| mid | String | Yes | | Unique ID to track and correlate command messages |
| pld | Object | Yes | | Contains the command details and parameters |
| pld.cmd | String | Yes | | Command action to execute. Valid values: **"set-params"** (update avatar settings), **"interrupt"** (stop current avatar response) |
| pld.data | Object | No | | Parameters for the command (required for **"set-params"**) |
| pld.data.vid | String | No | | **Deprecated. Use pld.data.vparams.vid instead.** Voice ID to change avatar's voice. Only used with **"set-params"**. Get valid IDs from [Voice List API](/ai-tools-suite/voiceLab#get-voice-list) |
| pld.data.vurl | String | No | | **Deprecated. Use pld.data.vparams.vurl instead.** Custom voice model URL. Only used with **"set-params"**. Get valid URLs from [Voice List API](/ai-tools-suite/voiceLab#get-voice-list) |
| pld.data.lang | String | No | | Language code for avatar responses (e.g. "en", "es"). Only used with **"set-params"**. Get valid codes from [Language List API](/ai-tools-suite/video-translation#get-language-list-result) |
| pld.data.mode | Number | No | | Avatar interaction style. Only used with **"set-params"**. "1" = Retelling (avatar repeats content), "2" = Dialogue (avatar engages in conversation) |
| pld.data.bgurl | String | No | | URL of background image/video for avatar scene. Only used with **"set-params"** |
| pld.data.vparams | Object | No | | Voice parameters to use for the session. |
**Voice Parameters**
| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| --------------- | -------- | ------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| vid | String | No | | Voice ID to change avatar's voice. Only used with **"set-params"**. Get valid IDs from [Voice List API](/ai-tools-suite/voiceLab#get-voice-list) |
| vurl | String | No | | Custom voice model URL. Only used with **"set-params"**. Get valid URLs from [Voice List API](/ai-tools-suite/voiceLab#get-voice-list) |
| speed | double | No | 1 | Controls the speed of the generated speech. Values range from 0.8 to 1.2, with 1.0 being the default speed. |
| stt\_language | String | No | | Language code for speech-to-text recognition to improve accuracy by using language-specific models |
| pron\_map | Object | No | | Pronunciation mapping for custom words. Example: "pron\_map": \{ "akool" : "ai ku er" } |
| stt\_type | String | No | | Speech-to-text type. `"openai_realtime"` = OpenAI Realtime |
| turn\_detection | Object | No | | Turn detection configuration. |
**Turn Detection Configuration**
| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| --------------------- | -------- | ------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| type | String | No | "server\_vad" | Turn detection type. `"server_vad"` = Server VAD, `"semantic_vad"` = Semantic VAD |
| threshold | Number | No | 0.5 | Activation threshold (0 to 1). A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. Available when type is `"server_vad"`. |
| prefix\_padding\_ms | Number | No | 300 | Amount of audio (in milliseconds) to include before the VAD detected speech. Available when type is `"server_vad"`. |
| silence\_duration\_ms | Number | No | 500 | Duration of silence (in milliseconds) to detect speech stop. With shorter values turns will be detected more quickly. Available when type is `"server_vad"`. |
**JSON Example**
```json Chat Request theme={null}
{
"v": 2,
"type": "chat",
"mid": "msg-1723629433573",
"idx": 0,
"fin": true,
"pld": {
"text": "Hello"
},
}
```
```json Set Avatar Params theme={null}
{
"v": 2,
"type": "command",
"mid": "msg-1723629433573",
"pld": {
"cmd": "set-params",
"data": {
"vid": "1",
"lang": "en",
"mode": 1,
"bgurl": "https://example.com/background.jpg"
}
},
}
```
```json Interrupt Response theme={null}
{
"v": 2,
"type": "command",
"mid": "msg-1723629433573",
"pld": {
"cmd": "interrupt"
},
}
```
```json Set Avatar Actions theme={null}
{
"v": 2,
"type": "command",
"pld": {
"cmd": "set-action",
"data": {
"action": "hand_wave", // "hand_wave"|"cheer"|"thumbs_up"|"pump_fists"|"nod"|"shake"|"waft_left"|"waft_right"
"expression": "happiness" // "angry"|"disgust"|"fear"|"happiness"|"sadness"|"surprise"|"contempt"
}
}
}
```
**Receive Data**
**Chat Type Parameters**
| **Parameter** | **Type** | **Value** | **Description** |
| ------------- | -------- | --------------- | --------------------------------------------------------------------------------------- |
| v | Number | 2 | Version of the message |
| type | String | chat | Message type for chat interactions |
| mid | String | | Unique message identifier for tracking conversation flow |
| idx | Number | | Sequential index of the message part |
| fin | Boolean | | Indicates if this is the final part of the response |
| pld | Object | | Container for message payload |
| pld.from | String | "bot" or "user" | Source of the message - "bot" for avatar responses, "user" for speech recognition input |
| pld.text | String | | Text content of the message |
**Command Type Parameters**
| **Parameter** | **Type** | **Value** | **Description** |
| ------------- | -------- | ------------------------- | -------------------------------------------------------------------------------------------------------- |
| v | Number | 2 | Version of the message |
| type | String | command | Message type for system commands |
| mid | String | | Unique identifier for tracking related messages in a conversation |
| pld | Object | | Container for command payload |
| pld.cmd | String | "set-params", "interrupt" | Command to execute: **"set-params"** to update avatar settings, **"interrupt"** to stop current response |
| pld.code | Number | 1000 | Response code from the server, 1000 indicates success |
| pld.msg | String | | Response message from the server |
**JSON Example**
```json Chat Response theme={null}
{
"v": 2,
"type": "chat",
"mid": "msg-1723629433573",
"idx": 0,
"fin": true,
"pld": {
"from": "bot",
"text": "Hello! How can I assist you today? "
}
}
```
```json Command Response theme={null}
{
"v": 2,
"type": "command",
"mid": "msg-1723629433573",
"pld": {
"cmd": "set-params",
"code": 1000,
"msg": "OK"
}
}
```
**Typescript Example**
```ts Create Client theme={null}
const client: IAgoraRTCClient = AgoraRTC.createClient({
mode: 'rtc',
codec: 'vp8',
});
client.join(agora_app_id, agora_channel, agora_token, agora_uid);
client.on('stream-message', (message: Uint8Array | string) => {
console.log('received: %s', message);
});
```
```ts Send Message theme={null}
const msg = JSON.stringify({
v: 2,
type: "chat",
mid: "msg-1723629433573",
idx: 0,
fin: true,
pld: {
text: "hello"
},
});
await client.sendStreamMessage(msg, false);
```
```ts Set Avatar Params theme={null}
const setAvatarParams = async () => {
const msg = JSON.stringify({
v: 2,
type: 'command',
pld: {
cmd: 'set-params',
params: {
vid: voiceId,
lang: language,
mode: modeType,
},
},
});
return client.sendStreamMessage(msg, false);
};
```
```ts Interrupt Response theme={null}
const interruptResponse = async () => {
const msg = JSON.stringify({
v: 2,
type: 'command',
pld: {
cmd: 'interrupt',
},
});
return client.sendStreamMessage(msg, false);
};
```
### Integrating Your Own LLM Service
Before dispatching a message to the WebSocket, consider executing an HTTP request to your LLM service.
```ts TypeScript theme={null}
const client: IAgoraRTCClient = AgoraRTC.createClient({
mode: 'rtc',
codec: 'vp8',
});
client.join(agora_app_id, agora_channel, agora_token, agora_uid);
client.on('stream-message', (message: Uint8Array | string) => {
console.log('received: %s', message);
});
let inputMessage = 'hello';
try {
const response = await fetch('https://your-backend-host/api/llm/answer', {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
question: inputMessage,
}),
});
if (response.ok) {
const result = await response.json();
inputMessage = result.answer;
} else {
console.error("Failed to fetch from backend", response.statusText);
}
} catch (error) {
console.error("Error during fetch operation", error);
}
const message = {
v: 2,
type: "chat",
mid: "msg-1723629433573",
idx: 0,
fin: true,
pld: {
text: inputMessage,
},
};
client.sendStreamMessage(JSON.stringify(message), false);
```
### Response Code Description
{" "}
Please note that if the value of the response code is not equal to 1000, the request
is failed or wrong
| **Parameter** | **Value** | **Description** |
| ------------- | --------- | --------------------------------------------------------------------------------- |
| code | 1000 | Success |
| code | 1003 | Parameter error or Parameter can not beempty |
| code | 1008 | The content you get does not exist |
| code | 1009 | Youdo not have permission to operate |
| code | 1101 | Invalid authorization or Therequest token has expired |
| code | 1102 | Authorization cannot be empty |
| code | 1200 | The account has been banned |
| code | 1201 | create audio error, pleasetry again later |
| code | 1202 | The same video cannot be translated lipSync inthe same language more than 1 times |
| code | 1203 | video should be with audio |
| code | 1204 | Your video duration is exceed 60s! |
| code | 1205 | Create videoerror, please try again later |
| code | 1207 | The video you are using exceeds thesize limit allowed by the system by 300M |
| code | 1209 | Please upload a videoin another encoding format |
| code | 1210 | The video you are using exceeds thevalue allowed by the system by 60fp |
| code | 1211 | Create lipsync error, pleasetry again later |