Streaming avatar

The resources (image, video, voice) generated by our API are valid for 7 days. Please save the relevant resources as soon as possible to prevent expiration.

To experience our live avatar streaming feature in action, explore our demo built on the Agora streaming service: AKool Streaming Avatar React Demo.

Knowledge Base Integration: You can enhance your streaming avatar with contextual AI responses by integrating a Knowledge Base. When creating a session, provide a knowledge_id parameter to enable the AI to use documents and URLs from your knowledge base for more accurate and relevant responses.

Upload Streaming Avatar

POST https://openapi.akool.com/api/open/v3/avatar/create

Request Headers

Parameter	Type	Required	Description
Authorization	String	Yes	Bearer token for API authentication. Obtain from GetToken endpoint.

Body Attributes

Parameter	Type	Value	Description
url	String		Avatar resource link. It is recommended that the video be about one minute long, and the avatar in the video content should rotate at a small angle and be clear.
avatar_id	String		avatar unique ID，Can only contain /^a-zA-Z0-9/.
type	String	2	Avatar type 2 represents stream avatar, When type is 2, you need to wait until status is 3 before you can use it, You can get the current status in real time through the interface GET https://openapi.akool.com/api/open/v3/avatar/detail.
url_from	Number	1，2	url source, 1 means akool and other links, 2 means other third-party links (currently only supports YouTube / TikTok / X / Google Drive)

Response Attributes

Parameter	Type	Value	Description
code	int	1000	Interface returns business status code(1000:success)
msg	String	OK	Interface returns status information
data	Object	`{ avatar_id: "xx", url: "", status: 1 }`	avatar_id: Used by creating live avatar interface. url: You can preview the avatar via the link. status: 1-queueing, 2-processing, 3-success, 4-failed

Example Request

curl --location 'https://openapi.akool.com/api/open/v3/avatar/create' \
--header 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjY0ZDk4NWM1NTcxNzI5ZDNlMjk5OTQ3NyIsInVpZCI6Mzc4MzM3LCJlbWFpbCI6Imh1Y2hlbkBha29vbC5jb20iLCJjcmVkZW50aWFsSWQiOiI2NjE1MGZmM2Q5MWRmYjc4OWYyNjFmNjEiLCJmaXJzdE5hbWUiOiJjaGVuIiwiZnJvbSI6InRvTyIsInR5cGUiOiJ1c2VyIiwiaWF0IjoxNzEyNzE0ODI4LCJleHAiOjIwMjM3NTQ4Mjh9.e050LbczNhUx-Gprqb1NSYhBCKKH2xMqln3cMnAABmE' \
--header 'Content-Type: application/json' \
--data '{
    "url": "https://drz0f01yeq1cx.cloudfront.net/1721197444322-leijun000.mp4",
    "avatar_id": "HHdEKhn7k7vVBlR5FSi0e",
    "type": 2,
    "url_from": 1
}'

Response

{
  "code": 1000,
  "msg": "ok",
  "data": {
    "_id": "655ffeada6976ea317087193",
    "disabled": false,
    "uid": 1,
    "type": 2,
    "from": 2,
    "status": 1,
    "sort": 12,
    "create_time": 1700788730000,
    "name": "Yasmin in White shirt",
    "avatar_id": "Yasmin_in_White_shirt_20231121",
    "url": "https://drz0f01yeq1cx.cloudfront.net/1700786304161-b574407f-f926-4b3e-bba7-dc77d1742e60-8169.png",
    "modify_url": "https://drz0f01yeq1cx.cloudfront.net/1700786304161-b574407f-f926-4b3e-bba7-dc77d1742e60-8169.png",
    "gender": "female",
    "thumbnailUrl": "https://drz0f01yeq1cx.cloudfront.net/avatar/thumbnail/1700786304161-b574407f-f926-4b3e-bba7-dc77d1742e60-8169.png",
    "crop_arr": []
  }
}

Get Streaming Avatar List

GET https://openapi.akool.com/api/open/v4/liveAvatar/avatar/list

Request Headers

Parameter	Type	Required	Description
Authorization	String	Yes	Bearer token for API authentication. Obtain from GetToken endpoint.

Query Parameters

Parameter	Type	Required	Default	Description
page	Number	No	1	Page number for pagination
size	Number	No	100	Number of items per page (max 100)

Response Properties

Property	Type	Description
code	Integer	Response status code (1000 indicates success)
msg	String	Response status message
data	Object	Container for response data
data.count	Number	Total number of streaming avatars
data.result	Array	List of streaming avatar objects

Streaming Avatar Object Properties

Property	Type	Description
avatar_id	String	Unique identifier for the streaming avatar
voice_id	String	Associated voice model identifier
name	String	Display name of the avatar
url	String	URL to access the streaming avatar
thumbnailUrl	String	URL for the avatar’s preview thumbnail
gender	String	Avatar’s gender designation
available	Boolean	Indicates if the avatar is currently available for use
type	Number	Avatar type identifier (2 for streaming avatars)
from	Number	Source identifier for the avatar, 2 for official and 3 for user created

Example Request

curl --location 'https://openapi.akool.com/api/open/v4/liveAvatar/avatar/list?page=1&size=100' \
--header 'Authorization: Bearer {{Authorization}}'

Response

{
  "code": 1000,
  "msg": "OK",
  "data": {
    "count": 1,
    "result": [
      {
        "_id": "67498e55cd053926b2e75cf2",
        "uid": 1,
        "type": 2,
        "from": 2,
        "avatar_id": "dvp_Tristan_cloth2_1080P",
        "voice_id": "iP95p4xoKVk53GoZ742B",
        "name": "tristan_dvp",
        "url": "https://static.website-files.org/assets/avatar/avatar/streaming_avatar/tristan_10s_silence.mp4",
        "thumbnailUrl": "https://static.website-files.org/assets/avatar/avatar/thumbnail/1716457024728-tristan_cloth2_20240522.webp",
        "gender": "male",
        "available": true
      }
    ]
  }
}

Create session

POST https://openapi.akool.com/api/open/v4/liveAvatar/session/create

Request Headers

Parameter	Value	Description
Authorization	Bearer `{token}`	Your API Key used for request authorization. getToken

Body Attributes

Parameter	Type	Required	Description
avatar_id	String	Yes	Digital human model in real-time avatar, The current system provides only one option: “dvp_Tristan_cloth2_1080P”. If you want to use a custom uploaded video, you need to call the https://openapi.akool.com/api/open/v3/avatar/create interface to create a template. This process takes some time to process. You can check the processing status through the interface https://openapi.akool.com/api/open/v3/avatar/detail. When status=3, you can use the avatar_id field to pass it in.
duration	Number	No	The duration of the session, in seconds. The maximum value is 3600 seconds.
knowledge_id	String	No	Knowledge base ID to provide context for AI responses. Create and manage knowledge bases using the Knowledge Base API. When provided, the AI will use documents and URLs from the knowledge base to enhance response accuracy.

Parameter	Type	Value	Description
code	int	1000	Interface returns business status code(1000:success)
msg	String		Interface returns status information
data	Object	`{ "_id": "", "status": 1, "credentials": {} }`	`_id`: Interface returns data, status: the status of session: [1:queueing, 2:processing, 3:completed, 4:failed], credentials: the join information for the Agora SDK

Example Body

{
  "_id": "6698c9d69cf7b0d61d1b6420",
  "status": 1,
  "stream_type": "agora",
  "credentials": {
    "agora_uid": 100000, // The user ID for the Agora SDK.
    "agora_app_id": "", // The App ID used for the Agora SDK.
    "agora_channel": "", // The specified channel name for the Agora SDK.
    "agora_token": "", // The authentication token for the Agora SDK, currently the valid time is 5 minutes.
  }
}

Request

curl --location 'https://openapi.akool.com/api/open/v4/liveAvatar/session/create' \
--header 'authorization: Bearer token' \
--header 'content-type: application/json' \
--data '{
    "avatar_id": "dvp_Tristan_cloth2_1080P",
    "knowledge_id": "64f8a1b2c3d4e5f6a7b8c9d0"
}'

Response

{
    "code": 1000,
    "msg": "OK",
    "data": {
        "_id": "6698c9d69cf7b0d61d1b6420",
        "uid": 100010,
        "type": 1,
        "status": 1,
        "stream_type": "agora",
        "credentials": {
          "agora_uid": 100000, // The user ID for the Agora SDK.
          "agora_app_id": "", // The App ID used for the Agora SDK.
          "agora_channel": "", // The specified channel name for the Agora SDK.
          "agora_token": "", // The authentication token for the Agora SDK, currently the valid time is 5 minutes.
        }
    }
}

Get Session Info Result

GET https://openapi.akool.com/api/open/v4/liveAvatar/session/detail?id=6698c9d69cf7b0d61d1b6420

Request Headers

Parameter	Value	Description
Authorization	Bearer `{token}`	Your API Key used for request authorization.getToken

Query Attributes

Parameter	Type	Value	Description
id	String	NULL	video db id: You can get it based on the `_id` field returned by Create Session .

Response Attributes

Parameter	Type	Value	Description
code	int	1000	Interface returns business status code(1000:success)
msg	String	OK	Interface returns status information
data	Object	`{ status:1, _id:"", credentials:{} }`	status: the status of live avatar:(1:queueing, 2:processing, 3:completed, 4:failed) credentials: the url of live avatar, _id: Interface returns data

Example Request

curl --location 'http://openapi.akool.com/api/open/v4/liveAvatar/session/detail?id=6698c9d69cf7b0d61d1b6420' \
--header 'Authorization: Bearer token'

Response

{
  "code": 1000,
  "msg": "OK",
  "data": {
    "_id": "6698c9d69cf7b0d61d1b6420",
    "uid": 100010,
    "type": 1,
    "status": 3,
    "stream_type": "agora",
    "credentials": {
        "agora_uid": 100000, // The user ID for the Agora SDK.
        "agora_app_id": "", // The App ID used for the Agora SDK.
        "agora_channel": "", // The specified channel name for the Agora SDK.
    }
  }
}

Close Session

POST https://openapi.akool.com/api/open/v4/liveAvatar/session/close

Request Headers

Parameter	Value	Description
Authorization	Bearer `{token}`	Your API Key used for request authorization. getToken

Body Attributes

Parameter	Type	Value	Description
id	String	NULL	session id: You can get it based on the `_id` field returned by Create session .

Response Attributes

Parameter	Type	Value	Description
code	int	1000	Interface returns business status code(1000:success)
msg	String	OK	Interface returns status information

Example Request

curl --location 'http://openapi.akool.com/api/open/v4/liveAvatar/session/close' \
--header 'Authorization: Bearer token'

Response

{
  "code": 1000,
  "msg": "OK"
}

Get Session List

GET https://openapi.akool.com/api/open/v4/liveAvatar/session/list?page=1&size=10&status=1

Request Headers

Parameter	Value	Description
Authorization	Bearer `{token}`	Your API Key used for request authorization. getToken

Query Attributes

Parameter	Type	Value	Description
page	Number	1	Current number of pages, Default is 1.
size	Number	10	Current number of returns per page, Default is 100.
status	Number	NULL	session status: (1:queueing, 2:processing, 3:completed, 4:failed).

Response Attributes

Parameter	Type	Value	Description
code	int	1000	Interface returns business status code(1000:success)
msg	String	OK	Interface returns status information
data	Array	`{count: 1, data: [{ credentials: {} }] }`	task_id: task id of session. url: the url of live avatar.

Example Request

curl --location 'http://openapi.akool.com/api/open/v4/liveAvatar/session/list?page=1&size=10&status=1' \
--header 'Authorization: Bearer token'

Response

{
  "code": 1000,
  "msg": "OK",
  "data": {
    "count": 18,
    "result": [
      {
        "_id": "666d3006247f07725af0f884",
        "uid": 100010,
        "type": 1,
        "status": 1,
        "stream_type": "agora",
        "credentials": {
          "agora_uid": 100000, // The user ID for the Agora SDK.
          "agora_app_id": "", // The App ID used for the Agora SDK.
          "agora_channel": "" // The specified channel name for the Agora SDK.
        },
      }
    ]
  }
}

Live Avatar Stream Message

IAgoraRTCClient.on(event: "stream-message", listener: (uid: UID, pld: Uint8Array) => void)

IAgoraRTCClient.sendStreamMessage(msg: Uint8Array | string, flag: boolean): Promise<void>;

Send Data Chat Type Parameters

Parameter	Type	Required	Value	Description
v	Number	Yes	2	Version of the message
type	String	Yes	chat	Message type for chat interactions
mid	String	Yes		Unique message identifier for conversation tracking
idx	Number	Yes		Sequential index of the message, start from 0
fin	Boolean	Yes		Indicates if this is the final part of the message
pld	Object	Yes		Container for message payload
pld.text	String	Yes		Text content to send to avatar (e.g. “Hello”)

Command Type Parameters

Parameter	Type	Required	Value	Description
v	Number	Yes	2	Protocol version number
type	String	Yes	command	Specifies this is a system command message
mid	String	Yes		Unique ID to track and correlate command messages
pld	Object	Yes		Contains the command details and parameters
pld.cmd	String	Yes		Command action to execute. Valid values: “set-params” (update avatar settings), “interrupt” (stop current avatar response)
pld.data	Object	No		Parameters for the command (required for “set-params”)
pld.data.vid	String	No		Voice ID to change avatar’s voice. Only used with “set-params”. Get valid IDs from Voice List API
pld.data.vurl	String	No		Custom voice model URL. Only used with “set-params”. Get valid URLs from Voice List API
pld.data.lang	String	No		Language code for avatar responses (e.g. “en”, “es”). Only used with “set-params”. Get valid codes from Language List API
pld.data.mode	Number	No		Avatar interaction style. Only used with “set-params”. “1” = Retelling (avatar repeats content), “2” = Dialogue (avatar engages in conversation)
pld.data.bgurl	String	No		URL of background image/video for avatar scene. Only used with “set-params”

JSON Example

{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "text": "Hello"
    },
}

Receive Data Chat Type Parameters

Parameter	Type	Value	Description
v	Number	2	Version of the message
type	String	chat	Message type for chat interactions
mid	String		Unique message identifier for tracking conversation flow
idx	Number		Sequential index of the message part
fin	Boolean		Indicates if this is the final part of the response
pld	Object		Container for message payload
pld.from	String	”bot” or “user”	Source of the message - “bot” for avatar responses, “user” for speech recognition input
pld.text	String		Text content of the message

Command Type Parameters

Parameter	Type	Value	Description
v	Number	2	Version of the message
type	String	command	Message type for system commands
mid	String		Unique identifier for tracking related messages in a conversation
pld	Object		Container for command payload
pld.cmd	String	”set-params”, “interrupt”	Command to execute: “set-params” to update avatar settings, “interrupt” to stop current response
pld.code	Number	1000	Response code from the server, 1000 indicates success
pld.msg	String		Response message from the server

JSON Example

{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "from": "bot",
      "text": "Hello!  How can I assist you today? "
    }
}

Typescript Example

const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});

Integrating Your Own LLM Service

Before dispatching a message to the WebSocket, consider executing an HTTP request to your LLM service.

const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});

let inputMessage = 'hello';

try {
  const response = await fetch('https://your-backend-host/api/llm/answer', {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      question: inputMessage,
    }),
  });

  if (response.ok) {
    const result = await response.json();
    inputMessage = result.answer;
  } else {
    console.error("Failed to fetch from backend", response.statusText);
  }
} catch (error) {
  console.error("Error during fetch operation", error);
}

const message = {
  v: 2,
  type: "chat",
  mid: "msg-1723629433573",
  idx: 0,
  fin: true,
  pld: {
      text: inputMessage,
  },
};
client.sendStreamMessage(JSON.stringify(message), false);

Response Code Description

Please note that if the value of the response code is not equal to 1000, the request is failed or wrong

Parameter	Value	Description
code	1000	Success
code	1003	Parameter error or Parameter can not beempty
code	1008	The content you get does not exist
code	1009	Youdo not have permission to operate
code	1101	Invalid authorization or Therequest token has expired
code	1102	Authorization cannot be empty
code	1200	The account has been banned
code	1201	create audio error, pleasetry again later
code	1202	The same video cannot be translated lipSync inthe same language more than 1 times
code	1203	video should be with audio
code	1204	Your video duration is exceed 60s!
code	1205	Create videoerror, please try again later
code	1207	The video you are using exceeds thesize limit allowed by the system by 300M
code	1209	Please upload a videoin another encoding format
code	1210	The video you are using exceeds thevalue allowed by the system by 60fp
code	1211	Create lipsync error, pleasetry again later

Authentication

AI Tools Suite

Implementation guide

Streaming Avatar SDK

Streaming Avatar Knowledge Base

Upload Streaming Avatar

Get Streaming Avatar List

Create session

Get Session Info Result

Close Session

Get Session List

Live Avatar Stream Message

Integrating Your Own LLM Service

Response Code Description

Authentication

AI Tools Suite

Implementation guide

Streaming Avatar SDK

Streaming Avatar Knowledge Base

​Upload Streaming Avatar

​Get Streaming Avatar List

​Create session

​Get Session Info Result

​Close Session

​Get Session List

​Live Avatar Stream Message

​Integrating Your Own LLM Service

​Response Code Description

Upload Streaming Avatar

Get Streaming Avatar List

Create session

Get Session Info Result

Close Session

Get Session List

Live Avatar Stream Message

Integrating Your Own LLM Service

Response Code Description