Streaming Avatar API Overview - Akool open api documents

Both the avatar_id and voice_id can be easily obtained by copying them directly from the web interface. You can also create and manage your streaming avatars using our intuitive web platform.Create and manage your avatars at: https://akool.com/apps/upload/avatar?from=%2Fapps%2Fstreaming-avatar%2Fedit

The resources (image, video, voice) generated by our API are valid for 7 days. Please save the relevant resources as soon as possible to prevent expiration.

To experience our live avatar streaming feature in action, explore our demo built on the Agora streaming service: AKool Streaming Avatar React Demo.

Knowledge Base Integration: You can enhance your streaming avatar with contextual AI responses by integrating a Knowledge Base. When creating a session, provide a knowledge_id parameter to enable the AI to use documents and URLs from your knowledge base for more accurate and relevant responses.

API Endpoints

Avatar Management

Upload Streaming Avatar - Create a new streaming avatar from a video URL
Get Avatar List - Retrieve a list of all streaming avatars
Get Avatar Detail - Get detailed information about a specific avatar

Session Management

Create Session - Create a new streaming avatar session
Get Session Detail - Retrieve detailed information about a specific session
Close Session - Close an active streaming avatar session
Get Session List - Retrieve a list of all streaming avatar sessions

Live Avatar Stream Message

IAgoraRTCClient.on(event: "stream-message", listener: (uid: UID, pld: Uint8Array) => void)

IAgoraRTCClient.sendStreamMessage(msg: Uint8Array | string, flag: boolean): Promise<void>;

Send Data Chat Type Parameters

Parameter	Type	Required	Value	Description
v	Number	Yes	2	Version of the message
type	String	Yes	chat	Message type for chat interactions
mid	String	Yes		Unique message identifier for conversation tracking
idx	Number	Yes		Sequential index of the message, start from 0
fin	Boolean	Yes		Indicates if this is the final part of the message
pld	Object	Yes		Container for message payload
pld.text	String	Yes		Text content to send to avatar (e.g. “Hello”)

Command Type Parameters

Parameter	Type	Required	Value	Description
v	Number	Yes	2	Protocol version number
type	String	Yes	command	Specifies this is a system command message
mid	String	Yes		Unique ID to track and correlate command messages
pld	Object	Yes		Contains the command details and parameters
pld.cmd	String	Yes		Command action to execute. Valid values: “set-params” (update avatar settings), “interrupt” (stop current avatar response)
pld.data	Object	No		Parameters for the command (required for “set-params”)
pld.data.vid	String	No		Deprecated. Use pld.data.vparams.vid instead. Voice ID to change avatar’s voice. Only used with “set-params”. Get valid IDs from Voice List API
pld.data.vurl	String	No		Deprecated. Use pld.data.vparams.vurl instead. Custom voice model URL. Only used with “set-params”. Get valid URLs from Voice List API
pld.data.lang	String	No		Language code for avatar responses (e.g. “en”, “es”). Only used with “set-params”. Get valid codes from Language List API
pld.data.mode	Number	No		Avatar interaction style. Only used with “set-params”. “1” = Retelling (avatar repeats content), “2” = Dialogue (avatar engages in conversation)
pld.data.bgurl	String	No		URL of background image/video for avatar scene. Only used with “set-params”
pld.data.vparams	Object	No		Voice parameters to use for the session.

Voice Parameters

Parameter	Type	Required	Value	Description
vid	String	No		Voice ID to change avatar’s voice. Only used with “set-params”. Get valid IDs from Voice List API
vurl	String	No		Custom voice model URL. Only used with “set-params”. Get valid URLs from Voice List API
speed	double	No	1	Controls the speed of the generated speech. Values range from 0.8 to 1.2, with 1.0 being the default speed.
stt_language	String	No		Language code for speech-to-text recognition to improve accuracy by using language-specific models
pron_map	Object	No		Pronunciation mapping for custom words. Example: “pron_map”: { “akool” : “ai ku er” }
stt_type	String	No		Speech-to-text type. `"openai_realtime"` = OpenAI Realtime
turn_detection	Object	No		Turn detection configuration.

Turn Detection Configuration

Parameter	Type	Required	Value	Description
type	String	No	”server_vad”	Turn detection type. `"server_vad"` = Server VAD, `"semantic_vad"` = Semantic VAD
threshold	Number	No	0.5	Activation threshold (0 to 1). A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. Available when type is `"server_vad"`.
prefix_padding_ms	Number	No	300	Amount of audio (in milliseconds) to include before the VAD detected speech. Available when type is `"server_vad"`.
silence_duration_ms	Number	No	500	Duration of silence (in milliseconds) to detect speech stop. With shorter values turns will be detected more quickly. Available when type is `"server_vad"`.

JSON Example

{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "text": "Hello"
    },
}

{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "set-params",
      "data": {
        "vid": "1",
        "lang": "en",
        "mode": 1,
        "bgurl": "https://example.com/background.jpg"
      }
    },
}

{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "interrupt"
    },
}

{
    "v": 2,
    "type": "command",
    "pld": {
        "cmd": "set-action",
        "data": {
            "action": "hand_wave", // "hand_wave"|"cheer"|"thumbs_up"|"pump_fists"|"nod"|"shake"|"waft_left"|"waft_right"
            "expression": "happiness" // "angry"|"disgust"|"fear"|"happiness"|"sadness"|"surprise"|"contempt"
        }
    }
}

Receive Data Chat Type Parameters

Parameter	Type	Value	Description
v	Number	2	Version of the message
type	String	chat	Message type for chat interactions
mid	String		Unique message identifier for tracking conversation flow
idx	Number		Sequential index of the message part
fin	Boolean		Indicates if this is the final part of the response
pld	Object		Container for message payload
pld.from	String	”bot” or “user”	Source of the message - “bot” for avatar responses, “user” for speech recognition input
pld.text	String		Text content of the message

Command Type Parameters

Parameter	Type	Value	Description
v	Number	2	Version of the message
type	String	command	Message type for system commands
mid	String		Unique identifier for tracking related messages in a conversation
pld	Object		Container for command payload
pld.cmd	String	”set-params”, “interrupt”	Command to execute: “set-params” to update avatar settings, “interrupt” to stop current response
pld.code	Number	1000	Response code from the server, 1000 indicates success
pld.msg	String		Response message from the server

JSON Example

{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "from": "bot",
      "text": "Hello!  How can I assist you today? "
    }
}

{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "set-params",
      "code": 1000,
      "msg": "OK"
    }
}

Typescript Example

const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});

const msg = JSON.stringify({
  v: 2,
  type: "chat",
  mid: "msg-1723629433573",
  idx: 0,
  fin: true,
  pld: {
      text: "hello"
  },
});

await client.sendStreamMessage(msg, false);

const setAvatarParams = async () => {
  const msg = JSON.stringify({
    v: 2,
    type: 'command',
    pld: {
      cmd: 'set-params',
      params: {
        vid: voiceId,
        lang: language,
        mode: modeType,
      },
    },
  });
  return client.sendStreamMessage(msg, false);
};

const interruptResponse = async () => {
  const msg = JSON.stringify({
    v: 2,
    type: 'command',
    pld: {
      cmd: 'interrupt',
    },
  });
  return client.sendStreamMessage(msg, false);
};

Integrating Your Own LLM Service

Before dispatching a message to the WebSocket, consider executing an HTTP request to your LLM service.

const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});

let inputMessage = 'hello';

try {
  const response = await fetch('https://your-backend-host/api/llm/answer', {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      question: inputMessage,
    }),
  });

  if (response.ok) {
    const result = await response.json();
    inputMessage = result.answer;
  } else {
    console.error("Failed to fetch from backend", response.statusText);
  }
} catch (error) {
  console.error("Error during fetch operation", error);
}

const message = {
  v: 2,
  type: "chat",
  mid: "msg-1723629433573",
  idx: 0,
  fin: true,
  pld: {
      text: inputMessage,
  },
};
client.sendStreamMessage(JSON.stringify(message), false);

Response Code Description

Please note that if the value of the response code is not equal to 1000, the request is failed or wrong

Parameter	Value	Description
code	1000	Success
code	1003	Parameter error or Parameter can not beempty
code	1008	The content you get does not exist
code	1009	Youdo not have permission to operate
code	1101	Invalid authorization or Therequest token has expired
code	1102	Authorization cannot be empty
code	1200	The account has been banned
code	1201	create audio error, pleasetry again later
code	1202	The same video cannot be translated lipSync inthe same language more than 1 times
code	1203	video should be with audio
code	1204	Your video duration is exceed 60s!
code	1205	Create videoerror, please try again later
code	1207	The video you are using exceeds thesize limit allowed by the system by 300M
code	1209	Please upload a videoin another encoding format
code	1210	The video you are using exceeds thevalue allowed by the system by 60fp
code	1211	Create lipsync error, pleasetry again later

​API Endpoints

​Avatar Management

​Session Management

​Live Avatar Stream Message

​Integrating Your Own LLM Service

​Response Code Description

API Endpoints

Avatar Management

Session Management

Live Avatar Stream Message

Integrating Your Own LLM Service

Response Code Description