# Realtime protocol and resources Authoritative reference: https://api.latenth.com/docs/realtime-api.html Event fields: https://api.latenth.com/docs/realtime-events.html Event JSON Schema: https://api.latenth.com/docs/realtime.schema.json Management OpenAPI: https://api.latenth.com/docs/admin.openapi.json Custom publishing: https://api.latenth.com/docs/custom-models.html ## Minimal exchange Wait `session.created`, send: ```json {"type":"session.update","session":{"type":"realtime","instructions":"请简短回答。","output_modalities":["text"]}} ``` After `session.updated`, send `conversation.item.create` with `item:{type:"message",role:"user",content:[{type:"input_text",text:"你好"}]}`, then `response.create`. Collect `response.output_text.delta`. Require `response.done.response.status == "completed"`; preserve response ID and `response.usage`. Handle `error`, failed/cancelled response, early socket close and deadline. Audio mode uses `output_modalities:["audio"]` (not text+audio together), `response.output_audio.delta` base64 PCM16LE mono24kHz, and `response.output_audio_transcript.delta`. Input chunks use `input_audio_buffer.append`. With `audio.input.turn_detection:null`, commit and request response explicitly. With server_vad, audio activity need not produce a transcript, interrupt playback or trigger a reply. An accepted turn can start a response when create_response is enabled. Don't send duplicate response.create in automatic mode. Select a voice from the chosen model in `GET /voice/v1/catalog`; use its default_voice when no override is needed. Select `session.audio.output.voice` before first audio; it cannot change afterwards. Cancel generation with `response.cancel`, stop local playback, and truncate assistant history to the actual played offset using `conversation.item.truncate`. ## Metering - If no selector is supplied, read `response.done.response.usage` once per response ID. Keep this default when integrating a client that consumes only responses. - To receive ASR usage independently, use the WebSocket handshake header `X-LatenTX-Usage-Mode: transcription` or URL parameter `usage_mode=transcription`. Check `session.created` → `session.latentx.usage_mode` before sending audio. Do not set this field in session.update. - In that mode, `conversation.item.input_audio_transcription.completed` has top-level `usage: {"type":"duration","seconds":1.25}` (example). Record it once per session and item_id, including when no response follows. Count text input/output and audio output from response usage separately. Never add an estimate for the same audio or count historical copies again. - No completed transcription or response means there is no such public usage event to count. Do not estimate from a microphone timer, WebSocket duration or disconnect. See [examples and units](https://api.latenth.com/docs/realtime-api.html#billing). ## Two kinds of Skill This installed developer Skill guides Codex/Claude. It is **not** automatically available to the voice model. A runtime business Skill is Markdown supplied in `session.latentx.skills` or registered in `/voice/v1/skills`. Do not mount this integration manual as customer-facing business instructions. ```json {"type":"session.update","session":{"latentx":{"skills":[{"id":"support","version":"v1","preload":true,"markdown":"---\nname: support\ndescription: Handle support questions\n---\nAsk for the missing order ID before looking up an order."}]}}} ``` `preload:true` places content in model context. `false` exposes description and read tools for on-demand loading, which may add a model/tool round. The runtime does not execute scripts in Skill folders. `session.latentx` is a LatenTX extension, not an OpenAI field. Inline `markdown` works only in `session.latentx.skills`; a custom model's `config.skills` must reference a saved `{id,version,preload}` version (see management.md). Saved Skill references and `mcp_refs` belong to the current tenant. A custom published model carries reusable configuration; do not create legacy Agent templates for a basic integration. ## Function tools and MCP Function declaration is `{type:"function",name,description,parameters:}` in `session.tools`. Collect `response.function_call_arguments.done`, wait `response.done`, then the customer backend validates arguments and executes. Reply `conversation.item.create` with `{type:"function_call_output",call_id,output:}`, then `response.create`. Return all pending tool results first. Pause the next user audio while tools are pending in this version. Do not use model output as proof an action ran. Remote MCP declaration: ```json {"type":"mcp","server_label":"orders","server_url":"https://YOUR_PUBLIC_HOST/mcp","allowed_tools":["lookup_order"],"require_approval":"always"} ``` Use real server schema and credentials only on backend. The endpoint must be public HTTPS:443; private addresses, redirects, URL query credentials, stdio and legacy SSE are unsupported. Configure Bearer `authorization` only where needed; never print it. Read the live approval lifecycle before implementing MCP. An `mcp_approval_request` requires a matching `mcp_approval_response` conversation item. Wait for response completion and tools to finish before requesting the next model response. Do not auto-approve destructive operations merely to make a demo pass. Use only the user's authorized tools; preserve their existing approval policy. Session updates require an idle response with no pending audio/tools. `instructions` replaces instructions; `tools:[]` clears external tools; `latentx.skills:[]` clears Skills. `tool_choice:"none"` also blocks on-demand Skill reading, but not preloaded text. For exact override and resource limits, use the live reference. ## Knowledge and session isolation Knowledge is a separately managed resource: `session.latentx.knowledge_refs:[{id,version}]`. It mounts the server-executed `search_knowledge` tool; do not define the same name as an external function/MCP, and do not wait to execute it on the customer backend. Only attached knowledge is searched. See [knowledge.md](knowledge.md). `session.update.instructions` replaces the current connection's business prompt; `response.create.response.instructions` replaces it for one response only. Neither saves a model or affects another connection. Changing instructions does not erase conversation history. `tools:[]` clears external tools; clear Skill/Knowledge references separately if needed. `tool_choice:none` disables hosted searches too. ASR/TTS language belongs to custom model configuration or web/phone credentials. Do not send `session.audio.input.language`, `session.audio.output.language`, `session.temperature`, or `response.max_output_tokens`; they are not supported. For custom models preserve the published version's configuration unless the user explicitly wants a session override. A custom model connection emits its default `session.updated` before acknowledging client updates; do not send the first turn twice. ## 会话记录查询 新建 Realtime 会话由服务端自动保存,不需要发送保存事件。使用当前 API Key 调用 `GET /voice/v1/conversations`,来源 `origin=realtime`;详情为 `GET /voice/v1/conversations/{session_id}`,会话事件为 `GET /voice/v1/conversations/{session_id}/events`。只可查询本账号数据。列表仍返回 data 数组,下一页使用末项 cursor;事件分页使用 data.next_cursor。 记录包含用户文本/最终转写、助手生成内容、工具过程、响应状态与用量事件。取消后的部分回答标记 incomplete;音频截断标记 truncated,不能把生成内容当成已播放内容。不保存音频 Base64、凭据或完整系统配置。历史未保存内容无法恢复;进程异常时保留到最近成功提交的记录。 收到 `conversation_storage_unavailable` 表示记录发生异常;异步保存异常也可能体现为历史采集状态不完整,不保证每次都通过实时错误事件通知。会话结束后检查 `capture_state`,不要将未确认提交的文本视为已完整保存。事件分页可能因响应大小提前结束,请以 next_cursor 判断是否还有下一页。 Public session echoes omit platform instructions and internal connection details. Customer instructions are echoed only when confirmed in the effective session; use saved model configuration to inspect your published settings. Error codes are public categories; use platform request/session IDs for support. This does not change audio delivery, function outputs, cancellation or model IDs. ## Session events and recordings New sessions record events by default. Query `/voice/v1/conversations/{id}/events` for every transport. Follow opaque `next_cursor` even when items is empty, and inspect `capture_state`; legacy, partial, unavailable and expired are not complete captures. Complete Realtime history `data` matches the live server event ID, type and content, except audio Base64 deltas. Sending is not proof of client receipt/playback. Refresh after the WebSocket closes to obtain finalized history. Incomplete records are reported as partial, not complete. Query `/recordings` for independently enabled Realtime audio tracks; phone/browser recordings remain in conversation detail.