true while a model response is in flight (generation started and not yet done).
Distinct from IsAudioPlaying: generation runs ahead of playback.
Hosts use this to gate interim narration so it never interrupts a reply.
ProtectedattachAttaches (replacing + closing any previous) the USER-microphone meter. null detaches.
ProtectedattachAttaches (replacing + closing any previous) the AGENT-output meter. null detaches.
has no cancel frame — the client OWNS the audio plane, so cancelling means:
flush the local playout queue (speech stops immediately), mark the in-flight reply
inactive, SUPPRESS residual reply.audio chunks of the cancelled reply until the next
reply boundary, and drain queued sends (a queued narration takes the floor next —
delivery is never dropped by a cancel). No-op when nothing is active.
ProtectedcloseReleases both meters — every driver calls this from its disconnect/teardown path.
Opens the client-direct session: socket to the token-authenticated endpoint, the
server-authored session.update as the first frame, then — once session.ready
confirms the config is applied — the audio plane at the provider's fixed 24 kHz
format. Reports 'listening' only after all of that (obligation #7).
ProtectedcreateCreation seam for the mic-capture pipeline at the provider's fixed 24 kHz rate.
Production delegates to the shared createPcmMicCapture; unit tests override
this with a no-op fake (and may capture onPcmChunk to simulate mic frames).
ProtectedcreateCreation seam for the playout engine at the provider's fixed 24 kHz rate. Production returns the shared RealtimePcmPlayback.
ProtectedcreateCreation seam for the agent websocket. Production wraps the platform-global
WebSocket opened against the token-authenticated URL; unit tests override this to
return an in-memory fake. Handlers are attached by Connect AFTER this returns,
so the implementation must not require them at construction time.
Tears down the socket, mic capture, mic tracks, and playout engine, resets the response
state machine, and emits a final 'closed' (unless already 'error'). Sends
session.end first — without it the provider holds the session (billable) for a
30-second resume window. Safe to call more than once.
ProtectedemitProtectedemitEmits a true barge-in interruption to the registered handler (if any).
ProtectedemitEmits the model/avatar's remote video stream to the registered handler (video drivers only).
ProtectedemitEmits a state change to the registered handler (if any).
ProtectedemitEmits a tool-call request to the registered handler (if any).
ProtectedemitEmits a transcript event to the registered handler (if any).
ProtectedemitEmits a usage update (token deltas for a completed response/turn) to the registered handler (if any).
The session's current audible activity, or null when this driver attached no
meters at all (hosts then keep their turn-state-driven visuals). Sampled by hosts
per animation frame — implementations are cheap, allocation-light reads of an
AnalyserNode; no per-call provider traffic.
OptionalGetReturns the AGENT's remote-audio MediaStream when this driver owns a tappable
remote-audio plane (e.g. a WebRTC peer-connection driver routes the model's audio track
here), or null otherwise. Hosts use it to MIX the agent's voice into a browser-side
recording alongside the mic; a null return (the default, and what every non-WebRTC
driver gives) degrades gracefully to mic-only capture.
Optional capability: the method itself is optional — call sites must use
client.GetRemoteMediaStream?.() ?? null. Drivers that can't expose a remote stream
simply don't implement it (or return null).
Registers the (single) error handler. Receives both fatal transport errors (which are
also followed by an 'error' state change) and non-fatal provider error frames.
Registers the (single) interruption handler.
True barge-in only: fires ONLY when USER INPUT CUT OFF ACTIVE MODEL OUTPUT — a model response in flight or audio audibly playing when the user took the floor. A user simply taking their normal turn while the model is idle is NOT an interruption, and drivers must not report it as one (e.g. a raw "speech started" frame must be gated on whether a response is actually active or audio is playing).
On interruption, drivers must also flush locally-owned playback and report
IsAudioPlaying === false promptly (driver obligation #3). Hosts typically
use this hook to abort in-flight delegated work so a stale result is never narrated
into a conversation that has moved on.
OptionalOnOptional capability: registers a handler invoked when the agent's remote-audio stream
becomes available — immediately if it has already landed, otherwise when the WebRTC track
arrives (typically AFTER Connect resolves). Lets a host attach the agent voice to a
recording that began before the track landed. Call as client.OnRemoteMediaStream?.(cb);
drivers without a remote stream simply don't implement it.
Registers the (single) remote-VIDEO handler — the model/avatar's video track for a VIDEO session (a talking-head the host renders, e.g. as the agent's tile). Invoked once the provider publishes its video track.
Optional capability: audio-only drivers (the default) never emit — registering is always safe, but hosts must not assume a video track arrives. Video-capable drivers (BaseRealtimeModel.SupportsVideo) call emitRemoteVideo when the track is live.
Invoked with the remote video MediaStream when it becomes available.
Registers the (single) state-change handler. States may be re-emitted (e.g. repeated
'listening' on successive barge-ins); hosts should treat emissions as idempotent.
Registers the (single) tool-call handler. The host executes the tool and calls SendToolResult with the outcome.
Registers the (single) transcript handler. Receives interim deltas (IsFinal: false)
and final turn transcripts (IsFinal: true) for both roles, tagged with their
RealtimeClientTranscript.Kind.
Registers the (single) usage handler.
Emissions carry token deltas for the response/turn that just completed (see
RealtimeClientUsage — deltas preferred; cumulative-only providers must convert
in the driver). Hosts accumulate and relay/persist on their own cadence (e.g. the voice
session service debounces a RelayRealtimeUsage mutation onto the co-agent prompt run).
Optional capability: drivers whose provider exposes no usage telemetry simply never emit — registering a handler is always safe, but hosts must not assume emissions arrive. See RealtimeClientUsage for per-driver availability.
Triggers ONE short spoken update — NATIVE via reply.create with per-response
instructions. The resulting reply is stamped Kind: 'narration' at send time (reset
on the reply boundary). Queued behind any in-flight reply per the base contract's
collision rule (an overlapping reply.create would collide with active generation).
Injects background context — EMULATED via the mutable system_prompt: the note is
appended under a "Background updates" heading and the FULL prompt is re-sent via
session.update. A config write never triggers or disturbs generation, so it is sent
immediately even while a reply is in flight (no queueing applies).
Injects typed text — EMULATED via reply.create (the protocol accepts no typed user
input), with instructions framing the text as the user's turn. No-op when the session
is not open. See the class-level fidelity caveat.
SendText implies barge-in (base-contract rule): an active spoken reply is cancelled via CancelActiveResponse first — playback flushed, reply marked inactive, queued sends drained — so the typed turn takes the floor immediately. If a drained queued send starts a new reply, the text queues behind it, preserving delivery order.
Feeds an executed tool's result back via tool.result, correlated by the provider's
call_id. EXACTLY-ONCE: the pending id is consumed when the result is accepted, and a
duplicate (or unknown) callID is dropped with a warning. Sent immediately when idle —
the provider speaks the result as the turn's continuation (no explicit generation
trigger exists or is needed) — otherwise queued behind the in-flight reply (e.g. a
progress narration) so the trigger is never lost and the narration's kind tagging is
not clobbered. The wire result slot expects a JSON-STRING, which is exactly the
contract's outputJson shape — it passes through verbatim.
Mutes / unmutes by toggling the mic tracks' enabled flag: the capture pipeline stays
up and streams SILENCE while muted (the provider's VAD sees a continuous stream and the
un-mute is glitch-free — same policy as the other client drivers).
AssemblyAI Voice Agent implementation of BaseRealtimeClient: a browser-direct agent websocket authenticated with the server-minted ONE-TIME client token (the
EphemeralToken— no API key ever reaches the browser).Registered with the ClassFactory under the key
'assemblyai'— theProviderstring the server'sAssemblyAIRealtimedriver stamps on itsClientRealtimeSessionConfig.Owns ALL AssemblyAI wire concerns (the behavioral sibling of the ElevenLabs client driver — the audio plane is client-owned over a websocket, no WebRTC):
wss://…/v1/ws?token=…→ send the server-authoredsession.update(from theSessionConfigpact: system prompt, tools, voice, turn detection) as the FIRST frame → wait forsession.ready→ build the audio plane at the provider's FIXED 24 kHz PCM16 format →'listening'. The state is gated onsession.ready(driver obligation #7): only then has the provider confirmed the session config is applied (audio sent earlier would be dropped anyway).input.audioframes.reply.audiochunks decoded into the shared RealtimePcmPlayback at 24 kHz; IsAudioPlaying comes from its playout clock.transcript.user.deltafragments (IsFinal: falsedeltas) finalized bytranscript.user; agent turns arrive as ONE finaltranscript.agent(after a barge-in it carriesinterrupted: trueand the text already reflects the truncated turn — no ElevenLabs-style correction event follows).IsBusyis set onreply.started/ firstreply.audio(and eagerly when this client triggers a reply) and cleared onreply.doneand on atool.callframe (the agent has yielded the floor pending the result — deadlock guard, obligation #2).reply.createwith the instructions (a real per-response-instructions channel — no user-turn emulation). The response kind is stamped'narration'at send time (sends are the turn triggers) and reset on the reply boundary; the kind tagging assumestranscript.agentprecedes itsreply.done, which is the provider's emission order.reply.create: the protocol accepts no typed user input, so the typed text rides as response instructions framing it as the user's turn. Fidelity caveat: the text enters as instructions rather than a conversation user turn, so the agent may occasionally reference "the message you sent" — hosts should treat typed input on this provider as best-effort.system_prompt: notes are appended under a "Background updates" heading and the full prompt is re-sent viasession.update— a config write that never triggers or disturbs generation, so it is sent immediately even mid-reply.input.speech.startedwhile a reply is active (or audio audibly playing) is the snappy flush point (per the provider's own guidance — ~300 ms faster than waiting forreply.done);reply.donewithstatus: 'interrupted'is the authoritative verdict and is the fallback flush when the speech-start gate missed. A speech start while idle is a user simply taking their turn — NOT an interruption (base contract).reply.audioframes of the cancelled reply until the next reply boundary — so late-arriving chunks of a cancelled turn are never played or allowed to re-assert'speaking'.