Realtime Session Capture & Recording Guide

How MemberJunction captures a realtime (voice) session: per-turn transcript with start/end timing and speaker identity, and an optional stored audio recording that an evidence player can seek against. Read this before touching persistRealtimeTranscript, the RealtimeSessionRunner recording seam, or the RealtimeRecordingController.

TL;DR

• Every realtime turn is persisted as a MJ: Conversation Details row with a create-on-start / update-on-complete lifecycle: an In-Progress row is created on the first interim transcript and updated to Complete on the final transcript. This gives each turn a real start (__mj_CreatedAt) and an immutable end (TurnEndedAt) — without relying on __mj_UpdatedAt (which moves on later edits).
• Speaker identity reuses the existing ConversationDetail.UserID (set on user turns only; an AI turn has no human speaker). No separate speaker column.
• When recording is active, each turn also carries media-relative offsets UtteranceStartMs / UtteranceEndMs (milliseconds from the recording t0) and MediaType='Audio'.
• Audio is mixed to a mono PCM16 WAV (a seekable RIFF/WAVE container — its header carries the exact duration and maps byte offset → time linearly, so HTTP-range seeking is exact) and stored via MJStorage; the session row holds RecordingFileID (→ MJ: Files), RecordingMedia, and RecordingStartedAt (the t0). Capture happens on both topologies: server-side (server-bridged) and in the browser (client-direct).
• A capture-time waveform (normalized 0..1 max-abs peaks) is computed cheaply during recording and written as a peaks.json sidecar in the same storage folder as the WAV, so a player renders the waveform instantly without re-decoding the audio.
• Playback streams the recording over HTTP Range via mj-storage-media-player → the CreateMediaAccessToken mutation → the GET /media/:fileId?token= route (not a base64 download).
• Recording is OFF by default, resolved runtime param > agent (RecordingDefault) > off, consent-gated, and fail-closed (no storage provider or no consent ⇒ no capture).

The pieces

Concern	Where
Turn lifecycle persistence	BaseAgent.persistRealtimeTranscript (packages/AI/Agents/src/base-agent.ts)
Recording resolution + finalize (server-bridged)	BaseAgent.resolveRealtimeRecording / finalizeRealtimeRecording
Server-side audio accumulation + WAV encode	RealtimeRecordingController (packages/AI/Agents/src/realtime/realtime-recording-capture.ts)
Recording store + peaks.json sidecar write	realtime-recording-store.ts (storeRealtimeRecording / writeRecordingPeaksSidecar)
Session attach + finalize seam	RealtimeSessionRunner (Recording / FinalizeRecording deps)
Client-direct browser capture	RealtimeAudioRecorder + encodePcm16Wav / PeakAccumulator (packages/Angular/Generic/conversations/src/lib/services/realtime-audio-recorder.ts, realtime-pcm-wav.ts)
Streaming playback (server)	CreateMediaAccessToken mutation + GET /media/:fileId route (packages/MJServer/src/rest/MediaStreamHandler.ts, peaksSidecar.ts)
Streaming playback (client)	mj-storage-media-player (@memberjunction/ng-media-player)
Schema	ConversationDetail.{TurnEndedAt,UtteranceStartMs,UtteranceEndMs,MediaType}; AIAgentSession.{RecordingMedia,RecordingStartedAt,RecordingFileID}; AIAgent.{RecordingDefault,RecordingStorageProviderID}

Turn lifecycle (always on, recording or not)

persistRealtimeTranscript(params, transcript) returns the new row id when a distinct turn is first created, and null when an existing in-flight row is merely updated — the runner counts the former so TranscriptTurnCount reflects turns, not the many interim+final events a turn emits.

• Interim (IsFinal=false): on the first delta for a role, create the row (Status='In-Progress', Message=delta, UserID for user turns, UtteranceStartMs if recording). Later deltas are no-ops.
• Final (IsFinal=true): update that in-flight row to Status='Complete', full Message, TurnEndedAt, UtteranceEndMs. If no interim was seen (some providers only emit final), the row is created and finalized in one step.

Consumer note: mid-turn In-Progress rows now exist transiently. UIs reading ConversationDetail should honor Status/HiddenToUser (most streaming chat surfaces already do).

Recording config resolution (per session)

resolveRealtimeRecording decides whether to record, fail-closed:

1. Media = runtime params.data.recording.media ▸ AIAgent.RecordingDefault ▸ None. None ⇒ off.
2. Consent must be explicit params.data.recording.consent === true; otherwise off (logged).
3. Storage = AIAgent.RecordingStorageProviderID ?? AIAgent.AttachmentStorageProviderID, resolved to that provider’s first account via FileStorageEngine.GetAccountsByProviderID. No account ⇒ off.

Only when all three pass is a RealtimeRecordingController created. Finalize (finalizeRealtimeRecording) encodes the WAV, UploadFiles it to the resolved account, writes the peaks.json sidecar beside it, creates a MJ: File Entity Record Links row (EntityID = MJ: AI Agent Sessions, RecordID = session id), and stamps RecordingFileID/Media/StartedAt on the session. Any failure is logged, never thrown — a recording problem must not fail the session.

Topology: where audio is captured

Recording works on both topologies; the difference is only where the audio is mixed.

Server-bridged sessions (LiveKit/Zoom/Teams/Twilio): the server-side IRealtimeSession carries the media (OnOutput = model speech; SendInput = inbound), so the RealtimeSessionRunner attaches the RealtimeRecordingController there — it taps OnOutput, wraps SendInput, mixes to a mono PCM16 WAV, computes capture-time peaks, and finalizes on Stop().

Client-direct browser sessions negotiate audio in the browser — the server never sees those frames — so capture happens in the browser via RealtimeAudioRecorder (@memberjunction/ng-conversations):

• It mixes the user’s microphone with the agent’s remote-audio stream through the Web Audio API and captures Float32 PCM frames off an AudioWorkletNode (preferred) or ScriptProcessorNode (fallback) — not a MediaRecorder (whose webm/opus output is header-less, so range seeking is unreliable). At Stop() the accumulated PCM is encoded to a seekable 16-bit PCM WAV via encodePcm16Wav.
• A bounded PeakAccumulator computes the waveform peaks during capture (cost stays O(buckets) regardless of call length); these become the peaks.json sidecar on upload.
• Agent-audio mixing: the agent’s WebRTC track usually lands after recording has already begun (mic-only). The recorder exposes AttachRemoteStream(stream), and the session service subscribes to the client driver’s OnRemoteMediaStream(cb) hook (BaseRealtimeClient / OpenAIRealtimeClient) so the agent stream is wired into the live mix the moment it arrives — capturing both sides, not just the mic. The recorder degrades gracefully (no AudioContext / no worklet+scriptprocessor / no remote stream ⇒ disable or mic-only) so a recording problem never blocks the live call.

Playback: streaming the recording

Playback streams the stored WAV over HTTP Range rather than downloading it. The mj-storage-media-player component (@memberjunction/ng-media-player) takes the recording’s RecordingFileID and:

1. Calls the CreateMediaAccessToken mutation, which runs a per-user permission check, mints a short-lived signed token, returns the authenticated GET /media/:fileId?token= streaming URL, and — when the peaks.json sidecar exists — returns the precomputed waveform peaks.
2. Hands the URL (and any peaks) to the generic mj-media-player, whose <audio> element streams the WAV natively over HTTP Range via the /media route (re-verifies the token, then range-streams from the storage provider through FileStorageBase.GetObjectStream). Supplied peaks render the real waveform instantly with no client-side decode.

The in-conversation session-review overlay (realtime-session-overlay) embeds mj-storage-media-player directly, passing the recording file id + the ConversationDetail transcript cues for click-to-seek + active-turn highlight. (The older RealtimeEvidencePlaybackComponent remains a generic, input-driven time-aligned player for callers that already hold an audio URL, falling back to __mj_CreatedAt − RecordingStartedAt when a turn has no precise UtteranceStartMs.)

The Media channel (showing media during a call)

Separate from capture: RealtimeMediaChannel (client) + MediaChannelServer (server, a state validator) let the agent show images/video/audio/PDF/web embeds during a conversation via Media_* tools, rendered as tabs. It reuses MJ: Files + the channel Config state — no new entity. Registered via the MJ: AI Agent Channels metadata row (ServerPluginClass='MediaChannelServer', ClientPluginClass='RealtimeMediaChannel'); LoadMediaChannelServer() is called from MJServer’s agentSessions static path to survive tree-shaking.

Showing an MJStorage file (not just a public URL)

Media_ShowMedia accepts either a public url or an MJ fileId (one of the two is required; neither ⇒ the tool rejects). A fileId points at an MJ: Files row, so the agent can put a permission-gated, MJStorage-backed file on the surface without ever exposing a public URL. MediaItem carries the optional FileID alongside Url, and the pure RouteForMediaItem helper centralizes the render decision per item:

Item	Route	Rendered by
audio / video with a FileID	storage-player	<mj-storage-media-player [FileID]> — HTTP-range, permission-gated, full transport + waveform
audio / video with only a Url	url-player	<mj-media-player> (replaces the old raw <audio>/<video>)
image / pdf / web (either source)	iframe-or-img	a CreateMediaAccessToken streaming URL for a FileID, or the Url directly, with loading + no-access states

FileID wins over Url for audio/video (secure streaming is preferred). A file-backed item streams the same way recordings do — through CreateMediaAccessToken → the GET /media/:fileId?token= route — so it inherits the per-user permission check and HTTP-range streaming for free; there is no base64 download and the model never sees a raw URL for a fileId item.

Because the storage player and the token mutation run server calls, the channel threads the live session Provider through RealtimeChannelContext into the surface (BindSurface), so playback and token-minting run under the session’s authenticated provider — multi-provider safe. When the context’s Provider is null, the surface falls back to the global default.

Meeting-Room recording (LiveKit egress → MJStorage)

Status: BUILT. The full loop — egress → register the MP4 as an MJ: Files row → stamp the Conversation → play it back in the Meet app — is implemented and unit-tested. The one deployment requirement is the storage config (MJ_MEETING_RECORDING_STORAGE_PROVIDER, below); without it the recording still stops, it just isn’t registered as a Files row.

The per-session recording above captures one agent session’s audio. A LiveKit meeting is different: it maps 1:1 to a Meeting-Room Conversation (ApplicationScope='Application', so it stays out of normal chat), and a single meeting can span multiple agent sessions. The room-level composite recording — one MP4 that LiveKit egress produces for the whole room — therefore belongs on the room (the Conversation), not on any single AIAgentSession (which keeps its own per-agent RecordingFileID).

• Schema (additive + nullable): Conversation.RecordingFileID (FK → MJ: Files, the composite egress MP4) and Conversation.EgressID (the LiveKit egress session id — set when recording starts, used to stop the egress and correlate its completion result back to the conversation).
• Egress service (LiveKitEgressService, packages/LiveKitRoomServer/src/livekit-egress-service.ts): StartRoomRecording starts a composite MP4 egress and returns its EgressID; on stop/complete, RecordingInfo surfaces the egress’s OutputLocation (the file’s path/key in the egress sink), OutputSizeBytes, and OutputDurationMs (normalized from the SDK’s nanosecond duration), extracted from EgressInfo.fileResults[0]. While a recording is still in progress (no fileResults yet), those three fields are undefined.
• Registration (packages/MJServer/src/resolvers/meetingRecordingRegistration.ts, invoked from RealtimeBridgeResolver):
  • On StartLiveKitRecording — best-effort correlateRecordingStart(roomName, egressID) stamps Conversation.EgressID onto the room’s Meeting-Room Conversation if it exists, so a live recording is correlated. If the conversation doesn’t exist yet, it’s skipped silently (the stop-flow resolves/creates it). The recording start never fails on this.
  • On StopLiveKitRecording — after stopping egress, registerMeetingRecordingFile(...):
    1. Resolves the Meeting-Room Conversation — prefer match by EgressID, fall back to room name (ExternalID + Type='Meeting Room'), else create one (scoped Application, mirroring the transcript sink).
    2. Resolves the storage account — the MJStorage account linked to the configured provider (see config below) whose sink LiveKit egress wrote to.
    3. Creates the MJ: Files row — ProviderID = that provider, ProviderKey = the egress OutputLocation, ContentType='video/mp4', Status='Uploaded', Name='Meeting Recording — <Room> — <date>'. v1 points the Files row DIRECTLY at the egress output — no byte copy — so playback streams straight from the sink.
    4. Stamps the Conversation — RecordingFileID (+ EgressID if unset) and saves.
    5. Returns the new file id, which StopLiveKitRecording surfaces as LiveKitRecordingResult.RecordingFileID.
  • All best-effort — wrapped in try/catch, returns a non-throwing failure result with a clear message (missing config, no account, save failure) and LogErrors. A failed registration never crashes the stop-recording mutation; the recording still stopped.

Required storage configuration

The egress sink and the MJStorage account must point at the same bucket/container so MJ can read the file LiveKit wrote. Configure the provider whose accounts target the sink:

• env: MJ_MEETING_RECORDING_STORAGE_PROVIDER=<MJ: File Storage Providers ID>
• config (mj.config.cjs): meetingRecordingStorageProviderID: '<provider id>'

When unset, registration returns a clear, non-throwing failure (the recording stops, but isn’t registered).

Point-at-sink (default) vs. copy-to-canonical (“copy into Box”)

• Default (point-at-sink) — the Files row’s ProviderKey IS the egress OutputLocation in the sink provider. No bytes are moved; playback streams from the sink. Use when the egress sink (S3/Azure/MinIO/GCS) is itself an MJStorage-readable provider.
• Optional (copy-to-canonical) — set a different canonical provider via MJ_MEETING_RECORDING_CANONICAL_STORAGE_PROVIDER (or meetingRecordingCanonicalStorageProviderID). When set and different from the sink, copyEgressOutputToCanonical(...) reads the bytes via the sink driver’s GetObject({ fullPath }) and re-uploads them into the canonical provider (e.g. Box) via FileStorageEngine.UploadFile, then points the Files row there. OFF by default.

Meet app playback

The Meet UI (mj-livekit-agent-room, packages/Angular/Generic/mj-livekit-room/) renders a dismissable “Meeting recording” panel with <mj-storage-media-player [FileID]="recordingFileId" [Provider]="Provider"> whenever the room has a recording. recordingFileId is set two ways: (a) when the user stops a recording, StopRecording returns the freshly-registered RecordingFileID; (b) on join/start, the component resolves any prior recording for the room from its Meeting-Room Conversation (ExternalID + Type='Meeting Room', RecordingFileID IS NOT NULL). Playback streams over HTTP Range via the same mj-storage-media-player pipeline used for per-session recordings.