WebRTC Fundamentals for Voice AI: Real-Time Audio Communication in the Browser

Why WebRTC for Voice AI Agents

WebRTC (Web Real-Time Communication) is a browser-native technology for peer-to-peer audio and video communication. For voice AI agents, WebRTC provides the lowest-latency path for getting audio from a user's microphone to your server and playing synthesized speech back — all without plugins, downloads, or special software.

Unlike WebSocket-based audio streaming, WebRTC handles echo cancellation, noise suppression, automatic gain control, and network adaptation out of the box. These features, which browsers have spent years optimizing, would take months to replicate manually.

Core WebRTC Concepts

RTCPeerConnection

The central object in WebRTC is the RTCPeerConnection. It manages the connection between the browser and a remote peer (in our case, the voice AI server). The connection negotiation follows the offer/answer model using SDP (Session Description Protocol).

flowchart LR
    CALLER(["Caller"])
    subgraph TEL["Telephony"]
        SIP["Twilio SIP and PSTN"]
    end
    subgraph BRAIN["Business AI Agent"]
        STT["Streaming STT<br/>Deepgram or Whisper"]
        NLU{"Intent and<br/>Entity Extraction"}
        TOOLS["Tool Calls"]
        TTS["Streaming TTS<br/>ElevenLabs or Rime"]
    end
    subgraph DATA["Live Data Plane"]
        CRM[("CRM and Notes")]
        CAL[("Calendar and<br/>Schedule")]
        KB[("Knowledge Base<br/>and Policies")]
    end
    subgraph OUT["Outcomes"]
        O1(["Booking captured"])
        O2(["CRM record created"])
        O3(["Human handoff"])
    end
    CALLER --> SIP --> STT --> NLU
    NLU -->|Lookup| TOOLS
    TOOLS <--> CRM
    TOOLS <--> CAL
    TOOLS <--> KB
    NLU --> TTS --> SIP --> CALLER
    NLU -->|Resolved| O1
    NLU -->|Schedule| O2
    NLU -->|Escalate| O3
    style CALLER fill:#f1f5f9,stroke:#64748b,color:#0f172a
    style NLU fill:#4f46e5,stroke:#4338ca,color:#fff
    style O1 fill:#059669,stroke:#047857,color:#fff
    style O2 fill:#0ea5e9,stroke:#0369a1,color:#fff
    style O3 fill:#f59e0b,stroke:#d97706,color:#1f2937

// Client-side: Create peer connection to voice AI server
const pc = new RTCPeerConnection({
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    {
      urls: 'turn:turn.yourserver.com:3478',
      username: 'user',
      credential: 'pass',
    },
  ],
});

// Get user microphone audio
const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
    sampleRate: 16000,
  },
  video: false,
});

// Add audio track to peer connection
stream.getTracks().forEach(track => {
  pc.addTrack(track, stream);
});

// Handle incoming audio from AI agent
pc.ontrack = (event) => {
  const audioEl = document.getElementById('agent-audio');
  audioEl.srcObject = event.streams[0];
  audioEl.play();
};

Signaling: The Offer/Answer Exchange

WebRTC requires an out-of-band signaling channel to exchange connection metadata. Most voice AI implementations use a simple WebSocket or HTTP endpoint for this.

// Client: Create and send offer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// Send offer to server via your signaling channel
const response = await fetch('/api/voice/offer', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    sdp: pc.localDescription.sdp,
    type: pc.localDescription.type,
  }),
});

const answer = await response.json();
await pc.setRemoteDescription(new RTCSessionDescription(answer));

ICE Candidates and NAT Traversal

Most users sit behind NATs and firewalls. ICE (Interactive Connectivity Establishment) finds the best network path between peers using STUN and TURN servers.

STUN servers help discover your public IP address. They are lightweight and free. TURN servers relay media when direct connections fail (about 10-15% of cases). They consume bandwidth and cost money but are essential for reliability.

// Gather and send ICE candidates
pc.onicecandidate = (event) => {
  if (event.candidate) {
    signalingChannel.send(JSON.stringify({
      type: 'ice-candidate',
      candidate: event.candidate,
    }));
  }
};

// Receive ICE candidates from server
signalingChannel.onmessage = (msg) => {
  const data = JSON.parse(msg.data);
  if (data.type === 'ice-candidate') {
    pc.addIceCandidate(new RTCIceCandidate(data.candidate));
  }
};

Server-Side: Handling WebRTC with Python

On the server side, the aiortc library provides a Python WebRTC implementation. This is where you connect the incoming audio to your STT-LLM-TTS pipeline.

from aiohttp import web
from aiortc import RTCPeerConnection, RTCSessionDescription
from aiortc.contrib.media import MediaRelay

relay = MediaRelay()
peer_connections = set()

async def handle_offer(request):
    params = await request.json()
    pc = RTCPeerConnection()
    peer_connections.add(pc)

    @pc.on("track")
    async def on_track(track):
        if track.kind == "audio":
            # Route incoming audio to the voice AI pipeline
            processor = VoiceAgentProcessor(pc)
            relayed = relay.subscribe(track)
            asyncio.ensure_future(processor.process_audio(relayed))

    @pc.on("connectionstatechange")
    async def on_state_change():
        if pc.connectionState in ("failed", "closed"):
            peer_connections.discard(pc)

    # Set remote description (the client's offer)
    offer = RTCSessionDescription(sdp=params["sdp"], type=params["type"])
    await pc.setRemoteDescription(offer)

    # Create answer
    answer = await pc.createAnswer()
    await pc.setLocalDescription(answer)

    return web.json_response({
        "sdp": pc.localDescription.sdp,
        "type": pc.localDescription.type,
    })

app = web.Application()
app.router.add_post("/api/voice/offer", handle_offer)

Audio Processing in the WebRTC Pipeline

Once you have raw audio frames from the WebRTC track, you need to feed them to your STT engine. The audio arrives as PCM frames at the negotiated sample rate.

class VoiceAgentProcessor:
    def __init__(self, pc: RTCPeerConnection):
        self.pc = pc
        self.stt = DeepgramSTT()
        self.llm = LLMProcessor()
        self.tts = TTSProcessor()

    async def process_audio(self, track):
        stt_connection = await self.stt.start_streaming(
            on_transcript=self.handle_transcript
        )

        while True:
            try:
                frame = await track.recv()
                # Convert aiortc AudioFrame to raw bytes
                raw_audio = frame.to_ndarray().tobytes()
                stt_connection.send(raw_audio)
            except Exception:
                break

    async def handle_transcript(self, text, is_final):
        if not is_final:
            return

        # LLM generates response
        response_tokens = self.llm.process_streaming(text)

        # TTS converts to audio and sends back via WebRTC
        async for audio_chunk in self.tts.synthesize_streaming(response_tokens):
            await self.send_audio(audio_chunk)

FAQ

Do I need a TURN server for a production voice AI agent?

Yes. Without a TURN server, roughly 10-15% of users will be unable to connect due to symmetric NATs or strict firewalls. For production, use a hosted TURN service like Twilio Network Traversal or deploy your own with coturn. Budget for TURN bandwidth costs since all relayed audio flows through your TURN server.

Can I use WebSockets instead of WebRTC for voice AI?

You can, but you lose significant benefits. WebRTC provides built-in echo cancellation, noise suppression, automatic gain control, and adaptive bitrate — all handled by the browser's media engine. With WebSockets, you would need to implement these yourself using the Web Audio API, which is complex and less reliable. WebRTC also uses UDP-based transport that handles packet loss more gracefully than TCP-based WebSockets.

How do I handle multiple concurrent voice sessions on the server?

Each RTCPeerConnection is an independent session. Use a session manager that tracks active connections and allocates resources per session. For scaling, run multiple server instances behind a load balancer with sticky sessions (since WebRTC connections are stateful). Each server can typically handle 50-200 concurrent voice sessions depending on hardware and processing requirements.

---

#WebRTC #VoiceAI #RealTimeAudio #BrowserAPIs #STUNTURN #PeerConnection #AgenticAI #LearnAI #AIEngineering