How AI Voice Agents Actually Work in Neo Dialer

Most descriptions of AI voice agents stop at "it uses AI to talk to people", which tells you nothing about whether it will work on your calls. Here is what the pipeline in Neo Dialer actually does, and which parts matter.

The pipeline

Three models sit in a loop for the duration of the call:

Caller audio
   │
   ▼
Speech to text   — Deepgram, or Groq Whisper
   │
   ▼
Language model   — OpenAI, or Groq   ──▶ tools: look up, check, book, transfer
   │
   ▼
Text to speech   — ElevenLabs, Cartesia, or self-hosted Piper
   │
   ▼
Caller hears the reply

Nothing about that arrangement is unusual. What decides whether it feels like a conversation is everything around it.

Where it runs matters more than which model you pick

The pipeline runs on the media server, beside Asterisk, sharing its network namespace so audio reaches the speech model over the loopback interface.

The alternative — running the AI runtime somewhere else and shipping RTP to it — adds a network hop in each direction on every turn. On a phone call, latency you would not notice in a chat interface is the difference between a conversation and an interrogation.

There is a second reason, and it is about credentials. Media servers are internet-facing SIP and RTP hosts. They never hold your integration credentials. When an agent needs to write to your calendar, it calls back to the application over an HMAC-signed channel and the application performs the write with credentials only it holds. The methods it can invoke are allowlisted, and the tenant is resolved on the application side rather than taken from the request.

Turn-taking is the hard part

Three things separate an agent people will talk to from one they hang up on:

Barge-in. The caller can interrupt and the agent stops. Without it, anyone who tries to correct the agent mid-sentence ends up talking over a machine that will not yield — which is where most people hang up.

Silence detection. The agent has to know when you have finished, not merely when you have paused. Too eager and it talks over you; too patient and the call goes dead. Voice activity detection runs on the media path to separate speech from line noise.

Filler injection. When the language model takes a moment, brief natural fillers cover the gap. Silence on a phone call reads as a dropped call, and callers start saying "hello? hello?" — at which point the transcript is a mess and so is the conversation.

Tools, and the confirmation gate

An agent that can only talk is a recording with extra steps. Ours can look up a caller by phone or email within your tenant, check calendar availability, book an appointment, write to a sheet, schedule a callback, and transfer to a human.

Booking is gated: the agent cannot book without an explicit confirmation turn first. That exists because a language model that is agreeable by default will otherwise book something the caller was still thinking about.

Failing honestly

This is the part we would rather be judged on than the model list.

If the calendar integration is unavailable, the agent reports the failure rather than claiming success. An agent that cheerfully confirms an appointment that was never created is worse than one that says it could not book — the first produces a customer who turns up to nothing.

Bookings the agent could not complete land in a Pending Bookings queue for a human to pick up, so a failed integration produces a task rather than a silent loss.

What you get back

Every AI call produces a transcript, an outcome, a summary, and a cost breakdown across speech, language and voice usage. It sits on the same call record as recordings from human calls, so reporting does not fork.

And if you do not want any of it

The AI is per-tenant and optional. Run Neo Dialer as a straight human-agent dialer and none of these providers is contacted at all. If you want AI speech but not an external vendor, Piper runs on our own infrastructure.

More detail in the AI voice agent documentation.