Production Healthcare Voice AI: Multi-Agent Orchestration, Evaluation & Latency
One-line summary: Built multi-agent orchestration for healthcare voice-AI workflows, pairing flexible model behavior with deterministic state, routing controls, evaluation, and observability.
Key Results
- Reduced first-turn voice-agent latency from approximately 14 seconds to 5 seconds in test flows.
- Implemented multi-agent routing with deterministic recovery paths for incomplete or inconsistent model decisions.
- Developed LLM-simulated-patient evaluations and added threshold-based regression gates to existing nightly pipelines.
Problem / Context
Voice interactions make the weaknesses of a sequential AI system immediately visible: initialization and routing time become awkward silence, while a malformed model decision can send a conversation down the wrong path. The orchestration layer needed to coordinate specialized healthcare workflows while keeping critical application behavior explicit and recoverable.
What I Built
- Multi-agent orchestration across patient-verification, appointment, and medication-related workflows.
- Parallel primary-agent and background-agent execution so the current task and subsequent routing work could progress concurrently.
- Persisted session state for continuity across turns and workflow transitions.
- Deterministic fallback behavior for missing, incomplete, or inconsistent routing output.
Reliability Around Probabilistic Models
Model-driven routing provides useful flexibility, but critical invariants should not depend on a perfectly formed response every time. I added explicit application-level validation and deterministic fallbacks around probabilistic decisions, allowing the system to recover gracefully while preserving the agent's ability to reason about the conversation.
Evaluation
I developed orchestrator evaluations using LLM-simulated patients with different personas and difficulty profiles. The evaluation covered task completion and conversational quality through LLM-as-judge scoring, then applied threshold-based gates in existing nightly pipelines to surface behavioral regressions. The work extended the established pipeline rather than replacing it.
Latency
Sequential session initialization added avoidable time to the first response. By prewarming primary and background sessions concurrently and overlapping preparation with routing work, I reduced first-turn latency from approximately 14 seconds to 5 seconds in test flows. The metric reflects those test flows, not a universal production guarantee.
Observability
The voice-AI path crossed multiple services, so a single service's timing did not explain end-to-end delay. I built tooling to correlate logs across that path and separate time spent in agent execution, orchestration, forwarding, and response generation or delivery.
Key Insight
Production agent systems require more than good model responses. Reliability comes from combining probabilistic reasoning with deterministic state, explicit invariants, evaluation, observability, and carefully designed fallbacks.
Technical Themes / Tools
Python, Java, Amazon Bedrock, LangGraph, ReAct-style agents, DynamoDB, persisted session state, LLM evaluation, distributed systems, latency analysis, and observability.