Three parts, the way you asked for them. BUILD is the architecture I would actually start tomorrow. DECIDE is the one question worth your attention as founder, and why I think the obvious question is the wrong one. SHIP is what one engineer can really have running by Day 30. Everything below is labeled and sourced, and I left the gaps as open questions instead of papering over them.
Cascade Aria runs the cascade I defend below: Qwen 3.6, expressive voice with delivery tags, a grounding knowledge base, crisis procedures, and guardrails. Native Aria runs ElevenLabs' own native realtime model (v1 Max) end to end. Same persona, two architectures. Try "show me your range" on Cascade Aria, interrupt either one mid sentence, or stop talking for eight seconds and watch the silence handling.
The pipeline I defend in this document. Long pauses, barge in, delivery tags like [chuckles], and the safety stack around it.
The native path as a live pilot: one model, audio in and audio out, so you can feel the difference yourself.
ElevenLabs Agents in Custom voice mode. Scribe v2 Realtime for speech to text, turn_v3 endpointing in patient mode, Qwen 3.6 as the conversational model at low reasoning effort, Eleven v3 Conversational with Expressive Mode for the voice. React Native and Expo for iOS and Android, Next.js for the web client, FastAPI and Postgres behind it, Langfuse for tracing and secondary evals.
One pipeline, every stage behind a config seam, so nothing in this stack is a trap.
Full choice, why, and when to replace it in section 2.
The backbone question, cascade versus native speech to speech, is not the decision that matters. Latency has already reached parity and an abstraction layer keeps the choice reversible.
What I would want your call on this week is different: the crisis escalation posture, whether we serve minors, and the line between companion and therapy in everything we say out loud. Those are founder decisions because they define the product's legal and moral identity.
Evidence and trade offs in section 4.
Only what is shipped and testable, section 5.
This is the whole system. The client is the product you described, iOS and Android with real time full duplex voice. The heart is the conversational pipeline, and the two blocks under it are what make it a mental health product rather than a voice demo: safety tools on one side, memory on the other. The green line is the audio going home.
The format is the one you asked for. My pick first, then why, then the trigger that would make me swap it out. If a row has no honest replacement trigger, it is a bet, not a decision.
| Layer | My pick | Why | When I would replace it |
|---|---|---|---|
| Client | React Native + Expo, Next.js web | One codebase for iOS and Android, and I can share the API layer with the web client. This is also what I proposed to you earlier and I stand behind it. | Native modules if WebRTC stability or thermal behavior on long calls forces my hand. |
| Transport | ElevenLabs managed WebRTC | Zero transport ops on Day 1. I ran this exact transport live in production on my interview platform for months and I know its failure modes. | Data residency or self hosting requirements (LiveKit Cloud or OSS), or scale economics flipping the math in section 6. |
| Turn taking | turn_v3, patient, speculative | It is the only option with an independent duplex behavior score today, 95.8% on Full Duplex Bench. Patient mode fits therapeutic pauses. Soft timeout fillers kill dead air without false interruptions. | My own long pause harness showing false cutoffs above 5% at the patience level users tolerate. |
| Speech to text | Scribe v2 Realtime | Bundled in the platform minute, and it captures non verbal events like laughs and pauses, which carry emotional context. | At scale, AssemblyAI bills session time including idle at about a third of the streamed rate, which matters when people sit in silence. Or medical vocabulary, where Scribe Medical exists. |
| Conversational model | Qwen 3.6 (qwen36-35b-a3b), effort low | Co located inside the platform pipeline, no cross provider tail latency. Non deprecated as of my 2026-11-01 catalog check. Your report of Gemini 3 and 3.1 hiccups is documented in real bug trails, and Speko measured 63.1% dead air turns on gemini-3.5-flash-lite. | Silent turn rate above 2% over a rolling week, or p95 first audio above 1.2 s, and I move to Claude Haiku, the only model measured with zero dead air and zero tool silence. |
| Voice | Eleven v3 Conversational + Expressive Mode | It is the only off the shelf voice with explicit emotional control plus emotional continuity across turns. For an empathic companion that is the product. | Cost at scale, or if I need prosody guarantees nobody sells: fine tuning a self hosted Kokoro class model, which is also the 40x cost lever. |
| Memory | Rolling summary + typed fact store | Steady state context stays around 2 to 5k tokens at any session length, and the hard delete path for GDPR lives in tables I own. | When cross session recall needs temporal provenance, I add a temporal knowledge graph, but only after building the erasure overlay first. |
| Safety | Four layers, not one model | Streaming triage as a structured procedure, async classifier over the transcript, post call evals, human SOP. No single classifier is trustworthy, Llama Guard recall is 69%. | Classifier models swap freely behind the tool seam. The escalation policy is yours to change, not mine. |
| Observability | Langfuse + platform evals + blind listening | Traces stay vendor neutral because I plan to swap stages. Voice quality claims only survive if people actually listen. | Add LiveKit simulations if I ever self assemble the stack. |
What I decided not to build in the first 30 days: multi listener pools, telephony, voice cloning, and any fine tuning. Fine tuning starts after the beta harvests consented data, because training on user conversations without a designed consent path is exactly the trap that sank other products in this space.
You asked for the single most important technical decision or uncertainty. I want to be direct about my answer, because it is a small twist on the expected one.
| Axis | Streaming cascade (my foundation) | Native speech to speech (my pilot lane) |
|---|---|---|
| Latency, measured | 755 ms best cascade, time to first audio (arXiv 2603.05413v2, 2026-03, independent) | 494 ms p50, gpt-realtime conversational latency (Speko live harness, 2026-07-08). Eleven Realtime at 570 to 620 ms TTFA (Artificial Analysis, 2026-10) |
| Interruptions and silence | Configurable, and patient mode fits long emotional pauses | Implicit but aggressive. Moshi takes the floor on 98% of user pauses and talks through interruptions (Full Duplex Bench, Talking Turns at ICLR 2025) |
| Transcripts for safety and memory | Structural. Every stage produces text, which is why ElevenLabs' own realtime product is still cascaded under the hood | Separate ASR with documented mismatch errors. Long session recall falls to 25 to 50% with 17 to 42% hallucinated transcripts. Moshi has no user transcript at all |
| Emotional control | Explicit. Expressive Mode, delivery rules, tags, and tone that carries across turns | Implicit prosody, no direction API |
| Cost per 60 minute session | $4.97 managed, $2.04 self assembled | About $5 with cached audio tokens, and about $115 without, because audio context gets re read every turn |
| Lock in | Low. Custom mode keeps every stage swappable | Model API lock, softened by realtime compatible endpoints |
Now the part I would want your call on. These three are yours, not mine, and this week is not too early:
| Decision | Why it is yours, today |
|---|---|
| Crisis escalation posture | California SB 243 and the New York companion law require a published self harm protocol, with a private right of action attached. The Garcia settlement showed disclaimers do not protect anyone. What we promise a user alone at two in the morning is a product and moral decision. |
| Minors, yes or no | The amended COPPA rule treats voiceprints as personal information. Character.AI removed open ended minor chat entirely and 40 state attorneys general still called their voluntary safeguards circumventable. |
| Companion, never therapy | Illinois, Nevada, Rhode Island and Vermont ban unlicensed AI psychotherapy. The words in our marketing copy are evidence in a courtroom, so the boundary has to be drawn before launch copy exists. |
| Session length reality check | One platform constraint I found while building the demo: the self hosted speech engine path caps sessions at 40 minutes without a sales conversation. If 60 minute sessions are contractual for us, that is a negotiation, not an engineering problem. |
And the strongest argument against my own recommendation, because you should hear it from me first. The counter argument to any flexibility play is that a solo engineer maintaining optionality burns the runway that should go to safety and emotion work. It is a fair hit. My answer is that the optionality here is one eval harness and one config seam, built in Week 1, not a permanent second stack to maintain. The kill criteria in section 5 exist so the architecture can be swapped by measurement, not by taste.
Two sequences. The top half is an ordinary turn with a barge in and a long pause, the behaviors that decide whether talking to this thing feels human. The bottom half is the situation the whole product exists for: it is late at night, the user is alone, and at some point they say something that means they might hurt themselves. I call it the mid-night case throughout, and how the system behaves in it is the first thing I would want to be challenged on.
Only what is shipped and testable. If I cannot demonstrate it on the milestone day, it goes to the next milestone or it gets cut.
What genuinely cannot be shortcut, with honest durations: safety review and the published crisis protocol (1 to 2 weeks), consent and the GDPR DPIA (3 to 7 days), the on call SOP (2 to 3 days plus drills), and app store review once mobile ships (1 to 3 weeks, unpredictable). Anything else is negotiable under pressure. These four are not.
Cost per 60 minute conversation, three complete stacks. Factors sourced, arithmetic shown, and the unknowns flagged as unknowns.
| Line | Managed platform (Custom mode) | Self assembled cascade | Native speech to speech |
|---|---|---|---|
| Platform or transport | 60 min at $0.08 = $4.80. Silence beyond 10 s bills at 5%, which helps a product where people pause, and I did not count that discount | 60 min at $0.010 = $0.60 | included in audio tokens |
| Speech to text | included | 60 min at $0.0025 (idle inclusive) = $0.15 | included |
| Model | 60 min at about $0.003 = $0.17 | $0.17 | audio tokens below |
| Voice | included | 30 min of speech, about 30k characters at $37 per million = $1.12 | in and out audio tokens |
| Audio tokens | not applicable | not applicable | input about $0.03 cached at $0.40 per million, or $2.59 uncached. Output about $3.46 |
| Per session | $4.97 | $2.04 | about $6 cached, or about $115 uncached |
At 10,000 conversations a day this is roughly 417 average concurrent calls and enterprise contract territory on any managed stack. The dominant line at scale is the voice, and a fine tuned self hosted TTS is the honest 40x lever there. That is the same model that solves the prosody guarantees problem, which is why the fine tune is planned, not improvised.
Long session context, same discipline. A 45 minute conversation at 150 words per minute is about 10k transcript tokens, which fits comfortably in any modern context window. The expensive regime is native audio, about 81k audio tokens re read on every turn, roughly $115 per session uncached versus $1.44 cached. My rolling summary design holds 2.2 to 4.9k tokens in steady state at any session length. The real risk in long sessions is not context size anyway, it is quality: prosody drift, repetition loops, and memory hallucination. Those get their own soak tests, because no vendor has published 30 to 60 minute streaming evidence and I am not going to pretend otherwise.
Staying on a managed platform forever is not the plan, and waiting too long to leave is not the plan either. Here is when each move happens and what forces it.
| Move | When | How | What triggers it |
|---|---|---|---|
| Fine tune our own TTS | After the beta harvests consented conversations, so month two at the earliest | A Kokoro class model, full fine tune with my own G2P style pipeline. I have done this end to end on an 8 GB GPU before, with an ASR round trip evaluation harness, so this is a known path, not a research project | Voice cost dominating the bill at scale (it is a roughly 40x lever), or we need prosody guarantees nobody sells: emotion arcs across 40 minutes, stable identity, warmth calibration |
| Behavior fine tuning of the conversational model | Once transcripts are curated and labeled, around the same window | Supervised fine tuning on curated multi turn examples, then preference tuning on hand labeled pairs. Prompting stays the tool of choice until volume justifies it | Eval scores plateauing on empathy or safety behavior where prompting is not moving them |
| Our own model endpoint in the platform | Any time, it is a config change | The platform accepts any OpenAI compatible endpoint, so our FastAPI LLM service slots in without leaving the platform | Latency tails from the shared catalog, or cost at scale, or we want a model the catalog does not carry |
| Self hosted voice stack | When a contract or regulation forces it, or unit economics say so | The platform's Speech Engine lets our server be the brain while it runs speech to text and voice, or we move to the self assembled cascade with the same eval harness. Agent configs export as JSON through the CLI, so nothing is trapped | Data residency requirements, HIPAA tier constraints, session length caps, or the per minute platform fee exceeding the cost of running it ourselves |
One of the reasons I picked this platform is that most of what a mental health companion needs is already a first class feature instead of custom code. Here is the honest map, including what is live in the demo on this page and what is not.
| Capability | What it is | Why and when to use it here | In the demo? |
|---|---|---|---|
| Knowledge Base | Upload sources (files, URLs, text). Retrieval modes: on demand, every turn, or prompt only. Plus a memory store for cross session facts | Grounding techniques, active listening patterns, and crisis resources live here so answers come from a document we control and can audit, with source attribution on every retrieval. On demand mode for facts the agent should pull only when relevant, every turn mode for a short safety card that must always be in reach | Live: grounding playbook, retrieval on |
| Procedures | Playbooks for one specific task, triggered by what the user says. Free form ones adapt their wording, structured ones run typed steps in a fixed order | Free form for grounding exercises where tone should adapt. Structured for the crisis escalation, because that is exactly the case where a skipped step costs too much. This is how safety behavior stays consistent instead of depending on prompt luck | Live: 3 procedures (crisis escalation, grounding, emotional range) |
| Workflows | A graph of sub agents with routing and per step control | For flows with real branching: intake and triage, escalation to a human, multi stage journeys. Not needed for a companion conversation that should feel like one continuous talk, so I would not start here | Not in demo, by choice |
| Guardrails | Focus, prompt injection, content thresholds (including self harm and medical or legal advice), and custom rules with retry or end call actions | Content guardrails on self harm and medical advice thresholds as a second line behind our procedures. Custom guardrail for the never pretend to be a therapist boundary | Live: content thresholds on |
| Tools | System tools (flag for review, end call, memory search, transfer, skip turn), webhook tools, and MCP servers. Interruptions can be disabled while a tool runs | Flag for review on any crisis signal. Interruption suppression during the flag tool so the safety action cannot be talked over. Webhook tools later for on call handoff and human review queues | Live: flag for review, end call, memory search |
| Analysis | Post call evaluation criteria, structured data collection, sentiment, topic discovery, and source attribution, over the transcript | Evaluation criteria like empathy before advice and safety handled turn quality into numbers we can regress against. Data collection of mood trend and risk flags feeds the learning loop and the review queue | Live: 3 criteria, sentiment, topics |
| Tests | Simulation tests with a simulated user, next reply scenario tests, and tool call tests, runnable from dashboard or CLI | The safety regression set runs here in CI from Day 5: crisis cases, escalation cases, tone under distress. Text level tests here, audio level behavior in my own listening harness | Not yet attached, first tests are a Day 5 item |
| Experiments and versioning | Immutable versions, traffic split A/B, instant rollback. Every change to a procedure or prompt is a version | How a prompt change reaches real users safely, and how a kill criterion reversal actually happens in practice | Versioning on by default |
| Channels | Widget today, and WhatsApp, Slack, SMS, telephony behind the same agent | Web widget first because it is where the demo and the beta live. Telephony matters later for accessibility and for crisis scenarios, and it is one of the things I would want your priorities on | Live: widget |
| Observability | Conversation search, per call transcripts and traces, latency metrics, and an observation layer that groups topics and flags shifts | Between this and our Langfuse traces we get both the platform view and the vendor neutral one. The vendor neutral one matters because stages will change | Traces on, Langfuse in the plan |
These are the questions I would ask me if I were in your chair. One sentence each, evidence attached in the detailed document.