# How it works
voicemate is a pipeline of small local models around one LangGraph agent. Everything runs
in one Python process except the language model, which runs in Ollama.
## The whole system
```mermaid
flowchart LR
subgraph Browser["Browser tab (localhost:8080)"]
MIC[Microphone
echo cancellation] --> WORKLET[AudioWorklet
16 kHz PCM]
PLAYER[Speaker
gapless scheduler]
UI[Status rail, conversation,
activity trace, notes]
end
subgraph Server["voicemate process (CPU)"]
VAD[Silero VAD
endpointing] --> ASR[Parakeet v3
speech to text]
ASR --> LID[Language
detection]
LID --> AGENT[LangGraph agent]
AGENT --> CHUNK[Sentence chunker]
CHUNK --> TTS[Phonon or Kokoro EN / Piper HU
text to speech]
AGENT <--> MEM[(Memory
LanceDB + bge-m3)]
AGENT <--> NOTES[(Notes
SQLite)]
AGENT <--> TOOLS[Tools]
BUS{{Event bus}}
end
subgraph GPU["Ollama (GPU)"]
LLM[Chat model
one profile: gemma4 or qwen]
end
WORKLET -- WebSocket --> VAD
TTS -- WebSocket --> PLAYER
AGENT <--> LLM
TOOLS -- only these go online --> NET((Web search, pages,
arXiv, weather))
VAD & ASR & AGENT & TTS -.-> BUS -.-> UI
```
The speech models run on the CPU on purpose: the GPU and most of the memory belong to the
language model (see the spec, §10).
## Modules and seams
Fourteen deep modules keep the rules in one place each. Dashed lines are the seams: the interface
other modules (and the tests) use. Behind each seam sits a real adapter and, in the tests, a
small hand-written fake.
```mermaid
flowchart TB
UI[ui/app.py
page + /ws/audio] -->|messages| PROTO[[audio.protocol
browser messages]]
UI -.-> RAIL[[Rail
lane state and durations]]
UI --> SESSION[VoiceSession
plumbing: wires the modules, owns timeout and cancel]
SESSION -.-> LISTEN[[Listener
frames to utterances]]
SESSION -.-> POLICY[[TurnTaking
what input means]]
SESSION -.-> REPLY[[Reply
a turn's answer rules]]
UI -.-> REMIND[[Reminders
one watcher, due once, every tab]]
REMIND -.->|announce| SESSION
SESSION -.-> CONV[[Conversation
thread, history, repair, stream]]
SESSION -.-> SKILLS[[Skill
answers known questions without the LLM]]
SESSION -.-> ACT[[Activation
always, or only after "Ava"]]
SKILLS -.-> HOTS[[HotS draft advisor
knowledge, ranking, commands]]
HOTS --> PAGES[(saved Icy Veins pages
data/icyveins)]
HOTS -.-> SCREEN[[Draft screen
capture, text, portraits, watcher]]
SESSION -.-> SPEECH[[Speech
async ASR + TTS, one worker each]]
LISTEN -.-> SPEECH
CONV -.-> GRAPH[(LangGraph + checkpoint)]
GRAPH --> TOOLS[agent tools]
TOOLS -.-> RESEARCH[[Research
cached search, listed addresses]]
RESEARCH --> MEM[(Memory: facts, episodes, research)]
REMIND --> NOTES[(Notes)]
RESEARCH --> NET((web, arXiv, pages))
SPEECH --> MODELS[Parakeet · Piper · Phonon · Kokoro]
classDef deep fill:#0f172a,color:#f8fafc,stroke:#0f172a;
class POLICY,CONV,SPEECH,RESEARCH,PROTO,LISTEN,REPLY,REMIND,RAIL,SKILLS,ACT,HOTS,SCREEN deep;
```
| Module | Owns | Faked in tests with |
|---|---|---|
| `pipeline.turn_taking` | hold and release, merging a cut-off question (20 s), answering a confirmation, barge-in. Pure: no event loop, no I/O | nothing needed: plain values in, decisions out |
| `pipeline.conversation` | the current thread id (`data/conversation.json`), history lines, repair of an interrupted turn, discarding a cancelled message, streaming a turn, the pending question | in-memory LangGraph checkpointer + a scripted chat model |
| `pipeline.listener` | the voice-activity segmenter, live partial transcripts (every interval, last 8 s), background transcription, what is still in flight (`settled`), flush on mic off | an energy-based VAD double from the test helpers + a controllable recognizer |
| `pipeline.reply` | one turn's answer: text to speakable sentences, the spoken filler for network tools, the apology, first-token and first-audio timings, the turn's metrics | an injected clock; plain calls |
| `pipeline.reminders` | polling due reminders, marking each notified before announcing (once only), the language prefix, the 30 s watcher; the server runs one watcher and tells every open tab, and nothing is consumed while no tab is open | a real notes database in a temp folder + a recording announcer |
| `speech` | transcription and synthesis, one worker thread per model (MLX is not re-entrant), clean shutdown | scripted recognizer, tone synthesizer |
| `agent.tools.research` | cache-first web and arXiv search, page and paper access, the listed addresses (anything else asks the user first), what leaves the machine | static search and arXiv backends, a local HTTP server |
| `memory.store` | one database and embedder with three narrow interfaces (`facts`, `episodes`, `research`) plus `recall`; each caller sees only its slice | hash-based embedder, injected clock |
| `ui.rail` | what each station of the pipeline rail shows: state, duration, tool name, reset on a new turn. No UI framework | plain events in, views out |
| `pipeline.skills` | the skill seam: a skill answers the questions it knows in one sentence (and its language), everything else goes to the agent; a skill may also work in the background and speak on its own when the assistant is idle; the exchange joins the thread once spoken | a scripted skill |
| `pipeline.activation` | when speech counts as a command: always, or only after the assistant's name (the name is stripped, other speech is dropped before anything shows, logs or stores it). Pure | plain strings in, command out |
| `hots` | the Heroes of the Storm draft advisor: page parsers, the SQLite knowledge base with a startup freshness check, the scoring (counters, picks, bans with reasons), spoken commands and the per-session draft. No LLM, no network | small hand-written pages in a temp folder |
| `hots.screen` | reading the draft screen: a `ScreenReader` seam (Quartz capture + Apple Vision text, or a saved screenshot), a pure interpreter (phase, map, timer, heroes by side, locked or pending), ban slots matched to portraits, the layout calibration (scale and offset measured from two anchor texts, so any screen size or window works), learned hero faces that cross-check misread labels, the draft watcher (modes idle, draft, loading, over; polling; screen log), the loading-screen briefing | a scripted screen with recorded text boxes and synthetic pictures |
| `audio.protocol` | the browser to server messages (mic off, playback started and ended, stop) | a contract test reads `audio.js` and fails if either side drifts |
`VoiceSession` is plumbing: it feeds inputs to the policy and the modules and executes the
decisions, and it imports nothing from LangGraph or LangChain (a test asserts that).
## The LangGraph agent
This diagram is generated from the compiled graph (`graph.get_graph().draw_mermaid()`):
```mermaid
graph TD;
__start__([start]) --> recall;
recall --> agent;
agent -.->|tool calls| tools;
tools --> agent;
agent -.->|answer| __end__([end]);
tools -.->|"interrupt(): needs permission"| pause([paused, waiting for you]);
pause -.->|"Command(resume=your answer)"| tools;
```
- **recall** embeds what you said and loads relevant facts, earlier conversations and
research into a per-turn context block (about 50 ms).
- **agent** calls the model with every tool bound. The system prompt never changes, so
Ollama reuses its cached prompt; only the newest message carries the context.
- **tools** runs the requested tools. Errors go back to the model as text, so it can retry
or explain. After 6 tool rounds the tools are removed and the model must answer.
- A **checkpointer** (SQLite) keeps the conversation, including interrupted answers and
paused runs. The current thread id lives in `data/conversation.json`, so a reloaded page
or a restarted server continues where you left off; **New conversation** starts a new one.
- If the model goes silent after a tool round, the agent asks it once more to answer.
## Asking before risky actions (human-in-the-loop)
```mermaid
sequenceDiagram
actor You
participant S as VoiceSession
participant G as LangGraph
participant T as write_file tool
You->>S: "Save my notes to plan.md"
S->>G: run
G->>T: write_file(plan.md)
T->>T: file exists
T-->>G: interrupt("plan.md already exists. Replace it?")
G-->>S: run paused (checkpointed)
S-->>You: dialog + spoken question
You->>S: "no, call it plan2" (voice, typing or a button)
S->>G: Command(resume="no, call it plan2")
G->>T: answer is not a yes → nothing replaced
T-->>G: "The user did not approve; they said: no, call it plan2"
G->>G: model writes plan2.md instead
```
The tool receives your exact words, so a "no, but…" can redirect the action. The pause
survives page reloads because it lives in the checkpointer.
## Choosing the model
```mermaid
flowchart TD
P[Preferred profile
fast / gemma / qwen] --> I{Installed in Ollama?}
I -- no --> F[Fallback profile]
I -- yes --> M{"1.5 × size on disk + 1 GB
+ 5 GB reserve fits in
free RAM?"}
M -- yes --> U[Use it]
M -- no --> F
F --> W[Use fallback, warn in the UI]
```
A model that does not fit gets paged out by macOS between turns; replies then take 15+ s
instead of about 1 s. Each profile is one model used for everything.
## One spoken turn
```mermaid
sequenceDiagram
actor You
participant B as Browser
participant S as VoiceSession
participant A as ASR
participant G as LangGraph
participant L as Ollama
participant T as TTS
You->>B: speak
B->>S: 20 ms audio frames
S-->>B: Listening
Note over S: about 1 s of speech:
live partial transcript
You->>B: stop talking
Note over S: 700 ms of silence ends the turn
S->>A: utterance
A-->>S: text (~0.3 s)
S->>G: astream(text, language)
G->>G: recall memory (~50 ms)
G->>L: prompt (cached prefix)
L-->>G: first tokens (~1 s)
loop each sentence while the model keeps writing
G-->>S: tokens
S->>T: complete sentence
T-->>B: audio
B-->>You: speech
end
```
The first sentence is spoken while the model is still writing the rest. A network tool
(search, weather) adds a short spoken filler ("One moment, let me check.").
## Talking while it works
```mermaid
stateDiagram-v2
[*] --> Idle
Idle --> Listening: you speak
Listening --> Thinking: silence ends the turn
Thinking --> Speaking: first sentence ready
Speaking --> Idle: playback finished
Thinking --> Listening: you speak again
note right of Thinking
Nothing was said yet: the unfinished turn is dropped and,
within 20 s, your two utterances become one question.
end note
Speaking --> Listening: you speak again
note right of Speaking
Playback stops at once. The part you heard is kept in the
history, marked as interrupted; cut-off tool calls are closed.
Your next words are treated as a correction or a new question.
end note
```
**Hold until I'm finished.** The Hold button (Ctrl+H) stops a running answer and suspends
endpointing: each pause still produces a transcript, but it is collected rather than
answered. **I'm finished**, or turning the microphone off, sends everything as one
question. Turning the microphone off without a hold also answers what was heard so far,
including a sentence cut off mid-word.
While it speaks, barge-in needs louder and longer speech (`vad.barge_in_*`), so its own
voice from the speakers does not interrupt it.
## Research memory
```mermaid
flowchart TD
Q[web_search query] --> C{Similar search stored
and still fresh?}
C -- yes --> M[Answer from memory
badge: from memory]
C -- no --> W[Search the web
badge: sent to web]
W --> S[(Store the search and
its snippets as documents)]
S --> R[Answer]
P[fetch_page / download_paper] --> D[(Chunk, embed and store
the full text)]
D -.-> REC[recall node finds it
in later conversations]
S -.-> REC
```
Freshness depends on the topic: news and jobs expire after a day, general pages after 30
days, papers after a year.