07 · Open Source
zeldaFlow
Hold Fn, speak, release. Your words land at your cursor, in any app, and your voice never leaves your Mac.
zeldaFlow is our fully local dictation and voice command platform for macOS. No cloud, no account, no subscription. It is open source under Apache 2.0, and it is the same app we use at the lab every day.
Why we built it
We dictate all day, and every tool we tried either streamed our audio to someone else's server or charged a monthly fee for typing with our own voice. That felt backwards. So we built the thing we wanted: dictation that is fast, accurate, and provably private, because the audio has nowhere to go.
zeldaFlow is the first project from the lab to ship as open source. Not a demo, not a teaser. It is the app we use every day, released with its documentation, its test harnesses, its measured performance numbers, and 36 architecture decision records that explain why every load bearing choice was made.
One rule shaped everything else: dictation never touches the network. That sentence is the design constraint, not a tagline, and any change that breaks it will not be merged. Everything on this page follows from it.
In the app
One minute of zeldaFlow: dictation, commands, and meeting notes, all running locally.
Draft a quick note to the team about Thursday
The pill
Hold Fn and talk. A small pill at the bottom of your screen shows a live preview of your words, and the finished text pastes wherever your cursor is when you stop.
“Open Notes and play some jazz”
Command mode
Triple tap Fn and speak a command instead. One sentence can chain several actions, and by default an email or a message shows you the recipient and waits for an Fn tap before it goes.
Summary
Q3 launch moves to the 14th…
Action items
Sarah: confirm venue by Friday
Meeting notes
When a meeting app takes the mic, zeldaFlow records both sides, separates the speakers, and writes editable notes. The audio is deleted as soon as the transcript is finalized.
These vignettes are recreated in code from the live interface so they stay sharp at any size. The real thing looks just like this, only it moves.
How it works
One key, four gestures
A low level tap on the Fn key handles everything: hold to dictate, double tap for hands free, triple tap for a voice command, and a bare tap to stop hands free or approve a gated action. Dictation only ever records between your press and your stop.
Whisper, on your Mac
whisper.cpp runs the large-v3-turbo model in process on Metal. A typical utterance transcribes in about a second after you release the key.
Silence never reaches the model
A voice activity filter keeps quiet stretches away from the decoder, which is how phantom text and caption junk get killed before you ever see them.
Optional local cleanup
A small local model strips fillers, applies your self corrections (say “no wait, Wednesday” and you get Wednesday), and fixes punctuation. It adds a fraction of a second, and if it ever fails you get the raw transcript instead of nothing.
Paste, then put things back
The text lands at your cursor through a synthetic paste, and whatever was on your clipboard before is restored. Your workflow never notices.
Commands are structured, never scripted
In command mode, a deterministic parser handles common phrasings from your exact words first, so “open Safari” can never become something else. Whatever it does not recognize goes to a small local model that fills parameters into a fixed set of actions. The model never writes code and there is no shell path. Hand written templates do the actual work, natively.
Humans sit at the approval gates
Dictation is friction free, but consequences are gated. Out of the box, before an email or message goes out, the pill shows you the recipient and waits for a deliberate Fn tap; that confirmation is yours to toggle in Settings. Background agent tasks are gated unconditionally, with no setting to turn that off. That split, frictionless input with human approval on anything consequential, is the zeldaLabs thesis running on a Mac.
Exactly two network exceptions, both documented
Command mode has two deliberate exceptions to fully local, and the repo names both. An anonymous Apple Music catalog lookup when you ask for a song that is not in your library, and the optional Claude agent bridge, which can be switched off in one click and is gated behind a keypress every single time. Dictation itself has zero. Not two, not one. Zero.
What it does
Push to talk dictation
Hold Fn, speak, release. Works in any app, in the language you spoke, in its own script. A floating pill previews your words live while you talk.
Voice commands
Open apps, play music, set reminders, send messages, draft content, chain several steps in one breath. Exact phrases are parsed deterministically; everything else becomes a structured action. The model never writes code.
Meeting notes
When a meeting app takes the mic, zeldaFlow records both sides, separates speakers, and writes a summary with decisions and action items. The audio is deleted the moment the transcript is done.
Agent mode, optional
A bridge to the Claude Code CLI for screen questions and background tasks. It is the one exception to fully local, it can be switched off in one click, and every task waits for an explicit keypress before it starts.
Private by design
Dictation never touches the network. Not opt out, not anonymized: the audio path has nowhere to go. Models, history, and settings all live on your disk.
Verified, not promised
Twelve headless test harnesses pin the behaviour, 36 architecture decision records explain the reasoning, and the performance numbers in the repo come with the script that measured them.
Built to be built on
A repo that says “PRs welcome” and means “good luck” helps nobody. So zeldaFlow ships with an honest extension map: where the seams already are, which ones would need real surgery first, and the one thing that will never be merged. No guessing, no archaeology.
Two house rules apply to anything contributed back: every behaviour change brings an architecture decision record, and every behaviour gets an eval pin. The hard line is just as plain: nothing that puts dictation on the network goes in, ever. That constraint is the product.
A new voice command
Four small edits, none of them touching the orchestration.
A new meeting app
One data table. Detection, monitoring, and capture all read it.
Different models
Every model path lives in one file, ready to be swapped.
New export formats
Markdown, text, SRT, and JSON today. Yours next.
Your own hotkey
The binding layer is already data driven and rebindable.
Sharper hallucination filters
Pure string functions, heavily pinned by the eval suite.
What the community could build
Wire it into MCP
Bridge command mode to MCP servers and “file that invoice” can reach your actual tools: Linear, Notion, Slack, your CRM. The executor already thinks in structured actions, and MCP tool calls are the same shape.
Run your home by voice
A Home Assistant or HomeKit bridge would make zeldaFlow the smart speaker that does not phone home. Lights, scenes, thermostats, spoken from your desk, with a mic that only opens while you hold a key.
Drive your editor hands free
Voice macros for VS Code or vim, spoken commit messages, a refactor narrated instead of typed. For RSI flare ups or just tired hands, a fast local dictation layer can carry a whole coding session.
Take it where the cloud is not allowed
Clinics, courtrooms, client sites, airplane mode. Because nothing leaves the machine, forks of zeldaFlow can live in places cloud transcription tools are banned from entering.
Notes that file themselves
Exporters that push a meeting's decisions and action items straight into your tracker, wiki, or second brain the moment the call ends. Four formats exist today, and the seam is one file.
Point it at any agent
The agent bridge speaks to Claude Code today, but the shape is simple: send a prompt, stream progress back. Wrap your favourite CLI agent and your voice becomes its front end, confirmation gate included.
Get it running
Three commands. The one time signing script gives the build a stable identity, so macOS permissions survive every rebuild. The install script fetches the models (about 5 GB) and builds the app, and first launch walks you through Microphone and Accessibility access.
$ git clone https://github.com/zeldaLabs/zeldaFlow.git && cd zeldaFlow
$ scripts/make-cert.sh
$ scripts/install.shHonest limits, up front
- Apple Silicon and macOS 15 or newer only. Whisper runs on Metal, with an optional Neural Engine encoder, and the cleanup model runs on the local GPU.
- You build it from source. There is no notarized download yet, and the one time make-cert script is what keeps permissions stable across rebuilds.
- Meeting detection leans on app identifiers and window titles, so an upstream rename can pause capture until a one line fix lands.
- Speaker separation tells you there were three voices in a meeting. It never tells you who they were, and it is not supposed to.
Your voice, your Mac, your build.
Star it, fork it, break it, extend it. And if you build something on top of it, tell us in Discussions. That is what it is there for.