Source: https://fnflow.app/whisper-custom-vocabulary

# Teaching Whisper your jargon: initial prompt and replacements
Whisper is good at ordinary speech and bad at your project's names. It has never seen your internal service or your teammate's surname, so it writes the closest common word. There are two fixes. The initial prompt nudges the model toward your spelling before it decodes, and replacements correct the text afterwards. FnFlow uses both, and here's how each one works.

## What the initial prompt does

Whisper writes text one token at a time, and each token depends on the text that came before. The initial prompt fills that "text before" with words you choose, as if they had just been said. OpenAI's own Python package describes it as context for the first window, for example custom vocabularies or proper nouns, to make those words more likely.

It is not an instruction. OpenAI's speech-to-text guide says Whisper doesn't follow instructions the way a general-purpose text model does. Its prompting cookbook adds that Whisper copies the style of the prompt, capitalization and punctuation included, and that a glossary of correct spellings fixed misspelled names in its examples. A prompt also can't change what was said. It only shifts the odds between spellings that fit the audio.

## Using it in whisper.cpp

With the command-line tool, pass the terms with --prompt:

whisper-cli -m models/ggml-large-v3-turbo-q8_0.bin -f memo.wav -l auto \
--prompt "Supabase, Postgres, pgvector, Zustand"
whisper-server accepts the same thing as a form field on each request:

curl 127.0.0.1:8080/inference -F file=@memo.wav -F prompt="Supabase, Postgres, pgvector"
The help text gives the limit as half the model's text context. In whisper.cpp that context is 448 tokens, so the prompt gets 224, the same figure OpenAI publishes. If you go over, the code keeps the last tokens and drops the start, so put the terms you care about most at the end.

Whisper reads audio in 30-second windows. The initial prompt is used for the first one, and later windows are conditioned on the text already decoded. For a long recording, --carry-initial-prompt prepends your prompt to every window. A typical dictation fits in one window, so this rarely matters there.

## Replacements catch what the prompt misses

A prompt makes a spelling more likely, and a replacement makes it certain. It's a plain find-and-replace on the finished text. FnFlow applies replacements to whole words, ignores case and runs longer entries before shorter ones, so "kubernetes" becomes "Kubernetes" without touching a word that merely contains it. Replacements work on the output of either engine, including Apple's on macOS 26.

## What we measured

We tried five terms on short test sentences. With large-v3-turbo, "update Zapot" became "update Saypad". The small model heard "post-GIRS" without the prompt and wrote "Postgres" with it. We also sent the same audio file to whisper-server twice. Without the prompt it returned "Zapot", and with the prompt it returned "Saypad".

It wasn't perfect. A Russian sentence came back with "Postgres" right and "Supabase" spelled "Supabaes", which is the kind of near miss a replacement fixes.

## Side effects to watch

Names can switch alphabet. When a Russian name was in the vocabulary in Cyrillic, the model sometimes wrote it in Latin letters.

The prompt can echo on silence. With no speech to decode, Whisper may hand your prompt back as the transcript. FnFlow treats a result identical to the prompt as empty, and trims a copy of the whole prompt from the start of a result when it has three or more terms. A single vocabulary word at the start of a sentence stays, because you might have said it.

Style carries over too. Since Whisper copies the prompt's style, write each term exactly as you want it to appear.

## FnFlow's vocabulary

Settings → Vocabulary takes up to 60 terms, with duplicates removed regardless of case. The list goes to Whisper as the prompt on every dictation, and an empty list clears it. Replacements live in the same place. Everything is stored in vocabulary.json in FnFlow's folder, readable only by your user. To check a new term, run an older recording that got it wrong through recognition again from the history. If you'd like to try the prompt with plain whisper.cpp first, [running Whisper locally on a Mac](run-whisper-locally-mac) covers the setup, and [how FnFlow was built](how-i-built-local-dictation-mac) explains where the vocabulary fits in.

## Questions

**How long can a Whisper prompt be?**
224 tokens. whisper.cpp's help lists the limit as half the model's text context, and OpenAI's docs give the same number. Tokens aren't words, and unusual names often take several tokens each. If the prompt is longer, only the end is used.

**Can I tell Whisper to always spell a word a certain way?**
Not with an instruction. OpenAI says Whisper doesn't follow instructions like a chat model. Write the word in the prompt with the spelling you want, and add a replacement if it still comes out wrong.

**Does the prompt work for languages other than English?**
In our tests it helped with Russian speech full of English product names, though not perfectly. One term came out as "Supabaes", which a replacement then fixed.

**Does FnFlow send my vocabulary anywhere?**
No. It's stored in a local file and passed to Whisper running on your Mac, through a server that listens on 127.0.0.1 only.
