A Voice Agent in the Browser, One Fix at a Time

The chat widget in the corner of this site started as a text box talking to an LLM through a tiny serverless function. I wrote about how that came together. This post is about the day I asked a simple question: can it talk?

I built this pair-programming with Claude Code, and like the sandbox infrastructure post, the shape of the final design was not planned. Every step below exists because the step before it was too slow, too heavy, or sounded wrong. I’m keeping the wrong turns in, because they taught me more than the parts that worked first time.

Step 0: Can Parakeet and Kokoro Even Run in a Browser?

The question I started with was specific: NVIDIA’s Parakeet for speech-to-text and Kokoro for text-to-speech, both running in the visitor’s browser, bolted onto the existing chatbot.

The answer was “yes, but”. Both models have ONNX builds that run through onnxruntime-web. Kokoro is a pleasant surprise: 82 million parameters, about 90 MB quantized, with a ready-made library in kokoro-js. Parakeet is the problem. The 0.6B model is a 600 MB+ download before a visitor can say hello, and phones would run out of memory long before they ran out of patience.

The good news was the chatbot itself. It already did text in, text out through /api/chat. Voice could live entirely on the client, before and after that call, without touching the server:

mic ─► speech-to-text ─► /api/chat (unchanged) ─► text-to-speech ─► speaker

So the plan became: use the browser’s built-in SpeechRecognition when it exists, fall back to a small on-device model when it doesn’t, and use Kokoro for the voice. Parakeet stayed on the shelf.

Step 1: The First Working Version

The first version had three pieces, all lazily loaded the first time the mic is pressed, so none of it touches page load:

• Speech-to-text. SpeechRecognition (or webkitSpeechRecognition) with interim results streamed into the input box, so you can watch your words appear. When it isn’t there, the fallback records with MediaRecorder, stops after 1.5 seconds of silence, resamples to 16 kHz mono, and hands the audio to Moonshine running through transformers.js in a Web Worker.
• Text-to-speech. Kokoro in its own Web Worker, posting audio back one sentence at a time so playback starts before the whole reply is synthesized. Markdown gets flattened to plain text first, and links and code blocks are dropped, because nobody wants to hear a URL read aloud.
• The widget. A mic button next to the input. Click to talk, click again to stop listening or stop speaking.

One detail mattered more than I expected: having the API is not the same as having it work. Brave exposes SpeechRecognition but has no backend behind it, so it fails with a network error the moment you start. The fallback has to trigger on that, not just on the API being missing:

// Errors that mean the built-in engine is unusable here (Brave exposes the
// API but has no backend), as opposed to "the user said nothing".
const NATIVE_FATAL_ERRORS = new Set(["network", "service-not-allowed", "language-not-supported"]);

To test the whole loop without a microphone, I had Kokoro speak a question, resampled its output, and fed it to Moonshine. “What projects has Sreekesh built recently?” came back as “What projects has S. Recash built recently?” Close enough to prove the plumbing, and a fair review of my name.

Step 2: The Stream That Never Ended

The first real bug was a hang. Kokoro spoke the first sentence of a reply and then went silent forever.

kokoro-js has a stream() method that splits text into sentences and yields audio for each one. Pass it a plain string and, in version 1.2.1, it builds a sentence splitter internally and never closes it. The iterator yields every sentence and then waits for more text that will never come. The fix is to build and close the splitter yourself:

// kokoro-js 1.2.1 never closes the splitter it builds from a plain
// string, so the stream hangs after the last sentence. Close our own.
const splitter = new TextSplitterStream();
splitter.push(text);
splitter.close();

The same session surfaced a dev-server trap that is worth a sentence. I had a second dev server running for testing, and both shared Vite’s dependency cache. When mine re-optimized it for the new packages, the original server started serving 504 (Outdated Optimize Dep). The lasting fix was to list the voice dependencies in optimizeDeps.include, so Vite bundles them at startup instead of discovering them the first time someone presses the mic.

Step 3: “Generating Voice Has Too Much Latency”

It worked. It was also painfully slow. Before guessing, I measured, and the numbers explained everything:

Setup	Model load	Time to synthesize ~5 s of speech
WASM, q8	~13 s	~11 s
WebGPU, fp32	~38 s, every visit	~1.8 s
WebGPU, fp16	~1 s once cached	~1.6 s

Two problems were hiding in there. Kokoro was taking about twice as long to generate a sentence as it takes to say it, and the model only started loading after the reply came back.

WebGPU looked like the obvious escape. The fp32 build was fast, but its 326 MB file never landed in the browser cache, so every visit paid the download again. fp16 was half the size, cached properly, and just as fast. My checks said the audio was healthy: no NaN samples, sensible loudness. I shipped it.

Step 4: The Revert

Then I actually listened to it. The fp16 voice was badly distorted.

So it was reverted, all of it, back to the clean WASM build. The lesson is not about fp16. It’s that for audio, “the numbers look fine” is not a test. Every metric I checked was a proxy for “does it sound right”, and none of them were the real thing. For anything a human perceives directly, the human check is the test.

Step 5: Make the LLM Talk Like a Voice Agent

Instead of making synthesis faster, I asked a different question: why synthesize so much? A typical text reply was 235 words with markdown headings. That’s a minute and a half of audio, at twice real time to generate.

The client now sends a mode flag with each message, and voice turns get an extra instruction appended to the persona:

body: JSON.stringify({ message: userText, mode: speakReply ? "voice" : "text" }),

The first attempt at the instruction went from 235 words to 82, but the model still plugged a blog post and read out my email address. That turned out to be my own doing: the persona tells the assistant to point visitors at posts and offer my email. The voice instruction now says outright that it overrides those rules unless the user asks. Replies settled at 21 to 51 words, which is about what you’d say out loud.

Step 6: Hide the Wait, Then Remove It

With replies short, the remaining latency had three separate causes, and each needed its own fix.

Warm up while the user is talking. Pressing the mic now starts loading Kokoro and runs a throwaway generate("Hi."), because the first inference is several seconds slower than the rest. Speaking a question and waiting for the LLM easily covers that. In testing the model was ready about 4 seconds after the mic press, before the reply would come back anyway.

Split the opening sentence. Synthesis time grows with text length, so a long first sentence delays the first sound. A long opener is now split at its first comma or clause break. This helped less than I hoped: in my test reply the first comma came after ten words.

Use all the cores. This was the big one. crossOriginIsolated was false on the site, which means the ONNX runtime ran on one core of an eight-core machine. Multi-threaded WASM needs SharedArrayBuffer, and browsers only allow that on cross-origin isolated pages. Two response headers turn it on:

{
  "source": "/((?!notes/naa-ninage-nee-nanage).*)",
  "headers": [
    { "key": "Cross-Origin-Opener-Policy", "value": "same-origin" },
    { "key": "Cross-Origin-Embedder-Policy", "value": "credentialless" }
  ]
}

I benchmarked it before committing: the same page served with and without the headers, same model, same text. With isolation, the first chunk took 31.8 s instead of 60.5 s, about 1.9 times faster. Those absolute numbers are terrible because the test tab was running in the background and throttled. The ratio is what counts, and inside the real app the opening sentence dropped from 11.2 s to 6.0 s.

The exclusion in that source pattern is there because COEP blocks cross-origin iframes that don’t opt in, and one note on this site embeds a YouTube video. That page skips the headers. But there was a catch I would have missed: this site uses Astro’s client-side router, and isolation belongs to the document, not the URL. Navigate client-side from an isolated page to the note, and the page stays isolated and the video breaks anyway. The router has a hook for exactly this. Cancelling astro:before-preparation makes it fall back to a full page load:

document.addEventListener("astro:before-preparation", function (e) {
  if (!window.crossOriginIsolated) return;
  var path = e.to.pathname;
  if (NON_ISOLATED_PATHS.some(function (p) { return path.indexOf(p) === 0; })) {
    e.preventDefault(); // router falls back to location.href: a real load
  }
});

On security: COOP and COEP are hardening headers. COOP cuts the link between your page and any window it opened or was opened by, and credentialless means any cross-origin resource your page loads without permission is fetched with no cookies. They do turn high-resolution timers and SharedArrayBuffer back on, but the browser only allows that because those same headers keep other sites’ private data out of your page’s process. The real risk is still your own XSS, and that was already game over before any of this.

Step 7: Line by Line, Done Right

The next suggestion was natural: generate audio line by line instead of in big chunks. It was already doing that, sentence by sentence. Cutting smaller only helps one thing, the wait before the first sound. It doesn’t remove pauses later, because the total work stays the same, and chopping mid-sentence wrecks the intonation, since Kokoro shapes a whole phrase at once.

The better move was to get short sentences from the source. The voice instruction now asks for an opening sentence of three to six words and every other sentence under twelve. The first try was inconsistent. One reply had a 26-word sentence, and another copied my example opener word for word. Two changes fixed it: removing the copyable examples, and repeating the key rules in a one-line reminder appended to the user’s message, right where a small model is paying attention:

const VOICE_MODE_REMINDER =
    "\n\n(Voice reply: answer as Sreekesh in the first person, start with a sentence of three to six words, keep every sentence under twelve words, no markdown.)";

After that, every sentence across four test replies was nine words or fewer, and the opening sentence was ready in 4.6 s instead of 6.0 s.

That reminder also fixed the last annoyance. Asked “Is he open to new roles?”, the assistant would happily answer “Yes, he is open to new roles”, drifting into the third person because the question did. One more rule, first person always, and all five test questions about “him” came back as “I”.

Step 8: Safari Says “undefined is not a function”

The first bug report after that came from Safari: Unhandled Promise Rejection: TypeError: undefined is not a function (near '...B of M...'). Minified variable names don’t say much, but the shape does. B of M is a for...of loop, so something was being iterated that Safari didn’t know how to iterate.

Searching the built worker bundle for that exact text found it, and it wasn’t in my code. Kokoro turns text into phonemes with the phonemizer package, which ships the eSpeak dictionary gzipped inside the bundle and unpacks it the moment it loads:

const stream = new Blob([data]).stream().pipeThrough(new DecompressionStream("gzip"));
for await (const chunk of stream) chunks.push(chunk);

for await over a ReadableStream works in Chrome and Firefox. Safari doesn’t support it.

I don’t have Safari in my test setup, but I didn’t need it. Deleting ReadableStream.prototype[Symbol.asyncIterator] in a Chrome worker before importing the TTS worker reproduced the failure exactly: is not async iterable, the moment phonemizer loaded. It also showed a second, worse problem. The rejection happened inside phonemizer’s own async code, out of reach of any try/catch, so the worker never answered. Safari visitors saw “generating voice…” forever, and the fallback to the browser’s built-in voice never ran.

Two fixes. A tiny polyfill, imported before kokoro-js so it runs before phonemizer does:

// Must stay first: it patches ReadableStream before phonemizer loads (Safari).
import "./readable-stream-iterator";
import { KokoroTTS, TextSplitterStream } from "kokoro-js";

And a guard in the worker that turns any unhandled rejection from a dependency into an error message the page can act on, so a failure always falls back to speechSynthesis instead of hanging:

self.addEventListener("unhandledrejection", (e) => {
    ttsPromise = null;
    self.postMessage({
        type: "error",
        id: currentJob > 0 ? currentJob : undefined,
        error: e.reason?.message || String(e.reason),
    });
});

The import order matters, so I checked the production bundle too: the polyfill is the very first statement in the worker, about 1.4 MB ahead of phonemizer’s loop.

Step 9: The Fix That Broke the Pauses

Then a subtler report: the lines were being read straight into each other, with no break between them. Had something broken?

Before guessing, I traced a reply through the real playback path, logging every audio clip it started. Each sentence played once, nothing overlapped, nothing repeated, and Kokoro’s own clips already carry about a second of silence at each sentence boundary. Playback was fine. So the problem had to be in what Kokoro was given.

It was, and Step 7 caused it. Once the prompt asked for short sentences, the model started putting each one on its own line. In markdown that’s a line break, which renders as <br>. Before speaking, the reply was flattened to plain text with textContent, and textContent keeps no trace of a <br> or a list item boundary:

Reply line breaks	What Kokoro was given
open. ↵ I welcome…	open.I welcome…, no space, so no sentence break
open ↵ I welcome…	openI welcome…, two words fused into one
a bullet list	every item in one sentence

Fixing one step had quietly changed the input of the next one. The repair treats every line, paragraph and list item as its own sentence, and gives any line without closing punctuation a full stop:

doc.querySelectorAll("br").forEach((el) => el.replaceWith("\n"));
doc.querySelectorAll("p, li, h1, h2, h3, h4, h5, h6, blockquote").forEach((el) => el.append("\n"));
return (doc.body.textContent || "")
    .split("\n")
    .map((line) => line.replace(/\s+/g, " ").trim())
    .filter(Boolean)
    .map((line) => (/[.!?…:;]["'”’)]*$/.test(line) ? line : line + "."))
    .join(" ");

A three-line reply with no punctuation now comes out as three sentences and plays as three clips, each with its natural pause.

Step 10: On Phones, Let the Browser Talk

Then I tried it on a phone, and every fix above stopped mattering. Kokoro on a phone CPU runs several times slower than real time, iPhones never get the multi-threading because Safari doesn’t support credentialless, and the first reply waits on a 90 MB download, often over mobile data.

I weighed a few options: a smaller TTS model, a server-side voice, timing the device and deciding per visitor. The simplest answer was already in every phone. iOS and Android ship decent system voices behind speechSynthesis, they’re instant, and they download nothing. I briefly made the browser voice the default everywhere, then settled on a split, because on desktop Kokoro is fast enough and sounds the same for every visitor:

Device	First choice	Fallback
Phone or tablet	Browser voice	Kokoro
Desktop	Kokoro	Browser voice

Detecting a phone uses navigator.userAgentData.mobile where Chromium provides it and the user agent elsewhere, with one special case: iPadOS reports a desktop Mac user agent, so touch support gives it away.

Making the browser voice sound right. The browser voice is a different instrument, and it needed its own tuning:

• Choose the voice deliberately. Listing the voices on my Mac turned up “Bad News”, “Bells”, “Boing”, “Whisper”, “Zarvox” and a dozen other novelty voices. My first scoring preferred any voice matching the visitor’s locale, so on an en-US Mac it could happily have answered in “Bad News”. Novelty voices are now excluded, and since the assistant speaks as me, an Indian English male voice wins when there is one: Prabhat on Edge, Rishi on macOS, Ravi on Windows. Then natural male voices, like “Google UK English Male” on Chrome. Brave has no Google voices at all, so it lands on the OS voices.
• One utterance per sentence. Chrome cuts off long utterances after about fifteen seconds, and separate utterances also give each sentence its natural gap. The rate is set to 0.95, because Google’s online voices rush.
• Unlock it inside the tap. iOS only lets speechSynthesis talk after a speak() made inside a user gesture. The reply arrives seconds after the mic press, so the mic press now speaks a silent utterance to unlock it.
• Watch for engines that never start. Some engines accept speak() and then do nothing, and my own test browser turned out to be one of them. If the first sentence hasn’t started within three seconds, the reply falls back to Kokoro instead of waiting forever.
• Respell what the voices get wrong. “Sreekesh” becomes “Shreekesh” and “Thrissur” becomes “Trishoor”. “Node.js” becomes “Node J S”, which also fixed a hidden bug: its full stop was splitting the sentence in two.

Guarding the fallbacks. Kokoro is now the backup on phones and the first choice on desktop, so two checks keep it from doing more harm than good:

• Data saver. If the browser reports Save-Data or a 2G connection, neither Kokoro nor Moonshine is ever downloaded. The mic is hidden when voice input would need Moonshine.
• A speed check. During warm-up, the worker times one short phrase and reports seconds of compute per second of audio. My Mac measured 1.09. With the CPU saturated by busy workers to imitate a weak phone, it measured about 4. Above 1.5, Kokoro is switched off, its worker closed to free memory, and the browser voice takes over.

When nothing can speak, the reply stays as text and the status line says why, for example “voice replies off to save data”, instead of failing silently.

Where It Landed

The full path for one spoken question, today:

1. Mic press. The voice module loads, and both the browser’s audio and its speech voice are unlocked inside the click. On desktop, Kokoro starts warming up and times itself.
2. Listening. Built-in speech recognition streams words into the input. Without it, the recording goes to Moonshine on-device.
3. Ask. The transcript goes to /api/chat with mode: "voice", which asks for a short, first-person, plain-spoken reply.
4. Speak. The reply is flattened to plain text, one sentence per line, with names and tech terms respelled for the ear. On a phone, the browser’s voice reads it sentence by sentence. On desktop, Kokoro synthesizes the sentences one at a time on all available cores, with a long opener split at its first clause, and each plays as soon as it is ready. Either engine falls back to the other, and if neither can speak, the reply stays as text with a note saying why.
5. Interrupt. Clicking the mic, typing, or closing the chat stops playback mid-sentence.

Everything except the LLM call runs in the visitor’s browser. No audio leaves the device unless the browser’s own speech service sends it. The models only download on devices that will use them, and then once.

Key Takeaways

• Measure before you optimize, and measure the right thing. The slow part was not where I guessed. It was a single-threaded runtime and a model that started loading too late. A quick benchmark table pointed at both.
• For perceptual output, listen. fp16 passed every numeric check and still sounded broken. No metric replaces the human check for audio.
• The cheapest latency fix is less work. Asking the LLM for 50 words instead of 235 did more than any runtime trick, and asking for short sentences made the first sound arrive sooner without hurting how it sounds.
• Small models follow what is close. Rules at the end of a long persona drifted. The same rules repeated next to the user’s message held.
• Isolation belongs to the document. With a client-side router, response headers alone are not enough. The router has to know which pages need a real page load.
• You can reproduce another browser’s bug in yours. Removing one missing feature in Chrome reproduced the Safari failure exactly, and showed a silent hang I would never have seen otherwise.
• Every async failure needs a path back to the user. A rejection deep inside a dependency left the page waiting forever. One listener turned it into a fallback.
• A fix upstream changes the input downstream. Short sentences from the LLM were the right call, and they broke the text flattening that came after. Trace the pipeline before you blame the last stage.
• Pick the engine per device. The best voice on a laptop was the worst on a phone. One engine order per device class beat tuning one engine for both.
• Look at the list you’re choosing from. “Prefer the user’s locale” sounded safe until the list turned out to include “Bad News” and “Zarvox”.
• Keep the reverts in the story. The fp16 detour was a mistake, and it produced the most useful rule in this post.

It isn’t perfect. Older desktops pay a moment for the speed check before falling back, and every phone now sounds like its own system voice rather than one consistent one. But it talks back now, as me, on a laptop or a phone, and that started as a question I wasn’t sure had a good answer.