Cloning My Voice to Read This Blog in Six Languages

This site publishes in six languages: English, Portuguese, Spanish, German, French, and Italian. For a while that was a text-only promise. You could read any article in your language, but if you wanted to listen on a commute or while cooking, there was nothing to press play on. Adding audio was the obvious next step. The interesting question was whose voice it should be.

Round one: a voice that ships

The first version used free neural text-to-speech voices, one per language. It was the right first move: zero cost, minutes to integrate, and it put a working player on every article. Shipping it is also what exposed the real bar. The voices were clean but formal, they read my sentences with someone else's cadence, and a personal site narrated by a stock voice is a placeholder. My name is on the writing. The voice reading it should be mine too, including in the five languages I am not narrating by hand any time soon.

The unlock: a local model that clones

The piece that made this practical is Chatterbox Multilingual, an MIT licensed open-source TTS model from Resemble AI. Three properties made it the fit:

• It covers all six of my languages with one model.
• It runs on a consumer GPU. Mine is an RTX 3080, and generation uses about 5.5 GB of VRAM.
• It does zero-shot voice cloning: give it a short reference recording, and it reads any text in that voice, in any of its languages.

That last one is the whole feature. No studio, no training run, no dataset of me talking for hours. The reference that drives everything on this site is a recording I made at my computer in a quiet room: about forty seconds of me reading one of my own articles in Portuguese, at a normal pace.

Why the reference matters

Out of the box, the model's stock voice was not mine to begin with, and it also leaned Brazilian in Portuguese, which is not how I speak. There is no voice menu to fix that, no "male, European Portuguese" option to select. The voice is set entirely by the reference audio you condition on, so those forty seconds were the only lever I had, and they moved a lot. Timbre, register, and rhythm are recognizably mine in every language. The accent came along only in part: the narration sits far closer to European Portuguese than the stock voice did, and it still drifts toward a Brazilian cadence in places, because the model learned its Portuguese from data where the Brazilian variant dominates, and forty seconds of me bends that prior without erasing it.

The part that still surprises me is that the reference is Portuguese only. I never recorded an English sample, or a Spanish one, or German, French, Italian. The same forty Portuguese seconds drive all six languages: the model lifts what my voice sounds like from the recording and applies it to whatever language it is asked to read next.

Two tuning notes from getting it to sound right:

• The default pacing was too quick for long-form listening. Lowering the model's guidance weight (cfg_weight 0.3) and easing expressiveness produced a calmer, article-appropriate read. Those settings are baked into the pipeline, so every file comes out consistent.
• In English, the cloned voice carries my Portuguese accent. I kept it. A personal site that suddenly sounds like a California native in English and like me in Portuguese would be stranger than an accent, and the accent is the truthful version.

The pipeline

The brand of this site is production AI, so the narration had to be a system, not a one-off batch of files. The whole thing runs from one command, node scripts/generate-audio.mjs, and does the following:

1. Scans every article in every language and compares file timestamps, so only new or edited articles get regenerated. Publishing a new post means writing the markdown and running the script.
2. Hands the texts to a Python worker that loads Chatterbox once and keeps it warm, instead of paying the model load per file.
3. Reads each article chunk by chunk and stitches the audio with natural pauses, which is what keeps a 15 minute read from sounding like one breathless paragraph.
4. Normalizes loudness on every file. Raw model output came out around minus 41 LUFS, which is whisper-quiet next to any podcast. Everything is brought up to podcast loudness, around minus 16 to minus 18 LUFS, so the player sits comfortably next to whatever you listened to last.
5. Writes the result as a plain MP3 per article per language. The audio ships with the site like any other static asset: no streaming service, no third party player, no tracking.

Quality control runs in the opposite direction: after a batch, slices of the generated audio go through Whisper, the speech-to-text model, and the transcripts are checked against the source text. Today that is a spot check on samples rather than a gate inside the pipeline, and it has already caught what my ear never could, in the four languages I cannot proofread by listening.

The numbers, so you can size this for your own setup: chunked generation takes roughly twice the length of the finished audio, so a long article costs 25 to 30 minutes of GPU time per language. The full catalog, five articles times six languages, is about a twelve hour run. The reference recording never enters the repository; the script reads it from a local folder and the repo only ever sees the finished MP3s.

What a long batch teaches you

A twelve hour job will find your weak spots. My first full run died partway through when the process that launched it was torn down, a classic long-job failure. What saved it was the same design that makes publishing cheap: because the pipeline is incremental and every finished file is durable, the crash cost nothing but the file that was mid-encode, and even that one was recoverable from its temp output. Restart the command and it continues from wherever the last run stopped. If you are building any long-running generation pipeline, that property, resumability by default, is worth more than a progress bar.

What it sounds like today

I will not oversell it. The narration is clearly listenable and unmistakably me, and it still has rough edges: every so often a phrase comes out clipped, or a chunk stutters like a scratched record before it recovers. And the Portuguese keeps the drift I described above. It is my timbre reading my words, with a cadence that a Portuguese listener will clock as Brazilian-leaning in places. Mild enough that I shipped it, present enough that I want it in writing here. The Whisper spot checks catch the gross failures; the subtler ones slip through, and in four of these languages I would never hear them myself. So I treat the audio as what it is today, a companion. Press play while you read and it carries you through the text. The next iterations are already visible from here: a longer, cleaner reference recording to pull the accent further home, and turning the QA check into a repair loop, where any chunk whose transcription drifts from the source gets regenerated automatically, so every published file converges on clean.

The consent line

It took forty seconds of audio to make a convincing copy of my voice, on a gaming GPU, with free software. That is true of your voice too.

I am comfortable shipping this for one reason: it is my voice, cloned by me, with my consent, reading my own words, and labeled as what it is. The same capability pointed at anyone else's voice without permission is fraud, and the fact that the barrier is now one short recording means the burden of proof has moved. Voice alone no longer authenticates anyone. If your company still treats "sounds like the CEO on the phone" as authorization, this paragraph is your incident report from the future.

What it costs

After the one-time setup, the marginal cost of narrating a new article in six languages is a few hours of background GPU time and the electricity behind it. There are no per-character API fees, and no article text ever leaves my machine. Hosted voice platforms clone with more polish, and for commercial products with narration at the core they are worth evaluating, but for a personal site the local route wins on cost, privacy, and control, and it turns the whole feature into a couple of scripts in a repo.

Every article here now has a player at the top. Pick a language, press play, and read along: you will hear this text roughly the way I would read it to you, give or take an accent I decided to keep and the occasional syllable the model is still learning to land.