One Live Stream, 80 Languages: Real-Time AI Translation over MoQ

You can watch the same live stream in Spanish, in Japanese, or in any of 80+ languages picked mid-stream from a menu, with a translated voice that stays in sync with the video instead of subtitles trailing three seconds behind.

That’s translate.fishjam.io. Publish from your laptop, open the link on your phone, pick a language, and hear yourself speaking a language you never learned, live.

The rest of this post is how it works. It’s shorter than you’d expect, because MoQ (Media over QUIC) did most of the work.

The Setup

We got access to Gemini’s 3.5 live translation model. You stream audio in, it speaks the translation back: one model, speech to speech, no transcribe-translate-synthesize pipeline to babysit.

Everyone does real-time transcription now. Some apps even translate conversations live. But it’s always a side channel: text scrolling next to the video, or a voice-over drifting behind the speaker. We wanted the real thing: a broadcast where the translated voice is the audio track, synchronized with the picture.

So the parts list is short:

• A web app with a publish page and a watch page.
• A Python service that feeds audio to Gemini and publishes the translation back.
• A Fishjam MoQ relay in the middle.

That’s the whole diagram. Everything is a client of the relay, so there’s nothing else in the picture: no backend, no signaling server, no socket between the browser and the service.

Subscribing Is Requesting

This is the part that sold us on MoQ.

The service watches relay announcements. New stream appears at brave-otter? It announces a companion broadcast at brave-otter/google/translation, where every track is a language code: es, pt-BR, ja. Captions live next door on es/transcript.json.

Now the fun part. None of those translations are running. They are created on demand.

When a viewer picks Spanish, their player subscribes to the es track. That subscription is the API call. The service sees it, opens a Gemini session, and starts publishing. When the last Spanish listener leaves, the session dies five seconds later.

There is no “start translating” endpoint. There is no config. The web app and the service never exchange a single message. They just agree on paths.

And the relay does fan-out, so a thousand Spanish viewers cost exactly one Gemini session. Model minutes are expensive; offering 80 languages costs nothing until somebody actually presses play on one. If nobody wants Japanese, Gemini never hears about Japanese.

The Buffering Trick

Translation has a physics problem: the model has to hear the sentence before it can say it in Spanish. Gemini takes about 2.5 seconds. No pipeline heroics will fix that: you can’t translate words that haven’t been spoken yet.

Play the translation as it arrives and the voice permanently trails the picture. The speaker points at a slide; the Spanish sentence about it shows up while they’re three slides ahead.

Our fix: stop fighting. Buffer everything. Video, original audio, all of it, just long enough for the translation to catch up. Nobody notices a few seconds on a live broadcast; “live” streams were never that live anyway. Each translation gets its own playback clock, offset by its measured delay, so the translated voice lands on the exact same play-head as the video.

And here’s the good part: we implemented all of this buffering client-side, in the viewer’s browser. That’s what makes synchronizing the original video with a translated voice possible at all. MoQ makes it almost boring: a subscriber owns its own playback clock, so holding a few seconds of media is a matter of choice. The publisher doesn’t know, the relay doesn’t care, and every viewer could pick a different delay if they wanted.

Now try that with WebRTC. Its whole personality is “play it as soon as possible”: the jitter buffer isn’t yours to command, playoutDelayHint is exactly what it sounds like, and holding several tracks at different offsets means decoding to a canvas and rebuilding audio/video sync by hand. You’d be writing a player from scratch to work around the one built in.

The buffer buys a second win for free: the video timeline never moves, so switching languages never touches the video: no freeze, no rebuffer, and the picture doesn’t even know you switched.

The switch itself: the new language warms up muted in the background, and we crossfade only when its decoder is actually producing audio, not when data has merely arrived, which happens a second earlier and would crossfade you into silence. The fade takes 200 ms, and the voice just changes mid-sentence.

Rebuilding the Timeline

The translator service is mostly audio plumbing, and the plumbing is mostly about time.

Gemini speaks in bursts: translated speech, silence while the speaker continues, another burst. Concatenate the bursts naively and your translated track drifts ahead of reality, and all that careful clock math above falls apart.

So the service rebuilds the timeline: it measures the gaps between Gemini’s responses and preserves them in the output track, resetting the encoder at each boundary so the codec doesn’t smear audio across a gap. The translated track ends up pausing roughly where the speaker paused. That’s what makes it line-up-able at all.

Captions get the same treatment. The service stamps each caption with the clock of the audio burst it came from, and the player holds it until the audio you’re hearing passes that stamp. Otherwise captions spoil the sentence before the voice says it.

Fun fact: captions follow the language you’re hearing, not the one you just clicked: during a language switch, the old captions keep running until the crossfade actually happens.

To Be Fair

• The first listener of a language waits a few seconds while the session spins up. Somebody always pays the cold-start tax. And the model itself warms up too: the first sentences are its roughest, and the quality climbs as it settles into the speaker.
• A few seconds of delay is fine for a broadcast, terrible for a conversation. This is a one-to-many design; don’t build a call on it.
• We only handle one speaker well. Crosstalk is a hard problem for any translation model, and today’s models handle it with… mixed enthusiasm.

Try It

The demo is live at translate.fishjam.io, and the code (web app and translation service) is in the Fishjam examples repo.

The pattern generalizes further than translation: anything that consumes a stream and publishes something derived (transcription, moderation, dubbing) plugs into the relay the same way, as just another client. The relays are part of Fishjam, so you can plug your own thing in today.

We’ve been circling this for a while: we built real-time transcription and a multi-speaker voice agent before. This one felt like the payoff.