Skip to main content
Version: Next

Gemini Live Integration

info

This tutorial requires a working Fishjam backend. If you haven't set one up yet, please check the Backend Quick Start.

This guide demonstrates how to build a real-time speech-to-speech agent using Fishjam and Google's Multimodal Live API. By connecting these two services, you can create a low-latency voice assistant that not only listens to peers in a room and responds with natural voice, but also provides a real-time transcription of the conversation.

Overview​

The implementation acts as a bridge between two real-time streams:

  1. Fishjam ➡️ Gemini: The Agent receives audio from the room and forwards it to Google GenAI.
  2. Gemini ➡️ Fishjam: The Agent receives audio generated by Gemini and plays it back into the room.

To ensure these streams connect without audio glitches (garbled voice, wrong pitch), the audio sample rates must match between services.

Prerequisites​

You will need:

  • Fishjam Server Credentials: fishjamId and managementToken. You can get them at fishjam.io/app.
  • Google Gemini API Key: Obtainable from Google AI Studio.

Installation​

Since the Google integration is optional, you need to install the specific dependencies for your SDK.

First, ensure you have the Google GenAI SDK installed alongside Fishjam.

npm install @fishjam-cloud/js-server-sdk @google/genai

Implementation​

Step 1: Initialize Clients​

We provide a helper factory to initialize the Google Client.

import { FishjamClient } from '@fishjam-cloud/js-server-sdk';
import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';

const fishjamClient = new FishjamClient({
  fishjamId: process.env.FISHJAM_ID!,
  managementToken: process.env.FISHJAM_TOKEN!,
});

const genAi = GeminiIntegration.createClient({
  // Pass standard Google client options here
  apiKey: process.env.GOOGLE_API_KEY!,
});

Step 2: Configure the Agent​

Create a Fishjam agent configured to match the audio format that the Google client expects (16kHz and 24kHz on Gemini input and output, respectively).

import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';

const room = await fishjamClient.createRoom();

const { agent } = await fishjamClient.createAgent(room.id, {
  subscribeMode: 'auto',
  // Use our preset to match the required audio format (16kHz)
  output: GeminiIntegration.geminiInputAudioSettings,
});

Step 3: Connect the Streams​

Encoding

Fishjam handles raw bytes, while Google GenAI SDKs often expect Base64 strings. Ensure you convert between them correctly as shown below.

Now we setup the callbacks. We need to forward incoming Fishjam audio to Google, and forward incoming Google audio to Fishjam.

import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';
import { Modality } from '@google/genai';

const GEMINI_MODEL = 'gemini-2.5-flash-native-audio-preview-12-2025'

// Use our preset to match the required audio format (24kHz)
const agentTrack = agent.createTrack(GeminiIntegration.geminiOutputAudioSettings);

const session = await genAi.live.connect({
  model: GEMINI_MODEL,
  config: { responseModalities: [Modality.AUDIO] },
  callbacks: {
    // Google -> Fishjam
    onmessage: (msg) => {
      if (msg.data) {
        const pcmData = Buffer.from(msg.data, 'base64');
        agent.sendData(agentTrack.id, pcmData);
      }

      if (msg.serverContent?.interrupted) {
        console.log('Agent was interrupted by user.');
        // Clears the buffer on the Fishjam media server
        agent.interruptTrack(agentTrack.id);
      }
    }
  }
});

// Fishjam -> Google
agent.on('trackData', ({ data }) => {
  session.sendRealtimeInput({
    audio: {
      mimeType: GeminiIntegration.inputMimeType,
      data: Buffer.from(data).toString('base64'),
    }
  });
});

Troubleshooting Gemini API keys​

If your agent joins the room but silently does nothing — no audio, no transcription, and no error — the issue is often related to the Gemini API key.

API key errors surface asynchronously

createClient is a thin synchronous wrapper around Google's GoogleGenAI and makes no network call, so it does not validate your key — an invalid or unauthorized key is not rejected when you create the client. Unless you verify it explicitly (see below), the key is first exercised when live.connect() opens the Live websocket, and authentication or access failures then typically surface through the Live session callbacks (onerror / onclose) rather than as a thrown exception. (Other problems, such as a malformed config, can still reject the awaited connect() call — keep a try/catch around it too.)

To catch a bad key up front, call checkCredentials before connecting: it makes a single network request and throws if the key is rejected, so misconfiguration fails fast instead of silently. Still implement the onerror and onclose callbacks and log the close code and reason — that is where model-specific rejections (see below) and other Live failures appear:

await GeminiIntegration.checkCredentials(genAi);

const session = await genAi.live.connect({
  model: GEMINI_MODEL,
  config: { responseModalities: [Modality.AUDIO] },
  callbacks: {
    onopen: () => console.log("Gemini Live connected"),
    onerror: (e) => console.error("Gemini Live error:", e),
    onclose: (e) =>
      console.error(`Gemini Live closed: code=${e.code} reason=${e.reason}`),
    onmessage: handleMessage,
  },
});

Common reasons a key doesn't work​

  • Wrong or mistyped key, or a key from the wrong Google Cloud project.
  • The Gemini API is not enabled for the key's project.
  • Region or country restriction — the Gemini API (or its free tier) is not available in your region.
  • Model-specific rejection. The native-audio Live models (e.g. gemini-2.5-flash-native-audio-preview-*) can reject an otherwise-valid key: the websocket closes with code 1008 and the message "Your API key was reported as leaked. Please use another API key." (even when the key is not actually leaked), or with code 1011 internal errors. A key that works for other models can still fail here.

How to fix​

  • Verify or regenerate your key at Google AI Studio.
  • Confirm the Gemini API is enabled for the key's project and that your region is supported.
  • If you see the 1008 "leaked" message, rotate to a freshly generated key.