Skip to main content
Version: Next

Render WebGPU effects into published video

This guide shows how to render each frame of the published video yourself with WebGPU. Whatever you draw is what other peers receive.

The rendering code is plain WebGPU on both platforms. The integration differs:

  • On the web, render into a <canvas>, capture it with captureStream, and publish the stream with useCustomSource.
  • In React Native, use the /webgpu entry point of @fishjam-cloud/react-native-vision-camera-source. For every camera frame, your worklet receives the camera as a GPU texture and an output texture to draw into. The useVisionCameraWebGpuSource hook handles publishing, GPU synchronization with the video encoder, timestamps and frame lifetimes.

Publish a canvas​

If you already render with WebGPU (or Three.js, WebGL, or another canvas-based library), this is all the Fishjam-specific code you need:

const { setStream } = useCustomSource("my-canvas");

setStream(canvas.captureStream(30));

setStream comes from the same useCustomSource API as any custom source, and other peers receive the feed among their customVideoTracks.

The rest of this guide builds a worked example that renders the camera in grayscale. Any render pass works the same way: replace the shaders and keep the rest.

Browser support​

WebGPU is available in modern Chromium browsers, Safari 26+, and Firefox 141+. It requires a secure context (HTTPS or localhost). Always feature-detect with navigator.gpu and fall back to a WebGL/canvas path where it matters.

Example: the camera in grayscale​

The example has two parts: the shaders, and a component that sets up WebGPU, renders every frame, and publishes the canvas.

1. Write the shaders​

There are two shader stages:

  • The vertex stage draws a full-screen triangle, so the fragment stage covers the whole canvas. uv spans the visible area.
  • The fragment stage samples the camera at uv and weighs the color channels into gray.

The camera arrives as a texture_external, which TypeGPU cannot declare itself. The camera bindings are therefore written in raw WGSL, and sampling is wrapped in a WGSL-bodied tgpu.fn that the fragment shader can call. (On React Native, createCameraShaderBindings provides this recipe ready-made.)

import tgpu from "typegpu";
import * as d from "typegpu/data";
import { dot } from "typegpu/std";

// TypeGPU cannot emit `texture_external` bindings itself, so declare them in
// WGSL and sample through a WGSL-bodied `tgpu.fn` that TGSL shaders can call.
const bindingDeclarations = /* wgsl */ `
@group(0) @binding(0) var cameraTexture: texture_external;
@group(0) @binding(1) var cameraSampler: sampler;
`;

const sampleCamera = tgpu.fn(
  [d.vec2f],
  d.vec4f,
)(/* wgsl */ `(uv: vec2f) -> vec4f {
  return textureSampleBaseClampToEdge(cameraTexture, cameraSampler, uv);
}`);

// Full-screen triangle; uv spans the visible area.
const vertexMain = tgpu.vertexFn({
  in: { vertexIndex: d.builtin.vertexIndex },
  out: { position: d.builtin.position, uv: d.location(0, d.vec2f) },
})((input) => {
  const positions = [d.vec2f(-1, -1), d.vec2f(3, -1), d.vec2f(-1, 3)];
  const p = positions[input.vertexIndex];
  return {
    position: d.vec4f(p.x, p.y, 0, 1),
    uv: d.vec2f((p.x + 1) * 0.5, 1 - (p.y + 1) * 0.5),
  };
});

const fragmentMain = tgpu.fragmentFn({
  in: { uv: d.location(0, d.vec2f) },
  out: d.vec4f,
})((input) => {
  const color = sampleCamera(input.uv);
  const gray = dot(color.xyz, d.vec3f(0.299, 0.587, 0.114));
  return d.vec4f(gray, gray, gray, 1);
});
note

The shaders are written in TypeGPU (TGSL): typed TypeScript functions compiled to WGSL by the unplugin-typegpu plugin in your bundler (for Vite: unplugin-typegpu/vite). TypeGPU is not required; plain WGSL strings work the same.

2. Render and publish​

The component has five stages, marked in the code: get a GPU device, open the camera, build the pipeline, draw every frame, publish the canvas.

Note stage 4: camera external textures expire with the frame, so the loop imports the current camera frame and rebuilds the bind group on every iteration.

import React, { useEffect, useRef } from "react";
import { useCustomSource } from "@fishjam-cloud/react-client";

export function GrayscaleCameraPublisher() {
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const { setStream } = useCustomSource("my-camera");

  useEffect(() => {
    const canvas = canvasRef.current;
    if (!canvas || !navigator.gpu) return; // no WebGPU on this browser

    let disposed = false;
    let frameHandle = 0;
    let camera: MediaStream | undefined;

    const run = async () => {
      // 1. Get a GPU device.
      const adapter = await navigator.gpu.requestAdapter();
      if (!adapter) return;
      const device = await adapter.requestDevice();

      // 2. Open the camera and play it into a hidden <video>.
      camera = await navigator.mediaDevices.getUserMedia({ video: true });
      const video = document.createElement("video");
      video.srcObject = camera;
      video.muted = true;
      await video.play();
      if (disposed) return;

      // 3. Configure the canvas and build the render pipeline.
      const context = canvas.getContext("webgpu") as GPUCanvasContext;
      const format = navigator.gpu.getPreferredCanvasFormat();
      context.configure({ device, format, alphaMode: "opaque" });

      const bindGroupLayout = device.createBindGroupLayout({
        entries: [
          {
            binding: 0,
            visibility: GPUShaderStage.FRAGMENT,
            externalTexture: {},
          },
          { binding: 1, visibility: GPUShaderStage.FRAGMENT, sampler: {} },
        ],
      });
      const sampler = device.createSampler();
      const module = device.createShaderModule({
        code:
          bindingDeclarations +
          tgpu.resolve({ externals: { vertexMain, fragmentMain } }),
      });
      const pipeline = device.createRenderPipeline({
        layout: device.createPipelineLayout({
          bindGroupLayouts: [bindGroupLayout],
        }),
        vertex: { module, entryPoint: "vertexMain" },
        fragment: { module, entryPoint: "fragmentMain", targets: [{ format }] },
      });

      // 4. Draw every frame.
      const renderFrame = () => {
        if (disposed) return;
        // External textures expire with the frame; import and bind every frame.
        const bindGroup = device.createBindGroup({
          layout: bindGroupLayout,
          entries: [
            {
              binding: 0,
              resource: device.importExternalTexture({ source: video }),
            },
            { binding: 1, resource: sampler },
          ],
        });
        const encoder = device.createCommandEncoder();
        const pass = encoder.beginRenderPass({
          colorAttachments: [
            {
              view: context.getCurrentTexture().createView(),
              loadOp: "clear",
              storeOp: "store",
            },
          ],
        });
        pass.setPipeline(pipeline);
        pass.setBindGroup(0, bindGroup);
        pass.draw(3);
        pass.end();
        device.queue.submit([encoder.finish()]);
        frameHandle = requestAnimationFrame(renderFrame);
      };
      frameHandle = requestAnimationFrame(renderFrame);

      // 5. Publish what the loop draws.
      setStream(canvas.captureStream(30));
    };

    void run();

    return () => {
      disposed = true;
      cancelAnimationFrame(frameHandle);
      setStream(null);
      camera?.getTracks().forEach((track) => track.stop());
    };
  }, [setStream]);

  return <canvas ref={canvasRef} width={1280} height={720} />; // self-view
}

The same canvas serves as the self-view, so peers see exactly what you see.

Unlike React Native, the browser handles encoder synchronization, timestamps and frame lifetimes, so there are no surface pools or fences to manage. The camera also arrives as decoded RGB; no YUV handling is needed.

Going further​

Sampling without external textures​

Pipelines written against plain 2D textures can copy the video in with queue.copyExternalImageToTexture instead of importExternalTexture. This costs one copy per frame, but avoids texture_external bindings in your shaders.

Frame pacing and background tabs​

captureStream(30) samples the canvas at up to 30 fps. For exact pacing, use captureStream(0) and call track.requestFrame() after each render. To render only when the camera produces a frame, drive the loop with video.requestVideoFrameCallback instead of requestAnimationFrame.

Background tabs freeze the stream

requestAnimationFrame stops when the page is hidden, which freezes the published video for other peers. If streaming must continue in a background tab, drive rendering from a worker with an OffscreenCanvas transferred from the captured canvas.