Building Real-Time Creative Pipelines: TouchDesigner to the Web : ERDNBT

When developing interactive generative installations, one of the greatest challenges is bridging physical venue installations to the web.

How can we take signals generated in TouchDesigner or vvvv gamma and stream them into client-side browser canvases with low latency?

Protocol Comparison: OSC vs. WebSockets vs. WebRTC

For local hardware setups, Open Sound Control (OSC) over UDP provides nearly zero overhead. However, web browsers cannot natively open raw UDP sockets.

[TouchDesigner / vvvv] 
       │ (OSC over UDP)
       ▼
[Node.js Bridge Daemon]
       │ (Binary WebSocket / ArrayBuffer)
       ▼
[Browser Canvas / WebGL Shader]

Parsing Binary Buffers in the Browser

Instead of serializing parameters as JSON strings (which incurs garbage collection overhead at 60 FPS), packing floating point values into Float32Array buffers ensures zero-allocation parsing:

const socket = new WebSocket('ws://localhost:8080');
socket.binaryType = 'arraybuffer';

socket.onmessage = (event) => {
  const floatView = new Float32Array(event.data);
  const bass = floatView[0];
  const mid = floatView[1];
  const high = floatView[2];
  
  // Directly update Three.js shader uniforms
  shaderMaterial.uniforms.uAudioBands.value.set(bass, mid, high);
};

Mathematical Harmonic Decay

Audio transients decay following an exponential smoothing filter:

y[n] = \alpha \cdot x[n] + (1 - \alpha) \cdot y[n-1], \quad \alpha = 1 - e^{-\frac{\Delta t}{\tau}}

Where $\tau$ is the desired release time constant. This ensures visual fluidity without jitter.