Piano of Life — video → Game of Life → filtered audio

// per-column luma change (cyan = darkening/strike, red = brightening/return)

// piano roll

// shaded Game of Life

detection

Direction

Luma-change threshold 8.0

Direction bias 1.8×

Band center Y 0.58

Band thickness 0.05

Detection columns 60

Cooldown ms 260

Debug overlay

audio

GoL → filter (32 bandpass bands, log-spaced A2–C7)

Click ON to initialize Web Audio. Video will unmute; audio flows through the graph.

Dry (raw piano) 0.35

Wet (GoL-filtered) 0.75

playback

t: 0.00s hits: 0 hits/s: 0.0 peak strike Δluma: 0.00 FPS: 0 columns: 60

How it works

The pipeline is four stages, each one line of thinking:

Detection. Pick a thin horizontal band across the video where a resting hammer's felt tip sits. Split the band into N uniform columns. Each frame, compute per-column mean luminance. Frame-over-frame signed change tells you which columns just darkened (felt moved up out of the band → strike) and which just brightened (felt returned → return). Threshold + a per-column cooldown gives you clean discrete hits.
Piano roll. Each hit spawns a bar at that column and falls into the Game of Life zone below.
Shaded Game of Life. When a bar arrives at the GoL zone, it seeds a small cluster there. Standard Conway rules on the grid. Every cell also carries an alpha value: newly-alive cells snap to full alpha, cells alive for a while hold near 1, dying cells fade by ×0.55 per tick. This gives the grid a warm, continuous texture rather than binary on/off.
PMF → spectral filter. Sum shaded-alpha per column, then bin into 32 log-spaced bandpass filters between A2 (110 Hz) and C7 (2093 Hz). Each band's gain smoothly tracks its PMF value. The piano video's own audio track is routed through the bank via MediaElementAudioSource. When the Game of Life develops structure in a particular column region, the corresponding frequency band opens up on the audio — you hear those partials of the piano bloom.

The loop closes: piano strike → detection → substrate → filter over that same strike's decay → back to your ears. Stable GoL attractors give the piano a fixed color; chaotic GoL states morph its timbre in flight.

Try it with your own video

Drop a top-down piano video at the top. If the detection is off, adjust:

Band center Y until the gold band sits on the resting felt-tip line.
Band thickness — thinner = more precise, but tune the threshold together.
Threshold — idle noise should stay below the dashed line, real strikes should clear it.
Cooldown — if one strike fires multiple times, raise it.
Direction bias — set to strike (dark) with 1.8–2.5× to filter returns out.

For non-piano sources — anything with a well-defined bright-vs-dark interface that moves — this same detector will probably work: keys typed on a keyboard, hands over a xylophone, drumsticks on a snare, feet on a treadmill. The Game of Life and the filter bank don't care about the source.