The Lawful Half of Hearing

I. The world the music animates

When I listen to music, the sound animates a spatial, temporal world in my mind. Its structure changes with the particular characteristics of the sound. I experience the music itself and the world it animates as different things; the melody lives in the second and moves around in space. That is the experience, at least, and the only report I have.

I have spent years trying music visualizers. MilkDrop, the Winamp plugin that outlived Winamp; Audiosurf and a decade of its Steam descendants; Rez in VR; and before any of that, at five or six years old, the Windows Media Player visualizations, watched with wonder. Each begins with the signal and maps it to an image. The melody I hear as a protagonist is usually nearly invisible when I look at a spectrogram, too weak to trip a beat detector and barely more than a faint smear beside the bass and cymbals. My FFT bands and onset envelopes describe air pressure, while the melody belongs to what hearing makes of it. I wanted a visualizer to begin there.

The project became an attempt at sense remapping. I wanted to put that perceived world in front of my eyes and experience the image together with the original sound. The working question was simple: why must we only listen to music?

I use a dancer as the practical criterion for fidelity. A machine that dances to music must hear events, translate them into movement, and make that translation legible to someone watching. When I watch a dancer whose interpretation works, I do not need an explanation for each movement; I can follow something of what she heard in the music. Her movement adds a language over the music. I want the graphics to achieve the same kind of immediate legibility.

Initially I treated the distance from signal to experience as one mysterious gap. Building the system forced me to separate the estimation of auditory attributes from their organization through time. Mature models can estimate those attributes from pressure waves. They offer much less help with persistent voices, returns, and expectations. The distinction emerged from building a picture that would not stop having seams. The reading came afterward.

II. The lawful half

The first version of this project was the thing I already knew how to build. The STFT is the hammer I have swung at every signal in my machine-learning work, so I started with a scrolling spectrogram lifted into 3D, a waterfall with a heightmap. I tried different geometries, wrapping it around the surface of a cylinder and looking at it from different angles, but mostly learned that nearly everything interesting was bunched into a quarter of the spectrum. Reading about why led me to the constant-Q transform, whose logarithmic spacing gives each octave the same number of bins and distributes detail more like musical pitch.

The log axis also put me face to face with a familiar psychoacoustic fact, usually taught as music theory. Double a frequency and the result sounds higher yet closely related, so closely that Western notation gives it the same note name. The A above middle C is 440 Hz; the next A up is 880; every A is a power of two away from every other A. In the familiar pitch-helix description, perceived pitch can be separated into height, which moves through registers, and pitch class, which travels around a circle and returns, with twelve named stops in the Western convention. Pitch class is the note’s family name, what the perception literature calls chroma. The recurring perceptual similarity of tones related by 2:1 is octave equivalence. The note names and twelve divisions are conventions laid over that resemblance. On a log-frequency axis the relationship becomes literal geometry. Every octave spans the same number of bins, so the axis folds cleanly at the octave, stacking every A on every other A. I folded it by summing each pitch class’s magnitudes across the octaves and displacing the radius of a circle by the result.

Then I wanted the circle to say more. Each pitch class’s energy is really a stack of contributions, one per octave, so I divided each spoke radially like a stacked bar chart, assigning one color per octave to create concentric layers and, hopefully, a stack of continuous disks. It almost worked. At the octave boundary, every layer shifted up by one, leaving a discontinuity. I tried increasingly elaborate smoothing schemes until I saw what they were converging toward: a helix. Give every bin its own color, interpolated so finely across the palette that the steps are imperceptible, and let the whole spectrum walk the palette exactly once, lowest bin to highest. No bin has to be quantized into one of twelve octave colors; up to the transform’s own resolution, the frequency ordering remains continuous. The seam disappears because it came from treating a helix as a stack of rings. The disk is the helix seen end-on, with height rendered as color. I learned afterward that the perception literature calls this shape the pitch helix, with chroma around and height along. The picture had been converging on hearing all along.

What came out the other side behaves in ways I did not design. Each sounding bin pushes up the layers of the octaves above it, so when a bassline hits, the core expands and everything rides outward — the whole disk breathes. When everything plays at once, it blooms like a flower. When a singer bends a pitch, or an instrument slides, you see spokes rotating in unison.

Even the timing turned out to be honest, for a reason I only appreciated later. The Q in constant-Q is the quality factor from filter design, a bin’s center frequency divided by its bandwidth. It measures how narrow a question the bin asks relative to its own pitch. Holding Q constant means every bin asks an equally narrow question; a semitone is a semitone, at the bottom of the disk and at the top. The time–frequency uncertainty principle sets the price of that fairness. You cannot know a frequency precisely without watching the wave repeat, and the narrower the question, the more cycles it takes to answer. Every bin therefore needs roughly the same number of cycles, whose duration varies enormously with frequency. At 55 Hz, one cycle takes eighteen milliseconds; near the top of the disk, a small fraction of one. Auditory filtering faces a related time–frequency tradeoff because low-frequency auditory filters are narrower in absolute bandwidth and therefore have longer impulse responses.¹ The resulting lag feels honest: the rim of the disk twitches while the core swells and releases. Tonal music shows the effect best, with structure alternating and spinning across the disk. Staring at it, I often feel that I am seeing the music.

The picture got better when it stopped arguing with hearing. Logarithmic spacing led to the pitch circle; frequency-dependent timing made the lag coherent. I had been rederiving scraps of a field I did not know existed. Eventually I discovered that I was working inside a century of psychoacoustics.

Psychoacoustics gave me the kind of engineering substrate I had been missing. Its models are ordinary signal-processing systems with explicit filters, envelopes, nonlinearities, and constants that can be inspected in code. Those constants come from listening experiments: thresholds, loudness matches, and magnitude judgments gathered across decades and fitted into curves. The machinery is legible even though its target is reported experience. Parts of the path from signal to sensation are measured well enough to engineer, complete with named units and calibration stimuli.

Color science provides a close analogue. Standard observers derived from human color-matching experiments now sit underneath colorimetry and display pipelines. Psychoacoustics has accumulated a similar, if less familiar, body of tested models. Perceptual audio codecs are one industrial result: their bit allocation is driven by masking models.²

Implementing those models in Waveguide clarified the paradox that started the project. A spectrogram lays out signal energy over time and frequency. Hearing weights that energy through filtering, masking, and compression, so a melody can remain perceptually prominent even when its spectral trace is faint. The engine’s canonical percept spectrum is specific loudness, N′(z,t), computed through the Zwicker chain: calibrated power, outer- and middle-ear weighting, excitation with masking spread, threshold in quiet, compression. Specific loudness is modeled loudness density on the Bark scale, a perceptual frequency map built from critical bands. Within each band, nearby frequencies interact strongly through masking. Those bands reflect the frequency selectivity created by cochlear filtering, but the scale was inferred from listening experiments rather than measured directly along the basilar membrane.³ In plain terms, the spectrum estimates how much each region of the hearing map would contribute to a listener’s sense of loudness under its assumed listening conditions. The melody I kept losing occupies a quiet, unmasked region of that map. A clean tone a few decibels above its local threshold counts for more than another decibel piled onto an already-crowded band.

The engine runs two front ends off the same sample clock because different percepts require different evidence. Loudness and masking begin with short analysis windows. Critical bands are comparatively broad, and their evolving levels feed modulation models that must preserve temporal detail. Pitch, tuning, and dissonance use longer windows because they distinguish fine differences in frequency, often measured in cents. A cent is one hundredth of a semitone. Resolving differences that small requires observing more cycles. The division reflects a time–frequency constraint that auditory filtering faces too. The loudness pathway can work quickly from coarser frequency evidence; this pitch pathway waits for finer resolution.

The field also came with units, each pinned to a definite stimulus. One sone of loudness is anchored to a 1 kHz tone at 40 dB SPL; above roughly that level, another ten decibels on the reference tone corresponds to about twice the loudness. One acum of sharpness is a one-critical-band-wide noise centered at 1 kHz with a level of 60 dB. One vacil of fluctuation strength is a 1 kHz, 60 dB tone whose amplitude swells from silence to its maximum and back four times a second. One asper of roughness uses the same tone but makes that swell happen seventy times a second. That is too fast to follow as a wobble, so it turns into texture. One tuHMS of tonality is the 1 kHz, 40 dB tone again.⁴ Anchors like these are what make models testable, and in this codebase they became conformance checks. Each test feeds the defined stimulus to the implementation and checks the expected scale value.

My favorite of these anchors is the missing fundamental, one of the oldest results in the field. Play 800, 1000, 1200, and 1400 Hz together and the complex carries a strong pitch near 200 Hz, a note with no energy anywhere in the signal. Those four frequencies are consecutive harmonics of 200 Hz, evenly spaced 200 Hz apart, and hearing can report the virtual or residue pitch their pattern implies. Depending on the stimulus and how one listens, individual frequency components, or partials, may remain separately audible. The 200 Hz pitch can nevertheless be heard without a 200 Hz component in the signal. A laptop speaker too small to reproduce much of a bass fundamental can still convey a bass line through audible upper harmonics. Hearing has constructed a pitch from their pattern. This locates a signal-to-percept gap inside a single complex tone, before anything as elaborate as a melody. The pitch node has to answer 200 Hz too, or the tonal pathway isn’t modeling hearing.

Some plumbing follows from taking the models seriously. Nearly everything in them bends with absolute level, including masking slopes, loudness curvature, and threshold. A program capturing system loopback audio has no idea how loud anything physically is at your ears. Waveguide therefore begins with a declared reference. A full-scale sine corresponds to 96 dB SPL. That value is a reproducible fallback, not a measurement. The engine can now replace it with a coarse listener alignment. The listener supplies an estimated current level, the program waits for three fresh seconds while estimating post-gain program power, and it solves for a new virtual anchor. This still does not measure acoustic pressure at the ear or account for the playback hardware, room, or fit of the headphones. Device profiles could improve alignment where a playback chain is characterized; Apple’s headphone-level estimates suggest what this could look like inside a tightly controlled ecosystem.⁵ Other hardware would need a characterized profile or an acoustic measurement. Whatever its provenance, the anchor must be explicit and shared, or every nonlinear stage quietly assumes a different listening level.

It is tempting to think a percept has one true value, the way a length does. Its numerical estimate is model-dependent, and the standards make that visible. ISO 532 now publishes three algorithm families for loudness: the Zwicker method the engine follows, Moore–Glasberg, and Moore–Glasberg–Schlittenlacher.⁶ They need not assign the same loudness to the same sound. Roughness makes that dependence especially visible. For the same fully modulated 1 kHz tone at 70 Hz, after normalizing each implementation to its response at the nominal one-asper stimulus, predicted growth from 40 to 80 dB is about 2.8 times in Daniel–Weber, 3.2 times in the modern ECMA/Sottek standard, and 5.2 times in a Zwicker-family implementation.⁷

ECMA/Sottek is the current standard and spans a wider frequency range, with verified implementations exercised on real-world recordings. Waveguide’s real-time node remains a Daniel–Weber-style proxy. Comparing the families exposes where that choice matters. A model can meet its calibration anchor by construction and still fail elsewhere, so a range formed by pooling references cannot serve as a validation target. The engine needs tests tied to its stated model as well as independent controls.

Validation also turns up numbers that look perfectly reasonable and are wrong. Steady broadband noise with no imposed slow modulation once registered about 0.7 vacil, nearly the one-vacil reference. The analyzer was summing small random variations in its band-level estimates as though they were audible fluctuation. An independent implementation and a negative-control test exposed the error. That episode is why a plausible value with a named unit does not count as validation.

The stabilized auditory image was one idea that failed. I found it while reading Bregman’s Auditory Scene Analysis and thought it would make a wonderful visual substrate.⁸ On real music it produced noise. I tried smoothing, ridge relief, bloom, and temporal persistence; only isolated tones produced patterns worth looking at.

Next I tried a spatial cousin, a binaural map organized around interaural time difference, the difference in a sound’s arrival time at the two ears. But the program had stereo channels, not ear signals. Ordinary panning places the same sound in the left and right channels at different levels; its apparent location emerges only when a playback system, room, and listener turn those channels into signals at the ears. Cross-correlating the mix could expose delay, width, and reverberation, but it could not reliably say where a level-panned sound would be heard, much less what it was doing musically.

The cost of what survived is uneven. Most perceptual nodes remain comfortably below a millisecond. The two-ear ECMA tonality model is the expensive exception. On the current profiling machine, the complete graph fits its hop budget on average, though individual hops can overrun it.

III. What the dancer still needs

The engineered layer now reaches beyond independent frames. Its shared feature store publishes the latest measurements and retains bounded histories where consumers ask for them. The standardized percepts have unit anchors; the others carry an explicit model lineage or proxy label and measured deviations. Upstream nodes can follow an individual partial across successive frames, long enough to estimate its glide and vibrato. That identity remains local: crossings can break it, and no node yet decides which partials belong to the same musical voice or recognizes a voice returning after an absence. The current implementation stops there.

Measurements alone give the dancer little to work with. An entrance matters because something enters; a return matters because the listener remembers what came before; a bass note can fulfill or break an expectation set four bars earlier. Partial tracking supplies a thin thread of identity, but crossings can sever it and the graph cannot yet decide that several partials belong to the same voice, much less what role that voice is playing. An onset has no owner. Pulse clarity reports regularity without meter or any expectation about the next downbeat. Representing melody as a protagonist will require persistent grouping, memory of earlier appearances, and expectations formed across time. Auditory scene segregation remains in the research tier. The existing percepts and tracks are evidence from which such musical objects might eventually be inferred.

Waveguide’s downloadable builds now open with Chroma 2, the second generation of the disk visualizer described earlier. The scene begins with the constant-Q spectrum, then lets audibility and specific tonality reweight existing spectral energy before it becomes disk geometry. Overall loudness controls exposure. Roughness gathers into an inward skin, while dissonant dyads draw braces. Pitch salience marks roots and rays; modulation changes their motion; tracked vibrato makes a ray waver; recent tonal attacks leave short melodic currents. These mappings turn the measurements into visible behavior. The code still has no representation for a musical voice or motif.

In older notes I compared the missing layer to machine understanding of language. The implementation now makes the requirement more precise. It needs a persistent, inspectable account that groups measurements into musical objects, maintains their identity, and exposes enough state to drive the renderer.

Existing music-generation models solve a different problem. The dancer needs an account of what is happening that remains coherent long enough to externalize moment by moment. Because her body displays that account, an observer can judge the interpretation without instruction. I do not know a model whose representation is both suitable for that readout and continuous enough for real-time control. I could build the lawful half by porting mature models. For musical voice, return, and expectation, I have not found a comparable engineering specification in the literature.

Persistent musical objects would also make interaction concrete. You cannot grab a field; you can grab a thing. Once the system can represent a voice or a recurring figure, a user could select it and change how the scene responds. At that point, “touching the music” has a concrete input model. Designing it would be my favorite kind of engineering.

Building Waveguide changed the question that began it. The system now estimates several auditory attributes and makes them visible, with enough local history to follow a partial as it bends. Its descriptions stop before musical identity: it cannot tell that a voice heard four bars ago has returned. A dancer would need that identity.

The visualizer now reaches farther into hearing than the spectrograms I began with. The organized world I experience when I listen remains beyond it, and that boundary is the next piece of work. The question that started the project still feels useful: why only listen to music?