The Lawful Half of Hearing

I. The world the music animates

When I listen to music, the sound animates a world in my mind. Not a metaphorical world — a spatial, temporal one: an evolving state on a structure whose topology is defined by the particular characteristics of the sound. There is the sense of the music, and there is the sense of the world the music animates, and they are not the same sense. The second one is where the melody lives. It wanders, travels, dances around in space — or at least that is the experience of the perception, which is the only report I have.

I have tried every visualizer there is. MilkDrop, the Winamp plugin that outlived Winamp; Audiosurf and a decade of its Steam descendants; Rez in VR; and before any of that, at five or six years old, the Windows Media Player visualizations, watched with wonder. Every one of them maps signal to image. And the signal was never the thing being heard. The melody I hear as protagonist is usually nearly invisible on the spectrogram when I actually look — not enough energy to trip a beat detector, a faint smear next to the bass and the cymbals. My FFT bands and onset envelopes don’t tell the story of a wandering melody, because they are stories about air pressure, and the melody is not a fact about air pressure. Every visualizer I had ever seen was visualizing air instead of hearing.

So the project was never “make graphics react to music.” It is sense remapping. I want to externalize the internal animation — put the world the music animates in front of my eyes — so that it can be re-perceived together with the original sound and produce a net-new perceptual experience. Perceiving perception. Why must we only listen to music? Why should we not see it too? And once an externalization faithfully maps the audio-cognitive elements to visual elements, the next question asks itself: why not touch it? Interact with the externalization under the rules of its own invented logic.

The criterion for “faithful” has a precise form, and it is not a similarity metric. It is a dancer. I am trying to make a machine that dances to music. It is no different from how we dance as humans: we hear things in the music, and without deliberation those events speak a language we translate into mechanical movement. Then we watch someone else dance and we understand it. We don’t ask why she moved that way in concert with that sound; it tickles something, because we hear what she hears and feel what she feels, and her embodied interpretation speaks a language over the music that enhances our perception of the whole. That is the bar. Not reactive graphics — interpretation, legible without explanation.

Initially, I held all of this as one problem: the signal on one side, the experience on the other, a single mysterious gap between them. I no longer believe that. There are two gaps, and they have very different characters — one from signal to sensation, from pressure waves to quantities that mature auditory models can estimate; another from sensation to perception, from those quantities to the melody-as-protagonist, to the animated world. The first is constrained, measured, and substantially modeled. The second is a much more open problem. I didn’t learn the difference from a book first. I ran into it, building a picture that wouldn’t stop having seams.

II. The lawful half

The first version of this project was the thing I already knew how to build. The STFT is the hammer I have swung at every signal in my machine-learning work, so I started there: a scrolling spectrogram lifted into 3D, a waterfall with a heightmap. I tried embedding it in different geometries — wrapped it around the surface of a cylinder, looked at it from different angles — and mostly what I learned was that nearly everything interesting was bunched into a quarter of the spectrum. Reading about why led me to the constant-Q transform: logarithmically spaced bins, equal resolution per octave, detail distributed more like musical pitch.

The log axis also put me face to face with a fact about hearing so familiar that nobody calls it psychoacoustics — we call it music theory. Pitch has a striking recurrence: double a frequency and the result is higher yet closely related, so closely that Western notation gives it the same note name. The A above middle C is 440 Hz; the next A up is 880; every A is a power of two away from every other A. In the familiar pitch-helix description, perceived pitch can be separated into height, which moves through registers, and pitch class — the note’s family name, what the perception literature calls chroma — which travels around a circle and returns, with twelve named stops in the Western convention. The recurring perceptual similarity of tones related by 2:1 is octave equivalence. The note names and twelve divisions are conventions; the resemblance is not merely a property of notation. On a log-frequency axis it becomes literal geometry: every octave spans the same number of bins, so the axis folds cleanly at the octave, stacking every A on every other A. I folded it: sum each pitch class’s magnitudes across the octaves, displace the radius of a circle by the result.

Then I wanted the circle to say more. Each pitch class’s energy is really a stack of contributions, one per octave, so I divided each spoke radially like a stacked bar chart — one color per octave, concentric layers, hopefully a stack of continuous disks. It almost worked. At the octave boundary there was always a discontinuity: a seam where every layer shifted up by one. I tried smoothing schemes to hide it, each more elaborate than the last, until I walked backwards into what the smoothing was converging toward. The limit was not a better blend between twelve colors. It was a helix. Give every bin its own color, interpolated so finely across the palette that the steps are imperceptible, and let the whole spectrum walk the palette exactly once, lowest bin to highest. No bin has to be quantized into one of twelve octave colors; up to the transform’s own resolution, the frequency ordering remains continuous. The seam is simply gone, because the seam was never in the sound. It was an artifact of treating a helix as a stack of rings. The disk is the helix seen end-on, with height rendered as color. I learned afterward that the shape has a name in the perception literature — the pitch helix, chroma around, height along — which suggested the picture had been converging on hearing all along.

What came out the other side behaves in ways I did not design. Each sounding bin pushes up the layers of the octaves above it, so when a bassline hits, the core expands and everything rides outward — the whole disk breathes. When everything plays at once, it blooms like a flower. When a singer bends a pitch, or an instrument slides, you see spokes rotating in unison.

Even the timing turned out to be honest, for a reason I only appreciated later. The Q in constant-Q is the quality factor from filter design: a bin’s center frequency divided by its bandwidth — a measure of how narrow a question the bin asks relative to its own pitch. Holding Q constant means every bin asks an equally narrow question; a semitone is a semitone, at the bottom of the disk and at the top. The time–frequency uncertainty principle sets the price of that fairness: you cannot know a frequency precisely without watching the wave repeat, and the narrower the question, the more cycles it takes to answer. So every bin needs roughly the same number of cycles — and cycles are not the same length. At 55 Hz, one cycle takes eighteen milliseconds; near the top of the disk, a small fraction of one. The treble bins answer in a blink; the bass bins need hundreds of times longer. Auditory filtering faces a related time–frequency tradeoff: low-frequency auditory filters are narrower in absolute bandwidth and therefore have longer impulse responses.¹ The rim of the disk twitches while the core swells and releases slowly, and that correspondence is why the lag reads as honest rather than sluggish. Bass genuinely swells; treble genuinely pierces. Tonal music stands out best; you can watch structure alternate and spin. Staring at it, you really do feel that you are seeing the music.

Every improvement in that sequence had the same character: the picture got better when it stopped arguing with hearing. Logarithmic spacing, the circular pitch domain, even the latency profile — each was a fact about ears that I met as a graphics problem. I was rederiving scraps of a field I did not know existed. When I found the field, it turned out to be a century old and better than I could have hoped.

Psychoacoustics occupies an unusual epistemic position. On one side it is ordinary signal processing — filters, envelopes, compressive nonlinearities, all implementable exactly, every constant printable and arguable. On the other side, those constants are experimental residue: thresholds, loudness matches, and magnitude judgments collected from human listeners across decades and compressed into curves. The models are legible the way physics is legible, but the thing they model is reported experience. Important parts of the gap from signal to sensation are lawful, measured, and mature enough to engineer — a century of listening experiments gave sensation its own units, its own anchors, and its own error bars. Color science occupies the same position: standard observers derived from human color-matching experiments underlie modern colorimetry and display pipelines, and that field’s lawful half is mapped so well that we forget the map describes experience rather than physics. Psychoacoustics is the colorimetry of hearing, and it has its industrial monuments: perceptual audio codecs turn masking models into bit allocation.²

Waveguide’s analysis engine now embodies that bridge, and the single most clarifying thing it did was dissolve my original paradox. The melody that was invisible in the spectrogram was never invisible to the ear, because the ear does not plot energy against frequency. The engine’s canonical percept spectrum is specific loudness, N′(z,t) — modeled loudness density on the Bark scale, an axis derived from critical-band measurements — computed through the Zwicker chain: calibrated power, outer- and middle-ear weighting, excitation with masking spread, threshold in quiet, compression. In plain terms: the model’s estimate of how loud each region of the hearing map would contribute for a listener under its assumed listening conditions, after its filtering, masking, and nonlinearity have had their say. In that representation the melody I kept losing is not faint. It occupies a quiet, unmasked region of the map, and the compression does the rest — a clean tone a few decibels above its local threshold counts for more than another decibel piled onto an already-crowded band.

The engine runs two front ends off the same sample clock, because the percepts want different things. Loudness and masking are computed from short windows: critical bands are comparatively wide, so coarse frequency resolution costs relatively little there, and the modulation percepts built on top — fluctuation, roughness — need the temporal speed. Pitch, tuning, and dissonance come from long windows and the constant-Q transform: they need cents, and cents take time. The same tradeoff roughly echoes listening — loudness reads fast and blurry, pitch precise and late — which I find reassuring rather than profound. When the constraints are the same, the solutions rhyme.

The field also came with units, each pinned to a definite stimulus. One sone of loudness is anchored to a 1 kHz tone at 40 dB SPL; above roughly that level, another ten decibels on the reference tone corresponds to about twice the loudness. One acum of sharpness is a one-critical-band-wide noise centered at 1 kHz with a level of 60 dB. One vacil of fluctuation strength is a 1 kHz, 60 dB tone fully amplitude-modulated four times a second; one asper of roughness uses the same carrier, level, and modulation depth at seventy times a second — too fast to follow as a wobble, so it turns into texture. One tuHMS of tonality is the 1 kHz, 40 dB tone again.³ Anchors like these are what make models testable, and in this codebase they became conformance checks: each test feeds the defined stimulus to the implementation and checks the expected scale value.

My favorite of these anchors is one of the oldest results in the field: the missing fundamental. Play 800, 1000, 1200, and 1400 Hz together and the complex carries a strong pitch near 200 Hz, a note with no energy anywhere in the signal. Those four frequencies are consecutive harmonics of 200 Hz, evenly spaced 200 Hz apart, and hearing can report the virtual or residue pitch their pattern implies. Depending on the stimulus and how one listens, individual partials may remain separately audible; the crucial result is that the 200 Hz pitch can be heard although no 200 Hz component is present. It breaks the obvious assumption about pitch — that you hear a note only because its frequency is physically there. You lean on such reconstruction constantly: a laptop speaker too small to move much air at a bass fundamental can still convey the bass line through audible upper harmonics. Pitch is already a construction rather than a reading — the signal-to-percept gap opening inside a single complex tone, before anything as elaborate as a melody. The pitch node has to answer 200 Hz too, or the tonal pathway isn’t modeling hearing.

Some plumbing follows from taking the models seriously. Nearly everything in them bends with absolute level — masking slopes, loudness curvature, threshold — and a program capturing system loopback audio has no idea how loud anything physically is at your ears. A sone is simply not computable without a listening level, so the engine declares one: full scale is defined as a 96 dB SPL sine, one constant, applied everywhere. It is a stand-in, not a measurement — nothing prevents it becoming a calibration knob for someone’s real setup — but it has to exist and it has to be consistent, or every nonlinear stage quietly disagrees about what level it is operating at.

It is tempting to think a percept has one true value, the way a length does. Its numerical estimate is model-dependent, and the standards make that visible. ISO 532 now publishes three algorithm families for loudness: the Zwicker method the engine follows, Moore–Glasberg, and Moore–Glasberg–Schlittenlacher.⁴ They need not assign the same loudness to the same sound. Roughness has at least three published model families, and their disagreement is not subtle: ask how much it grows from 40 to 80 dB and they answer anywhere from about 2.8× to 5.2×. A percept is defined by human judgments, and more than one defensible model can reproduce those judgments with different machinery — different auditory filters, different time constants, different calibration sets. A model can be exact at its calibration anchor by construction and still diverge from another model elsewhere. So the honest target for a real-time implementation is not one unquestionable oracle but a documented range of agreement among relevant references, and the engine aims to sit inside it rather than silently choosing whichever number looks best. That is the most a measurement can honestly claim when the published models do not coincide.

Validation turns up the opposite failure too — numbers that look perfectly reasonable and are wrong. For a while the fluctuation-strength node read about 0.7 vacil on steady, unmodulated noise, most of the way to the anchor value, where the textbook curve puts the percept at zero and an independent implementation reads essentially nothing. Nothing was fluctuating. The node was measuring the random wobble of its own envelope estimator and reporting the noise as signal, and from inside the output there was no way to tell — it took an outside reference to catch it. Sensation is lawful; it is not settled, and it is not self-certifying.

Not everything hearing-flavored earned its keep. I found the stabilized auditory image while reading Bregman’s Auditory Scene Analysis and thought it would make a wonderful visual substrate.⁵ On real music it was noise — and I tried: smoothing, ridge relief, bloom, temporal persistence. Only isolated tones produced patterns worth looking at.

I tried a spatial cousin next: a binaural map organized around interaural time difference, the difference in a sound’s arrival time at the two ears. But the program had stereo channels, not ear signals. Ordinary panning places the same sound in the left and right channels at different levels; its apparent location emerges only when a playback system, room, and listener turn those channels into signals at the ears. Cross-correlating the mix could expose delay, width, and reverberation, but it could not reliably say where a level-panned sound would be heard — and still less what it was doing musically.

What survived, though, costs almost nothing. Most of the sensation layer runs in well under a millisecond per hop; on my current profiling machine, the one full standard port in the graph — the ECMA tonality model — averages about one millisecond of a ten-millisecond budget and remains the most expensive node in the engine. Hearing’s lawful half is nearly free once you know the laws.

III. Where the dancer would have to begin

Here is precisely where the engineered layer ends. The feature store publishes frames: specific loudness on its Bark axis, sharpness, roughness, fluctuation, specific tonality, dissonance with its interval histogram, pitch salience with its candidates, tuning, a pulse-clarity scalar. The standardized percepts have unit anchors; the others carry an explicit model lineage or proxy label and measured deviations. But every one of them describes now. A frame is a fresh present. Nothing in the store is an object; nothing persists, returns, or intends. There is no node anywhere in the graph that says: this peak is the same voice that was here a moment ago. That is not an oversight. It is the boundary of the lawful half.

The dancer cannot work in the present tense. A dancer does not dance to fields; she dances to things — an entrance, a return, a promise the bass made four bars ago being kept or broken. The unbuilt parts of the roadmap draw this frontier in negative space: partial tracking, the birth and death of partials across time, is the first step from peaks to voices, and it does not exist yet; auditory scene segregation — Bregman again, this time as destination rather than detour — sits in the research tier; the pulse-clarity number is a declared proxy, a peakiness statistic on the loudness envelope, not a beat, not a meter, and it expects nothing about the next downbeat. Melody-as-protagonist needs all three: identity over time, grouping into streams, expectation. Sensation gives the dancer her senses. It does not give her anything to recognize.

The same boundary runs through what is on screen. The disk drives from the constant-Q transform and its octave fold; the percepts are consumed, today, by a debug dashboard and an offline analysis harness. The next act is clear enough: a scene whose geometry is animated by specific loudness and audibility — one that draws what the ear receives rather than what the microphone received. The disk already carries the sensation-shaped structure it stumbled into: the log axis, the circular pitch domain, the honest latency. What no scene can yet show is a voice.

My old notes ended with an observation about language models: that transformer LLMs are the closest thing we have built to machine understanding of a language, and that nothing equivalent exists for the language of music. I can now say what that intuition was pointing at, because the substrate clarifies the division of labor. Psychoacoustics is the tokenizer, not the language model. It estimates a set of intermediate descriptions from pressure — calibrated, lawful, cheap, and implemented here — but it does not read their musical meaning. What is missing is the thing that reads the token stream and maintains a world: voices with identity, motifs with history, harmonic expectation with tension and release. The engine’s peak lists and percept frames are a token stream in search of a reader. And the requirement on that reader is stricter than generation. Models that continue music plausibly now exist; that is fluency, and it is not what the dancer does. The dancer’s understanding is legible — her internal state is externalized in her body, moment by moment, and observers decode it without instruction. What this project ultimately needs is a model of musical listening whose internal world-state is stable, inspectable, and continuous enough to drive an externalization: understanding with a readout. I do not know of a model that meets that requirement. The lawful half was built by porting mature models; there are no comparably settled engineering laws yet for return and expectation, only literatures.

If that layer ever exists, the interaction thesis comes due with it. You cannot grab a field; you can grab a thing. The moment the externalized world contains objects — this voice, that pulse, this returning figure — touching the music under the rules of its own invented logic stops being a slogan and becomes an input-handling problem, which is to say an engineering problem, my favorite kind.

The claim I started with survives contact with the implementation, sharpened. Music animates a world in the mind, and that animation can be externalized and re-perceived — but the path runs through two gaps of different character. The first, from signal to sensation, contains lawful phenomena with units, anchors, and mature models; those models are now the working substrate of this codebase. The second, from sensation to the animated world, is a language problem, and it begins exactly where the store’s fresh present ends. The machine can model parts of hearing, in a strict and measurable sense. It cannot yet listen. Why only listen to music? — the question still stands, except that now it has an address: everything downstream of the peak list, everything that would let a machine dance and be understood without anyone asking why.