One Long Number

When you speak into a call, your voice leaves the microphone as a pressure recording and leaves the machine as something else: about fifty packets per second, each covering twenty milliseconds, most only a few dozen bytes long. On nearly every call in the world those packets are built by the same piece of software: Opus, the codec inside WebRTC, the machinery under Discord and WhatsApp and most of what we casually call “a call.” It is one of the most-executed perceptual models ever written. Here is what its packets contain.

A packet is not a miniature recording, and it is not a small spectrogram. Exactly one byte of it has fields you can read: the first.

Those five configuration bits choose between two different models of sound. SILK, descended from Skype’s speech codec, models sound as a voice: a pitched buzz from the larynx, shaped by the resonances of a throat and mouth. Modeling the speaker rather than the waveform is what makes SILK cheap on speech: a vocal tract is a few dozen slowly-changing numbers, while the waveform it produces is sixteen thousand samples a second. It is also why SILK alone can’t handle music: a cymbal crash has no pitch to track and no vocal tract shaping it, and a model built from one buzzing source and one filter has no setting of its parameters that produces one.

CELT, which came from Xiph.Org (the nonprofit behind Vorbis and FLAC), assumes nothing about what made the sound: it codes the spectrum itself, band by band, which costs more bits for speech but can represent anything — cymbals included. Between the two sits a hybrid mode that splits the signal at eight kilohertz and runs SILK below, CELT above. The encoder picks whichever representation fits each twenty-millisecond frame, and switches as the signal changes.

After that first byte there are no more fields. The rest of the packet is, in a precise sense, a single number.

The decisions doing that narrowing are the interesting part. In a SILK frame, the encoder writes down: how loud you are, as four gains covering five milliseconds each; the shape of your vocal tract, as sixteen numbers; your pitch, as a lag and a measure of how exactly the buzz repeats. Whatever those models fail to account for goes in last, as the excitation, the residual left over once the gains, the filter, and the pitch predictor have each subtracted what they can. The excitation takes most of the bits, and it has to: everything predictable about the signal has already been absorbed into the model parameters, so what remains is close to patternless, and patternless data is the one thing entropy coding cannot compress.

The sixteen vocal-tract numbers are worth a closer look:

CELT’s inventory matches its model. It transforms each frame to the frequency domain and splits the spectrum into twenty-one bands whose widths follow the Bark scale, an empirical map of the ear’s frequency resolution: fine at the bottom of the range, increasingly coarse toward the top. Matching the ear’s resolution means the codec never spends bits on distinctions no listener could hear. The first thing coded in every CELT frame, before any detail, is one coarse energy per band.

None of this structure is supposed to be visible in the bytes. The range coder’s job is to spend bits only on what could not be predicted, and its output, run perfectly, is statistically indistinguishable from random noise. Any visible pattern would be a prediction the model failed to make. This is why compressed data looks encrypted before any encryption has touched it. The vocal tract, the band energies, the pitch track: all of it is in the packet, but at no fixed offset, dissolved into the digits of one long number.

Even the one property the entropy coder cannot hide — how many bytes each packet needed — says more than it should: researchers have recovered spoken phrases from encrypted calls from packet sizes alone, and there is an RFC politely warning about it.

Whether the noise promise actually holds is easy to check: take one second of speech (a single word), encode it at three bitrates, and draw every byte as a pixel.

At 48 kb/s the promise is nearly kept: the panel is uniform speckle, and the only structure left is its outline. At 6 kb/s it isn’t. Horizontal bands run through the stack at the same byte offsets, packet after packet; the first bytes form a flat, unvarying line along the base; and the skyline itself, which shows how many bytes each packet needed, rises and falls with the word. The closer an encoder runs to the entropy limit, the less structure it should leave visible; at six kilobits, it is plainly leaving some.

But the visible leftovers are a curiosity, not the point. Whether or not anything shows in the bytes, everything cataloged above is inside every packet. A codec is a compiled theory of its signal: decades of speech science and psychoacoustics reduced to running code. A stream of packets is that theory applied, twenty milliseconds at a time: a vocal-tract shape, a spectrum drawn on the ear’s own bands, a pitch track. An Opus stream is not a compressed waveform; it is a description of one.

That reframes the phone call we began with. While you speak, the encoder on your side fits a fresh model of your voice every twenty milliseconds and writes it into the stream. The decoder on the other side uses the model to reconstruct sound, then discards it. And when a machine needs to analyze that audio, it decodes back to a waveform and computes a spectrogram, rebuilding from scratch a description that was in the packets all along.

I have wondered since 2022 whether the description could be read directly: whether a model could work from what the codec wrote instead of decoding it and starting over. The idea has precedent in other media: image networks have been trained on JPEG’s DCT coefficients rather than pixels, and action recognizers have worked from the motion vectors in compressed video rather than from decoded frames. For audio it remains nearly untouched, though there are places it would matter: anywhere sound is analyzed at scale, on a battery, or under a latency budget.

This August I started finding out. The results belong in a paper before they belong here, so that is the order they’ll arrive in.