ProsodySSM

1Long-term acoustic continuity

Speech-to-speech models recompute the speaker from scratch on every call. Whatever the model inferred about a voice is discarded when the session ends, and a transcript keeps none of it: the fact that someone is quieter and flatter than they were on their last three calls does not survive transcription.

ProsodySSM is the long-term acoustic continuity layer for those models. The speaker is the state: each speaker owns one recurrent state that advances with every frame of their audio and persists between calls. A person your organization enrolls resumes their own state on the next call, and everything the system knows about them lives there. Every output is speaker-relative: louder, faster, higher than this person's own baseline; this moment is significant relative to this person. That state is also generative. It is a model of the voice itself, so identity is whose model explains this audio. Identity, continuity, presence, and significance are four readouts of one trajectory: who you are, the conversation so far folded into a state, where you are right now relative to yourself, and what this moment moved in you.

2Neural latent representations

Audio is encoded by a frozen Mimi codec into continuous 512-d latents at 12.5 Hz. The pre-quantization latent is used because a codec is trained to reconstruct: pitch contour, voice quality, timing, breath and room all survive, whereas a recognition front end is trained to discard exactly that variation. The frame rate matters as much as the content. At 12.5 Hz the recurrence steps 12.5 times per second of audio instead of ~100 for a 10 ms mel hop, and everything downstream compounds off that factor of eight.

3The voice clone

The state is a model of the voice, complete enough to rebuild how this person sounds. That is what makes identity a model-selection question: a voice resolves when its own state explains the incoming audio better than every competing state, and when no other state explains it at all. Difficulty is a property of the clip, so it cancels. What was said, the line it came over and how noisy the recording is read the same way for every candidate at once, and a bad connection stays a bad connection for all of them.

The invariant underneath is strict. Every accepted revision of a person's state, from the first to the current tip, has to decode back to that person's voice and to no one else's.

4One state per speaker

The trunk is six Mamba blocks at width 512, carrying one recurrent state per voice. That state is the whole of what the model holds about a speaker: for each of the six blocks, a rolling convolution window and a 128-wide SSM state per channel. Its size is fixed, so a step is constant time and constant memory: read the state, fold in one frame, write the state back. There is no window of past frames to attend over and nothing to evict, because everything the voice has said is already compressed into those tensors.

Selectivity is what decides which acoustic changes are written into the state and which pass through it. Mic distance, vocal effort and speaking rate move a voice around without making it a different voice, and the state is trained to carry the person while those move. Measurement is the difference between the state now and where that person's state sits at rest, so a voice that has drifted is still the same voice, reported as drifted.

Every lane departs from the same origin. A fresh lane starts at zeros, and two states differ only because the audio folded into them differs, so none of a person's identity comes from how their lane was initialized. All of it is in the trajectory.

Audio is the only clock. Every frame advances the recurrence; speech presence is a measurement with no authority to open a segment, gate an update or trigger a response, and a transport chunk is an amortization with no say in any of it. Each lane $\ell$ carries one state, and the model commits each frame $x_t$ to a lane, to a new lane or to hold:

$$h_{\ell }(t)=f(h_{\ell }(t-1),x_t)\text{iff }A(x_t)=\ell$$

The assignment $A(x)\in \{{\ell }_1,\dots ,{\ell }_k,\mathrm{new},\mathrm{hold}\}$ is decided by explanation: $A(x)=\ell$ when lane $\ell$'s clone accounts for the audio and every other lane rejects it, $\mathrm{new}$ when every lane rejects, $\mathrm{hold}$ otherwise. The invariant is exclusivity: each lane's state folds exactly one speaker's audio, for the life of the conversation and every conversation after it.

A flash on a lane is that speaker's state being advanced by what the model just heard. Lanes stay empty until a new voice opens one. Uncertain audio simply waits: a speaker's state advances only from audio that is decisively theirs, so what the state holds stays true.

Everything the model reports is a defined function of the waveform with a unit and a range: loudness in dBFS, pitch in Hz and its movement in semitones, voicing and pause ratios, spectral tilt in dB per octave. What the model has not measured, it does not report, and an interpretation laid over those numbers downstream never travels back up into identity, memory or the write gate.

5The micro-batch scheduler

A continuity layer has to hold a separate trajectory for every voice it is listening to and advance each of them in real time. Ordinary inference batching concatenates independent inputs; here every lane arrives carrying six blocks' worth of recurrent state and leaves carrying an updated copy, so the batch is assembled around that state. Within one voice, chunk t+1 needs what chunk t produced: that is the latency floor. Across voices, per-lane state is fixed-size and disjoint, so many lanes fuse into one GPU pass.

A lane is the state the model holds for one voice: the rolling convolution window and the SSM state for each of six blocks, 1.6 MB, and that number is identical in minute one and minute ninety. An attention model tracking the same conversation at the same depth keeps a key/value cache that grows with every frame it has ever seen. Capacity planning becomes a multiplication, and nothing is truncated or evicted as a call runs long.

Advancing one voice by one second measures 57 ms, and a single lane leaves the GPU almost entirely idle. That idle width is why fusing lanes is nearly free: more voices widen the pass the GPU is already running, and the pass costs about the same at 32 lanes as at one.

The ceiling follows: hundreds of voice-seconds advanced per wall-clock second on a single GPU, with the state for every voice held in a small fraction of its memory. Sharing a pass never means sharing a trajectory: each lane's state is sliced and copied out before it returns to its session. Because lanes are plain tensors, a dropped connection resumes into the same trajectory it left, and twenty uploaded recordings are indistinguishable to the scheduler from twenty live calls.

Per-voice serving cost is a function of how many independent trajectories can share one pass, and a recurrent model with fixed-size state is the rare architecture where that number has no natural ceiling.

6Across sessions

Within a session a speaker is a state lane. Across sessions a person is the lineage of that persisted state. An enrolled voice profile retrieves the person's persisted states from the tenant's index, and resolution asks which state explains the incoming audio. Known states preload as lanes before the first frame. An application-supplied user id selects the corresponding persisted state when present. A name labels the state lineage after the person supplies it or an operator sets it.

A lineage is its increments. Each accepted commit moves the state by ${\delta }_j$, and the state at any point along the lineage is the origin plus everything folded in since:

$$h_n=h_0+\sum _{j=1}^n{\delta }_j$$

The tip is the operational handle: the one state a call resumes from, advances and judges membership against. The increment is the measured object, and it is computable only inside the model that has been carrying this person's state, because it takes both a baseline and time to say how far an instant moved someone.

Alongside identity sit the saved moments: one record per significant moment, attributed to the person who produced it, keyed by what the moment moved in them, and carrying their state at that instant. The state decodes back to the voice, so every record is certifiably theirs. The transcript of a moment supplies its meaning; how far it moved that speaker against their own baseline supplies its weight. A textually ordinary "I'm fine" matters when it lands unlike anything this voice has done before. Recall ranks a person's history by how each moment moved them, matched against what this moment is moving in them now, and returns a primer an agent can condition on mid-sentence. A moment in a person's life, recovered from their speech alone, becomes a row you can query.