Anticipatory Music Transformers

At the Music and AI hackathon, our team built duet.ai: you hum, sing, or play into a mic or a MIDI keyboard, and after four bars it locks onto your tempo and key and plays drums, bass, keys and lead along with you. The app has four interchangeable engines behind one interface: a rule-based pattern player, Google's Lyria RealTime, ACE-Step, and the one I worked on, an Anticipatory Music Transformer (AMT).

The app itself is w1ne/duet.ai, mostly Andrii's work; my proof of concept for the AMT engine is DuetDotAI. This post is about the model and about getting it to play live.

On latency

What does Latency even mean? Consider

1. Input latency. MIDI is ~1–3 ms. Audio input plus pitch tracking is 20–50 ms and brings its own failure modes.
2. Inference latency. 100 ms to 1 s+ (!).
3. Output latency. Soundfont / DAW buffer, 5–20 ms.
4. Musical latency. The model cannot react to a note that hasn't been played yet.

Only (4) is real. (2) is a scheduling problem: you hide inference behind a buffer of already-scheduled future notes. If the model is always writing a few beats ahead of the playhead and returns before the buffer drains, the audience hears zero delay. The model's think-time happens during music that is already committed.

The reference design here is ReaLJam (DeepMind / Mila, CHI 2025) (¹), which runs a transformer chord accompanist against a live pianist. Their protocol is worth stealing wholesale, and it has three parameters:

• lookahead: how far ahead of the playhead the model writes (~4 beats).
• commit: how much of that is frozen and can no longer be rewritten (~2–4 beats). Zero commit produces rapid unnatural chord changes; several beats of commit makes the plan stable under the player's fingers.
• listen first: how long the agent hears before it plays anything. Letting it listen 8 beats almost always made the first chord harmonic rather than dissonant.

What the Anticipatory Music Transformer is

AMT is Thickstun, Hall, Donahue and Liang's model (²), from Stanford CRFM. The checkpoint we used is stanford-crfm/music-small-800k: 128M parameters, a plain GPT-2 decoder, Apache 2.0, trained on the Lakh MIDI dataset.

Arrival-time encoding. A piece is a list of notes, and each note is three tokens:

001def make_event(time_s, dur_s, instr, pitch):
002    t = TIME_OFFSET + max(0, round(time_s * TIME_RESOLUTION))
003    d = DUR_OFFSET + min(MAX_DUR - 1, max(1, round(dur_s * TIME_RESOLUTION)))
004    n = NOTE_OFFSET + instr * 128 + pitch
005    return [t, d, n]

Onset time is absolute and quantized at TIME_RESOLUTION = 100 bins per second, one centisecond. Duration likewise, up to 10 seconds. The third token packs instrument and pitch into one symbol: 129 instruments × 128 pitches = 16512 note tokens. Context is 1024 tokens, so roughly 341 events.

Anticipation. This is the main contribution. A standard autoregressive model over time-sorted events can only condition on the past, so "here is a melody, write an accompaniment" does not fit: the conditioning information is spread through the whole timeline, including the future. The usual fix is a seq2seq setup with the conditioning prepended, which destroys locality: a control note at second 90 sits hundreds of tokens away from the output it should influence.

Anticipation instead places control events into the event sequence, each control ahead of its arrival time by an anticipation interval $\delta$, which is 5 seconds in the released models. So when the model is predicting events around time $t$, it has already read every control arriving up to $t + \delta$. Locality is preserved: a control and the notes it affects are near each other in the token stream, and the model is still an ordinary left-to-right decoder. Training is still next-token prediction, but infilling becomes a sampling problem rather than an architectural one.

For a duet this is almost exactly the right shape: accompaniment-given-melody is the infilling task the model was trained on. The mismatch is only that AMT was designed to infill offline, over a complete piece.

Constrained sampling

Sampling one event means sampling three tokens in sequence, each with a different constraint:

001for i in range(3):
002    prefix = [AUTOREGRESS] + history + new_token
003    logits = _cached_logits(model, prefix, cache) / temperature
004    idx = len(prefix) - 1
005    logits = safe_logits(logits, idx)
006    if i == 0:
007        logits = future_logits(logits, current_time - offset)
008    elif i == 2:
009        logits = _instr_mask_logits(logits, accomp_instrs, accomp_bias)
010    logits = nucleus(logits, top_p)
011    probs = F.softmax(logits, dim=-1)
012    new_token.append(int(torch.multinomial(probs, 1)))

safe_logits restricts each position to the token kind that belongs there (time, then duration, then note). future_logits forbids generating an onset earlier than the current time. _instr_mask_logits is mine, and it exists because of the first real problem.

Problem 1: the model free-associates

Given a sparse piano prompt, the base checkpoint sprays notes across a dozen-plus unrelated GM instruments. This is because Lakh MIDI is full of multi-track pop songs. The fix is to mask the note logits down to the melody instrument plus the companion voices, and add a fixed bias toward the companion:

001def _instr_mask_logits(logits, accomp_instrs, accomp_bias):
002    keep = torch.full((MAX_NOTE,), float("-inf"), device=logits.device, dtype=logits.dtype)
003    keep[MELODY_INSTR * 128:(MELODY_INSTR + 1) * 128] = 0.0
004    for instr in accomp_instrs:
005        keep[instr * 128:(instr + 1) * 128] = accomp_bias
006    logits[NOTE_OFFSET:NOTE_OFFSET + MAX_NOTE] += keep
007    return logits

The melody instrument stays unmasked deliberately. The model generates both parts jointly: its own guess at how the human will continue, and its accompaniment. We throw the predicted human part away. That's ReaLJam's trick, and it's what lets the model plan coherently without seeing the future. Since the melody continuation is discarded anyway, there is no coherence cost to biasing sampling toward the instruments we keep.

Problem 2: the model writes nothing

Even masked down, the model is free to spend an entire window "predicting" more piano and commit zero accompaniment notes.

The solution is accomp_bias. Sweeping it from 2 to 10 across several melodies raised average note density but individual runs still landed double-digit-second silences.

When a window comes back empty, ask again. This works since resampling doesn't touch the musical decision.

Problem 3: total silence lock-in

The problem: once a few windows commit nothing, the model has no precedent for that instrument anywhere in its growing context, and increasingly favours continuing its absence. Retrying doesn't help, because nothing about the prompt has changed.

A quick fix:

001escalated_bias = self.accomp_bias + SILENCE_BIAS_STEP * min(
002    self._consecutive_silent, MAX_SILENCE_BIAS_STEPS,
003)

Against the same stuck sequence afterwards: three silent windows, then it broke through and never got stuck again.

The scheduler

The core loop runs a real wall-clock transport. The melody is only ever revealed up to the current playhead, so the scheduler cannot peek at its own performer's future, even when the performer is a synthetic Python list (used for testing). A background thread continuously asks for lookahead beats of accompaniment past the last committed point, of which only commit beats are frozen:

001                  playhead
002                     |
003 ─── committed ──────┼── frozen (commit) ──┬── discarded ──┐
004                     |                     |               |
005                     └── the model is generating here ─────┘
006                         while this plays

The rest is thrown away and regenerated once more real melody has arrived. If the model misses its deadline, that's logged as an underrun rather than hidden, and after an underrun generation resumes at the current transport rather than spending the next calls on windows that are already in the past.

LiveDuet doesn't know where the melody comes from. SyntheticMelodySource and MidiKeyboardInput expose the same poll(playhead) / exhausted() interface, so the benchmark harness and a real keyboard drive identical code.

Problem 4: an unbounded session

Building the MIDI keyboard path surfaced a bug invisible in the 20-second synthetic demo: _maybe_kick_generation was handing the model's preprocessing (ops.sort / clip / pad, all O(history length)) the entire accumulated history every cycle. In a real session with no fixed length, generation time escalated 19 s → 36 s → 53 s per call as the session ran past ~100 s of music time. The model was drowning in a list most of which it couldn't even see, since _add_token only looks at its own trailing 1017-token window anyway.

Clipping the history to a trailing 90 seconds is half the fix. The other half is re-basing it to start near zero:

001window_start = max(0.0, gen_start - HISTORY_LOOKBACK_S)
002hist_snapshot = ops.clip(self.history, window_start, gen_start, clip_duration=False)
003if window_start > 0:
004    hist_snapshot = ops.translate(hist_snapshot, -window_start, seconds=True)

ops.pad pads silence from absolute time zero up to the first real event, so an old absolute timestamp keeps the O(session length) scaling even after the event list itself has been trimmed. Clipping alone doesn't fix it. Re-verified past 190 s of music time: generation stayed in the 0.3–1.3 s range, realtime factor 2.44×, 0 underruns, 297 companion notes delivered over a virtual MIDI port.

A related one: committed_horizon had no upper bound, so once the melody ended the loop kept pipelining lookahead windows with no melody left to inform them. A 19.1 s melody produced a 33.5 s MIDI file. Capping it at melody_len_s + tail_s gives 21.85 s, and finishes in 9.4 s wall time instead of 17.4 s with 13 inference calls instead of 21.

Examples

Two recordings of the AMT engine playing along. The accompaniment will start some time after the piano starts, as expected. In both examples, you can hear the AMT adapting to the player.

The numbers

On an M-series Mac over MPS, solo voice, 1.5–2.5 beat commit windows at 80 BPM: realtime factor 1.3×–3.6×, so net faster than real time, but with high per-call variance, 0.1 s to 9 s for similarly sized chunks. A standing buffer (priming the first chunk a full lookahead early, then continuous pipelining) absorbs most of that. A few underruns still happen on a loaded machine.

In the app, the same code ended up on a Raspberry Pi 5 running the model locally. At 100 BPM with four-beat windows each plan has a 2.4 s deadline:

Keeping GPT-2's KV state within a generated window, projecting only the final hidden state, and computing output weights only for the token kind currently being sampled gets a 128M model onto a Pi inside a 2.4 s deadline. Int8 quantization is the one that didn't pay: slower notes per second and 1.8 GB of RSS.

Footnotes

1. Scarlatos, Alexander, Yusong Wu, Ian Simon, Adam Roberts, Tim Cooijmans, Natasha Jaques, Cassie Tarakajian, and Cheng-Zhi Anna Huang. 2025. "ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformers." In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. doi:10.1145/3706599.3720227. Link ↩

2. Thickstun, John, David Hall, Chris Donahue, and Percy Liang. 2024. "Anticipatory Music Transformer." Transactions on Machine Learning Research. Link. Implementation: Link ↩