Orbit rollout architectures

The naive loop: generate a full batch, train, push the full model, repeat — with no overlap. The two GPU pools take turns, so each sits idle while the other works. And a batch can't close until its slowest sample finishes, so fast slots stall at the barrier (hatched). That straggler waste, plus the serial structure, is the motivation for everything that follows. Rollouts are on-policy (staleness 0).

Hypothesis: maybe the heavy full-model push is what makes Sync RL slow — so swap it for a tiny OFT adapter push and keep everything else the same. The push does get cheap (warm update_weights ≈ 0.105 s), but the loop is still serial: the trainer waits for rollout generation, offload, onload, and the adapter sync before it can step. Warm train wait stays at ≈ 7.142 s, and warm step time is ≈ 8.651 s — not meaningfully better than full-weight sync, because the push wasn't the bottleneck; the serial loop was. This is the panel that motivates overlap (the next step).

A background producer keeps generation in flight while the trainer takes gradient steps — they now overlap, so the trainer stays busy and slots never wait at a barrier. But the push is still the full model, so each update_weights() forces a long pause where the engine can't accept work. Pushes are too expensive to do often, so the engine lags the trainer by several versions (v0 then jumps to v2). That gap between the gen colors and the current version is staleness (≈ 2 here).

Only LoRA/OFT deltas cross the wire, so the push is tiny. With a single adapter slot, UpdateWeightFromTensor.update_weights still has to pause_generation → flush_cache → overwrite → continue_generation, but the stall is brief instead of a full-model pause. Cheap syncs can run every step, so the engine keeps up with the trainer and rollouts stay fresh (staleness ≈ 1).

All three stacked. The new adapter is broadcast into the inactive slot while the active slot keeps serving in-flight requests, so the long sync no longer blocks generation. Orbit still wraps the swap with the standard updater lifecycle — pause_generation → flush_cache → activate → continue_generation — so there is a brief pause at the flip point, but the activation itself is essentially instant (no tensors crossing the wire). Measured: warm update_weights drops from 0.169 s (single-slot) to 0.118 s (double-buffer), a ~30% reduction, and the knock-on effect is what matters — warm rollout throughput climbs +50.2% and warm train wait drops -81.2%. Gens that span a flip keep their start-version (older color); that residual is the staleness the fully_async metrics track (≈ 1).