SGLang RFC Proposes Token Replay for Seamless Prefill-to-Decode Handoff

A Request for Comments (RFC #32143) published on the SGLang GitHub repository proposes an opt-in seamless handoff mode for prefill-decode (PD) disaggregated serving. Under current disaggregated serving architectures, prompt KV cache transfer creates a handoff bubble where Decode workers cannot proceed until the required KV state is received. The proposed protocol addresses this by allowing the Prefill worker to temporarily continue decoding and streaming tokens to the client while immutable prompt KV transfer remains in flight. Under this proposal, emitted token IDs are appended to a monotonic handoff log. The Decode worker receives the frozen prompt KV, replays the token log using short parallel extend operations to reconstruct the post-prefill KV tail, catches up with the Prefill worker, and atomically acquires sole output ownership at a specified token boundary. By substituting delta-KV migration with a compact token log and deterministic replay, the protocol aims to hide the remaining KV handoff tail when Decode replays faster than Prefill emits. Author updates on early test runs using an H20 testbed (1P1D setup with Qwen3-8B) indicated an exposed residual handoff tail of approximately 63 ms at 16K and 32K context lengths. Because short token extensions showed an initial 13–16 ms startup cost, the prototype added a seal-drain-commit sequence to prevent infinite catch-up loops during live token emission before ownership transition.

Key takeaways

• RFC #32143 proposes bridging PD disaggregated KV transfer latency by having Prefill workers temporarily decode while Decode workers catch up via parallel token replay.
• The mechanism replaces post-prefill delta-KV migration with a monotonic token log and deterministic short-extend replay, keeping immutable prompt KV transfer intact.
• Early testbed measurements on an H20 setup showed an exposed handoff tail of around 63 ms at 16K/32K prompt lengths, prompting the addition of an explicit seal-and-drain phase to ensure deterministic commit.
• Initial MVP limits include 1P1D setups, Mooncake transfer backends, identical model and KV configurations, greedy decoding, and default-disabled opt-in adoption.