Reinforcement learning for large language models is sensitive to tiny numerical errors. Even small discrepancies between the step that generates a model's outputs and the step that updates its weights can destabilize training or stall convergence. Engineers at Google say they have found a way to close that gap on Tensor Processing Units, or TPUs, so that the two systems agree exactly.

Why do reinforcement learning sampler and trainer probabilities differ?

The problem sits in how production reinforcement learning stacks are built. Sampling and training run through separate, specialized execution paths. The sampler picks each action token during incremental decoding and records its probability. The trainer later evaluates that same token in batched forward and backward passes to compute its current probability and gradients. Even when both systems hold identical weights, they can assign different probabilities to the same token in the same context. The schedule is on-policy, but the arithmetic is not, and the mismatch builds up over multi-turn conversations.

The cause is rounding. A reduction, which combines many numbers into one, is a tree of floating-point additions. Floating-point addition is not associative, so changing the order of the tree changes where rounding happens. The effect is easier to see in bfloat16, a format with less precision than float32. In the team's example, near the value 2.0, each small input is half the distance to the next representable bfloat16 number. Adding the inputs one at a time discards them, while grouping them first reaches the next value.

The same problem shows up throughout a model. Matrix products accumulate partial sums, normalization computes statistics, and tensor-parallel shards combine partial outputs. In the team's original stack, the sampler and trainer used different token-array shapes and different forward implementations. JAX, the numerical framework the team uses, specializes computations to their input shapes, and the training forward pass also runs inside a differentiated program. Together, those differences meant reduction order and intermediate precision were not consistently aligned. The weights and model equations were the same. The arithmetic was not.

To fix this, the team set a strict standard. Across sampler decoding, serving prefill and trainer forward passes, all three paths must produce bitwise-identical logprobs for every sampled action. The comparison looks at raw floating-point bytes, with the same rollout weights, token history and sampling settings, and no rounding or tolerance allowed.

The team added serving prefill as a middle checkpoint. A direct comparison between sampler and trainer can show that a gap exists, but not where it begins. With the middle step, engineers can run a two-stage check. Comparing sampler decoding with prefill tests whether the serving engine agrees with itself when it generates tokens incrementally and when it rescores the full sequence. Comparing prefill with the trainer tests whether the serving and training engines agree across compilation and differentiation boundaries.

What is Zero-TIM and how does it work?

Zero-TIM does not require merging the sampler and trainer into a single engine. Instead, the team uses a four-part approach. First, it locks the accumulation order inside each tensor-parallel rank using batch-invariant Pallas kernels, which are custom kernels written for the TPU. Projection kernels move through fixed blocks in a set order, and partial products accumulate in float32 before being cast to the output type. Second, the team orders cross-shard reductions. Each rank produces a partial output, and the partials are transported separately and then added in the same source-rank order every time. Third, it standardizes the logprob scoring step, so that decoding, prefill and training apply matching sampling transforms and the same log-softmax construction. The paths also preserve the sampled token IDs across environment turns, so every path evaluates the same history. Fourth, the trainer reuses the serving prefill computation. Custom gradient rules keep the serving kernels for forward evaluation, while a separate replica of the computation is used only to build the backward pass.

How was Zero-TIM tested?

The team tested the method on a synthetic FrozenLake environment. The benchmark ran for 200 training steps using a public Qwen3-8B checkpoint on 64 TPU v5p chips arranged in a DP8 by TP8 mesh. The recipe used a GSPO-token loss with leave-one-out advantages. The team compared three approaches: standard training with trainer-old logprobs and no sampler correction, token-level truncated importance sampling, and Zero-TIM.

Zero-TIM recorded zero mean absolute difference between sampler and trainer logprobs at every logged step in the test window, while its training solve rate increased. Standard training and truncated importance sampling both kept nonzero differences. The authors note that truncated importance sampling adjusts how those differences affect the update but does not remove the underlying mismatch. Zero-TIM also reached higher observed solve rates in these runs. The results come from single-seed training runs across FrozenLake grids ranging from 2 by 2 to 9 by 9, with up to five interaction turns, so they reflect one set of conditions rather than a broad survey.

The work builds on recent open-source efforts across the AI systems community. The authors cite vLLM with TorchTitan and RadixArk and LMSys's Miles as projects that have shown how batch-invariant kernels and bitwise alignment can prevent off-policy drift. Google's contribution is an open-source, modular JAX, XLA and Pallas implementation for TPUs within Tunix, which the company describes as its open-source JAX-native library for LLM reinforcement learning and post-training on Cloud TPU.

The team says the kernel patterns and three-way verification approach are designed to be modular, so other multi-host JAX and TPU serving and training pipelines can adopt them. The engineers say they are extending support to linear-attention and Mixture-of-Experts architectures. A preview of the current implementation is available.

Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.