Understanding SFT dynamics around the best pre-RL checkpoint

A recent study showed that when training reasoning models, the best SFT checkpoint for downstream RL can occur well before the checkpoint with the strongest SFT performance, with validation loss emerging as a surprisingly effective predictor of post-RL performance. In this post, I reproduce this observation on a much smaller language model and examine how the model evolves around the optimal pre-RL checkpoint. Along the way, I explore several training dynamics that may help build intuition for why validation loss within a training run has the capacity to predict downstream RL performance. Github link coming soon.

---

1. Background

Supervised fine-tuning (SFT) followed by reinforcement learning (RL) with verifiable rewards has become a canonical recipe for training reasoning models¹. During SFT, the model is trained on high-quality chain-of-thought² (CoT) traces, often distilled from stronger models, to acquire plausible reasoning behaviors. This is followed by RL, typically using GRPO or its variants, which reinforces reasoning trajectories that lead to correct final answers. Together, these two stages progressively reshape the model’s response distribution.

Viewed as part of the SFT→RL pipeline, the role of SFT is not to produce the strongest standalone model, but to provide the best initialization for downstream RL. This naturally raises the question of whether conventional SFT metrics, such as validation pass@1, are the right signals for selecting the checkpoint to hand over to RL.

This question was recently investigated by Kang et al.³, who showed that the checkpoint achieving the best downstream RL performance can occur well before the checkpoint with the strongest standalone SFT performance. They also found that validation loss on held-out reasoning examples closely tracks downstream RL performance. Motivated by this observation, I reproduce the result on a much smaller language model before examining how the model’s predictive distribution evolves throughout SFT. Throughout this post, validation loss refers to the standard token-level validation cross-entropy (CE), the default likelihood-based objective for autoregressive language models.

2. Experiment setup

I use allenai/OLMo-2-0425-1B⁴ as the base model and GSM8K⁵ as the training and evaluation dataset. For SFT, I generate think+answer chain-of-thought traces using DeepSeek-V4-Flash⁶, retaining only samples with correct final answers. Model checkpoints are saved periodically throughout SFT, from which I select five representative checkpoints for downstream RL evaluation. RL is performed using GSPO⁷ with 8 rollouts per group, two random seeds per checkpoint, and early stopping based on the validation reward.

3. The early peak of post-RL potential

As inexpensive proxy metrics, I compute the validation reward and validation loss for every SFT checkpoint. Figure 2 shows that the validation loss reaches its minimum substantially earlier than the validation reward reaches its maximum.

I then perform RL on five representative SFT checkpoints. Following Kang et al. ³, I also evaluate pass@64 on these checkpoints. The resulting downstream RL performance is averaged over two RL runs per checkpoint.

Step	validation pass@64 post-SFT	test pass@1 post-SFT	test pass@1 post-RL
step-0512	0.8797	0.2665	0.5306
step-6144	0.9203	0.3767	0.5692
step-12288	0.9172	0.4068	0.5451
step-24576	0.9094	0.4030	0.5458
step-36864	0.9047	0.3974	0.5318

The best downstream RL performance occurs well before the strongest standalone SFT checkpoint. Moreover, the checkpoint producing the best RL performance also coincides with the lowest validation loss and the highest pass@64, reproducing the central empirical observation by Kang et al. despite using a much smaller model.

The intuition behind pass@k as a predictor of downstream RL performance is relatively straightforward. It measures whether successful reasoning trajectories already occupy meaningful probability mass in the model’s output distribution, giving RL something useful to sample and reinforce. Larger values of k increasingly reward models that retain probability mass over multiple plausible reasoning trajectories rather than concentrating it on a single dominant one. Following Kang et al.³, I report pass@64 here and revisit later how this intuition is reflected over the course of SFT.

Validation loss is a less intuitive predictor. Unlike pass@k, it is computed from teacher-forced token likelihoods rather than sampled rollouts, making its connection to downstream RL less obvious. The rest of this post explores this question by examining how validation loss, rollout confidence, and pass@k evolve along a single SFT trajectory.

4. Characterizing the model dynamics

We first examine how the validation loss evolves throughout SFT. Since validation cross-entropy is the average negative log-likelihood assigned to held-out tokens, understanding the evolution of the underlying token probability distribution provides a useful lens for interpreting the validation loss. Figure 2 plots the distribution of teacher-forced conditional probabilities assigned to every token in the held-out validation set at three checkpoints: before, around, and after the point of minimum validation loss. Figures 3a and 3b show how the distribution changes between successive checkpoints.

At macroscopic level, two trends stand out. First, probability mass progressively accumulates toward the high-probability end of the distribution, with fewer validation tokens assigned intermediate probabilities. This can be interpreted as one manifestation of capability acquisition, whereby the model increasingly assigns high likelihood to held-out reasoning traces. Second, as training continues, the distribution becomes increasingly polarized: while the heavy right tail continues to grow, an increasingly heavy left tail also emerges as some validation tokens become extremely unlikely. Because the $−\log p$ penalty grows rapidly for low-probability tokens, this expanding left tail eventually outweighs the gains from the right tail, causing the validation cross-entropy to rise.

In aggregate, the behavior is consistent with the familiar onset of overfitting under cross-entropy training. Throughout SFT, optimization progressively concentrates probability mass around reasoning patterns represented in the training data. Early in training this concentration is largely generalizable, increasing the likelihood of validation reasoning traces. Later, it also causes some alternative validation token patterns to become increasingly unlikely, producing the growing left tail that eventually dominates the validation cross-entropy.

More importantly, the same trend is reflected in the model’s generated rollouts. Figure 5 shows that the model becomes progressively more confident in its generated reasoning throughout SFT. This suggests that validation cross-entropy captures not only changes in teacher-forced likelihoods, but also aspects of the evolving rollout distribution that downstream RL ultimately learns from.

These observations have different implications for downstream RL. Increasing probability assigned to correct reasoning trajectories should make useful behaviors easier to reinforce. Conversely, increasing confidence in generated trajectories, particularly incorrect ones, may reduce the likelihood and diversity of successful alternatives reachable during exploration. This intuition is echoed by the evolution of pass@k. Empirically, the checkpoint maximizing pass@k shifts progressively earlier as k increases, consistent with the less concentrated rollout distribution earlier in SFT.

Taken together, these observations suggest that validation cross-entropy captures more than the likelihood assigned to held-out reasoning traces. It is also sensitive to the progressive concentration of the model’s predictive distribution, which influences the rollout distribution that downstream RL learns from. This offers one possible intuition for why validation cross-entropy can track downstream RL performance better than pass@1 within a single SFT training run.

5. Conclusion

I reproduced the early pre-RL checkpoint phenomenon reported by Kang et al. on a much smaller reasoning model and examined the training dynamics around that checkpoint. The analyses suggest that validation cross-entropy reflects not only improvements in held-out reasoning likelihood, but also the progressive sharpening of the model’s predictive distribution, offering one possible intuition for why it can predict downstream RL performance well. Do note that they do not explain yet why the optimal balance between continued learning and predictive sharpening for validation loss coincides with the best pre-RL checkpoint. These observations characterize a single SFT trajectory rather than establish a general mechanism, but I hope they provide a useful lens for thinking about checkpoint selection before RL.

---

References