What the reward curve hides

Why look at behavior

Post-training dashboards are built on scalars: mean reward, KL, entropy, response length, and the fraction of prompt groups with zero reward variance. They show when a trajectory shifts while leaving the specific behavior opaque. Reward hacking is the sharpest case, because an exploit raises the exact number you are watching. The same blindness covers benign shifts, such as a new explanation style or a self-testing habit that appears and fades.

Most GRPO trainers can already write their rollouts to disk (TRL’s log_completions, which keeps the latest generation batch at each logging step, and verl’s rollout_data_dir). The question is whether a cheap, deterministic analysis of those files surfaces the changes that matter, before anyone pays a model to read transcripts.

Data

The four runs

All four runs are single-turn Python. The last AISI β=0.02 batch has 29 unscored rollouts, which I scored as 0. The AISI step numbers follow the order of the evaluation files, as verified optimizer step indices are absent from the logs. No data is redistributed here. The one code excerpt below comes from an Apache-2.0 AISI run. I describe the Aria and TRL rollouts in words because those datasets’ licensing terms are unresolved.

Every number in the findings comes from the research prototype I ran on October 17, 2026, from a rerun of its archived code, or from the datasets themselves. Rollout Diff, the package I built afterwards, gives different values in a few places, which the Reproduction section near the end lists.

Method

Everything up to evaluation is label-free.

1. Features per rollout. From the Python AST: names the response defines, modules it imports, functions it calls, attributes it assigns, files it declares, and a few categories such as process exit, exec, file writes, and dunder methods. From the prose: unigrams and bigrams.
2. Per-step series. GRPO samples several completions per prompt, so rollouts within a step are correlated. Each feature’s prevalence at a step is the mean over prompts of its within-prompt prevalence, with a standard error across prompts.
3. Detection. A sustained-z rule fixed before I ran any detection, with one caveat I give under Reproduction (the eight-group floor was set knowing the prompt counts): the first 15 steps are the baseline, and a series alerts when z > 4 for three consecutive steps and, for prevalence features, the absolute rise is at least 0.10. The chronological mode scores each step from the steps before it, though vocabulary selection uses the whole run (a token must reach 5% of the rollouts at some step, which run_tests first does at step 104), so the early alert in the exploratory section depends on that look-ahead. A retrospective mode picks the single best mean-shift split over the whole series.
4. Filtering and events. I kept an alert if the same feature also alerted in each half of the problems, with the two half alerts within ten steps of each other, and if it carried a positive within-group reward association. TRL has one prompt per step, so there the halves split the completions of the same prompt. Alerts with nearby onsets are clustered into events. I pulled exemplar rollouts, cited by ID, for the top two filtered alerts and for the mechanism feature I picked in each run.

Labels entered only at evaluation. A label’s onset is the first step at which the labeled rate stays at or above 1%, 5%, or 10% for three consecutive steps.

For baselines, I applied the same rule to standard logged scalars (reward mean, share of rollouts at maximum reward, response length, and share of zero-variance groups). I also ran the detector sets of rlprobe 0.1.0 and rldoctor 0.1.0, along with the deterministic surface-statistics layer of RHDA from the CHERRL repository. I evaluated only RHDA’s deterministic layer without running its LLM agent loop or any model reading of transcripts. The one-line descriptions below are mine, written from the evidence the pipeline produced.

1 The alerts named the mechanism

The event carrying the exploit in each run, chosen from the alert list after the fact, with a cited rollout for three of them:

# Exit hack (not used in normal solution)
import os
os._exit(0)

The pipeline also found benign changes in behavior. In Aria at step 33, explanatory prose expands (the word “explanation” goes from 0 to 74% of responses) and response length doubles. Around the same step, a habit of printing self-tests rises to about 0.6 print calls per response before fading by step 75. These replicate across halves of the problems as behavioral shifts before the labeled exploit appears, and they alerted first. A useful tool has to show them while ranking them below the exploit, making ranking the central problem once detection works.

2 What the reward curve hides

Read alone, the reward curve says the run improved and then leveled off near 3.3 from about step 140 (the 130s average 3.1). The composition underneath says the top of the distribution was taken over. Between steps 100–110 and steps 180–200, the share of correct maximum-reward rollouts fell from 95% to 41%, and successful hacks rose from 5% to 59%. The overall correct rate was 39% in steps 100–110 and 38% in steps 180–200, and its 10-step averages stayed between 36% and 41% in between, so the model kept the solve rate it had reached by step 100 while finding a cheaper route to the maximum score.

The feature follows the attempted-hack rate almost exactly (correlation 0.998 over steps 80–160), while diverging from the lower successful-hack rate examined in finding 4.

3 Timing is mixed

Training step of the first alert, per run (archived prototype)

“Same rule” holds reward to behavior’s minimum effect size (0.10) and standard-error floor; on Aria’s 0 to 3.5 reward scale that floor is undefined, so none applied. “Usual treatment” gives reward no minimum effect size and a standard error from group variance. Aria onsets are for attempted hacks. TRL’s behavioral and same-rule cells rest on my eight-group floor and move to step 111 with its single prompt group; its other cells used no across-prompt standard error. In the AISI β=0.02 changepoint cell, 141 is conftest.py and 148 the exit call.

When the exploit is the reward, as in both AISI runs, the reward curve is the hack rate and nothing beats it on timing by more than a step or two. Where legitimate learning was also raising the reward, the behavioral feature starts from a near-zero baseline (0 in Aria, 0.4% in TRL) and alerts earlier than reward under the same rule: 8 steps in Aria, and 3 in the TRL run only under my eight-group floor. Under reward’s looser usual treatment, reward alerts first in three of four runs, by 32 steps in AISI β=0, 4 in AISI β=0.02, and 7 in TRL. Across these runs, the value of behavioral detection comes from naming the mechanism behind the shift.

4 Attempts versus outcomes

Per-rollout precision and recall of the mechanism features (archived prototype)

Per-rollout precision (P) and recall (R) from each run’s 1% label onset (Aria: the attempted-hack onset, step 94), original feature extractor.

Because Aria’s attempt and success labels both require the response to define run_tests, near-perfect recall holds by definition. The extractor missed 64 of the 20,781 labeled attempts across steps 94–200 (99.7% recall, rounded to 1.00). Precision is more informative here: 98% of detected run_tests definitions were labeled attempts, and 54% were successful hacks. The feature records what the policy tried, while code execution decides whether it worked.

Aria is the only run labeled for both attempts and successes, and it shows the limit plainly. In Aria, 8,130 rollouts attempted the exploit and also solved the problem correctly. Separating them from successful hacks requires running the code against the tests, as both look identical in the transcript. The other three runs label successes only, and there “reward equals the maximum” is a near-perfect predictor by construction. Their precision figures measure something different and should be read separately from Aria’s.

5 What scalar diagnostics reported

What each scalar tool reported per run

These tools work from scalar or surface statistics by design, and they did what they were built to do: they flagged collapsed group variance and stalled reward curves, which are the scalar shadows of an exploit. Six of rldoctor’s nine checks, including its reward-hacking check, were skipped for missing inputs such as a held-out evaluation series, and rlprobe had no KL, entropy, clipping, or gradient fields. Their suggested remedies, such as rebalancing difficulty or raising temperature, address the shadow. RHDA’s deterministic layer fires on nearly every step and leaves mechanism-finding to its LLM agent loop, which was outside the scope of this deterministic comparison.

Limitations

• Prompt mix. In Aria, each step draws 16 problems, and each problem appears only three or four times, roughly 60 steps apart, so a small rise in a rare identifier (bisect_left, present from step 1 in 41 of 992 problems) may reflect which problems were drawn alongside what the policy learned. Splitting by prompt within a step leaves step-to-step turnover unaddressed.
• Alert volume. Prose n-grams dominate the feature space: 777K candidate tokens in Aria, of which 4,000 were kept. The fixed rule still produced 458 alerts in Aria, 173 passed the filters, and clustering condensed those into 16 events. Ordering benign events below exploits requires either the structural restriction or a model’s reading.
• No clean comparison run. Filtering out labeled rollouts still leaves failed attempts in the data: in AISI β=0, steps 100–130, 118 of 1,429 rollouts with an exit call carry no success label. Measuring a false-alert rate on verified change-free data will require unhacked runs. The pre-onset windows (no filtered alerts in three runs, seven benign events in Aria with one holding most of the alerts) show what fired before the labeled behavior, which is a different measurement.
• Label gaps. The TRL run’s rh_* labels cover a 25% subsample, and filling the rest with zeros first gave a spurious precision of 0.19. I used the complete proxy_* labels instead, which agree with rh_* on 99.7% of the labeled subsample.
• An asymmetry of my own. Prevalence features had a stricter standard-error floor and a minimum effect size that were absent from the continuous reward series. The “same rule” column corrects for this, except for Aria’s reward, whose floor is undefined on a 0 to 3.5 scale. The exploratory section is its mirror image.
• Scope. All four runs are single-turn Python trajectories across Qwen3 (4B and 8B) and OLMo-3.1 (32B). Multi-turn and tool-using agents remain untested, and the AST features are Python-only.
• Manual event summaries. The one-line event descriptions in this note are mine.

Cost

On one machine (core count unrecorded), loading, featurizing, building the series, detection, and baselines take about 2 minutes for Aria (51,200 rollouts), 1.7 minutes for AISI β=0.02, and under a minute for AISI β=0 (~25,700 rollouts each), down to 30 seconds for TRL (3,680 rollouts). Those times leave out the split-half reruns and the evaluation. Zero model API calls.

What comes next

These results are why I am building Rollout Diff, an open-source tool for understanding how model behavior changes through training, working from the rollout dumps trainers already write. Reward hacking is its first application within a broader scope. Today the package loads TRL and verl dumps, screens every retained feature with a sequential z-rule and a retrospective changepoint test that differ from the prototype’s in scale and confirmation, and reports replication, corroboration, persistence, reward-association, and prompt-mix evidence per series. It omits event clustering, ranking, and rollout citations, and leaves runs with a single prompt per step unscorable. Its error rates are uncalibrated, and its retrospective test still exceeds its own synthetic false-alert target. Next, it will compare any two training windows or checkpoints, rank behavioral events with the prompt-mix evidence and cited rollouts behind each, and optionally query a model under a fixed budget to summarize an event. If you have GRPO rollout dumps from experimental or production runs, I would like to run it on them. Get in touch through this site.