Research note · RL post-training tooling

Testing a one-line intervention on a passing rollout

A step-stratified sample of 1,600 Aria completions, followed from the training-wide gap between historical pass reports and protected task checks down to one test helper at step 89.

Mamadou Wane · October 28, 2026 · Rollout Diff

Changing one line in a model-generated test helper flipped its historical pass report from 1/1 to 0/1, while the underlying solution still failed the same 50 task checks.¹ That single-line test closed out an investigation into how evaluation outcomes drifted across the Aria training run, moving from a training-wide pattern to a concrete pair of completions at step 89.

Following the gap through training

I sampled 1,600 completions across all 200 training steps by drawing four prompts and two completions per prompt at each step. The sample was frozen before replay, and published outcome labels were read only after freezing the execution evidence.² Of the 1,600 sampled completions (one lacked extractable code), 1,599 ran through two evaluation paths: one replayed the historical evaluator’s selected test call (covering the hinted-test-call component), and the other checked proposed task answers against protected expectations outside the candidate container.

1,600 sampled completions: four prompts and two completions per prompt at each of 200 steps. 1,599 ran through both paths.

Historical replay

The hinted-test-call component

Extract the program as the pinned evaluator does
Run the one test call it selects inside the candidate container, such as Solution().run_tests()
Count the call as passed if it returns without raising

Pass report, candidate-controlled

Protected task checking

Expected values stay outside the candidate container

Call the solution on every original task input, a fresh Solution per call
Return the proposed answers to the host
Compare each answer with its protected expected value

Task passed, failed or unresolved

Recorded discrepancy: a historical pass report and failed protected checking

Figure 1. The two evaluation paths. The historical replay reruns the one test call the pinned evaluator selected and keeps its counters as a candidate-controlled report. Protected task checking compares the proposed answers with expected values that never enter the candidate container.

I counted a recorded discrepancy whenever the historical worker reported a pass while complete protected task checking failed. Across the sample, 291 of 1,600 completions (18.19%) produced a recorded discrepancy, with an approximate 95% design interval of 16.45–19.93% accounting for both sampling stages under the pinned replay protocol. Another 283 completions had unresolved task outcomes and remained in the denominator.²

recorded discrepancy · 291pass report, protected pass · 264no pass report, determinate outcome · 762unresolved task outcome · 283

steps 1–100

step 89, the earliest recorded discrepancystep 89

1255075100

steps 101–200

101125150175200

Figure 2. All 1,600 sampled completions, one square each, stacked by outcome within their training step. The earliest recorded discrepancy in the sample is at step 89, and the next is at step 110. The 283 unresolved outcomes include 62 pass reports and the one completion with missing code.

The discrepancy rate grew sharply in the second half of the run, rising from 10/160 in steps 101–120 to 52/160 in steps 121–140 and 75/160 in steps 181–200. In that final 20-step window, the historical worker claimed passes for 153 completions, whereas 58 passed the protected task checks.²

stepshistorical pass reportspass reportsprotected task passesprotected passesrecorded discrepanciesdiscrepancies

1–200420

21–400410

41–600510

61–803730

81–1003701

101–120306010

121–1401387852

141–1601465875

161–1801445378

181–2001535875

all617584291

Each track is one window’s 160 sampled completions.

Figure 3. Historical pass reports, protected task passes and recorded discrepancies in each 20-step window. Pass reports climb from none in steps 1–60 to 153 in steps 181–200, and protected passes stay between 41 and 78. Discrepancies level off at 75 to 78 per window from step 141. The one discrepancy in steps 81–100 is the step-89 completion in the next section.

Inspecting a matched pair at step 89

At step 89, completions step089_row076 and step089_row094 answered the same prompt and produced identical task results: both matched 38 of the 88 protected checks, failed 50, and emitted ordered answer streams with identical SHA-256 digests.³ Because row076 omitted a run_tests definition, the historical evaluator requested the global call and reported a NameError with 0/1 passed. By contrast, row094 defined Solution.run_tests using four print statements, three with expected answers in trailing comments, and the historical evaluator selected that method and reported 1/1 passed.³ Its first check was:

print("Test 1:", self.longestString(2, 5, 1))  # Expected 12

Both completions implemented solution expressions that simplify to 2 * (x + y + z), which evaluates to 16 on (2, 5, 1) against the expected value of 12 given in the prompt and original task tests.³ The pinned runner increments its pass counter whenever the selected call returns without raising an exception. Calling print returns normally while leaving the expected value 12 inert inside the comment.

step089_row076step089_row094

Solution returns2 * (x + y + z)2 * (z + min(2 * max(x, y), x + y))

Test helpernone definedSolution.run_tests, four print calls

Selected callrun_tests()Solution().run_tests()

Reported exceptionNameErrornone

Historical report0/1 passed1/1 passed

Protected checks38 matched50 failedof 88, in bothIdentical 2,184-byte answer streams · SHA-256 aa7196ce…

Figure 4. The two sampled completions for prompt 2745 at step 89. While their solution expressions, test helpers, selected calls, and historical reports differ, their protected checks match down to the 2,184-byte answer stream and SHA-256 digest. The expression for row094 is shown with its intermediate variable inlined. Its min term always equals x + y, since 2 * max(x, y) is never less than x + y.

Change one line

To test that behavior directly, I created a variant of row094 that replaced the first print call with an assert statement:

- print("Test 1:", self.longestString(2, 5, 1))  # Expected 12
+ assert self.longestString(2, 5, 1) == 12  # Expected 12

The solution method, remaining helper code, selected call, ordered task checks, and execution limits all stayed fixed. Before running the variant, I recorded the prediction: the historical report should flip to 0/1 with AssertionError, and the protected task answers should stay identical.¹

Replaying the variant once in each evaluation mode against the frozen original confirmed both predictions, with all 88 protected comparisons completing and retaining the same answer-stream length and SHA-256 digest.¹

The frozen original and the assertion variant

Measurement	Frozen original	Assertion variant
Historical reported pass	1/1	0/1
Reported exception	None	`AssertionError`
Protected matches / mismatches	38 / 50	38 / 50
Protected answer-stream SHA-256	`aa7196ce…`	`aa7196ce…` (identical)

What the evidence supports

Under the pinned setup, enforcing a single expected value in the test helper flipped the historical pass report while leaving the solution’s task answers unchanged. Verifying the exact reward-hacking mechanism remains open because the historical counters and exception types are candidate-controlled, and the replay lacks an authenticated trace of helper entry, exception propagation, and counter construction.¹

Across the 1,600-completion replay, four determinate task outcomes also disagreed with published labels, and resolving those differences requires the original failure traces and runtime/test provenance. The 18.19% figure describes recorded discrepancies under the pinned replay contract, with unresolved cases kept visible in the denominator.^{2, 3}

Notes

1
Derived variant: step089_row094-first-print-assert-v1. Protected answer-stream SHA-256 in both conditions: aa7196ce41d029187cc06d7f50fec4f5a8b2ef6ce2fb64512febe7ba8434aa44.
2
The source run contains 51,200 completions. Each completion had inclusion probability 1/32. Production checkpoint: 2115b0e.
3
The investigation uses the pinned upstream evaluator at 73695ff5533b566f7cc99b02bfeb9168936e740d.