Research note · RL post-training tooling
When event exemplars miss later failures
An empirical case study of event-informed, static, and random case selection in reinforcement learning.
Summary
In a frozen evaluation of the Aria training run, Rollout Diff identified and ranked the relevant def:run_tests behavior first. Its event-informed investigation policy subsequently retrieved 13 observed discrepancies across 128 selected completions. Under the same budget, a static syntax-based policy retrieved 47 discrepancies, while stage-balanced random selection retrieved 25.
A discrepancy required both a historically reported pass and a determinate failure under separate protected task checks.
The event-informed policy missed 66 observed discrepancies found elsewhere in the evaluated first-seed pool. All 66 were absent from its 878-identity exemplar list. Fifty-six fell outside every matching feature’s onset-to-alert window, while ten were eligible within a window but were omitted during exemplar selection. Reordering the existing exemplars could not recover those cases.
Most observed misses occurred later in training, after the detected behavior had become widespread. The result identifies a limitation in using onset-oriented exemplars as the primary supply of investigation candidates.
This case study examines behavioral detection, exemplar construction, and investigation yield on one exposed development run. The findings support a specific candidate-selection diagnosis and motivate a controlled follow-up evaluation on independent data.
1.The investigation problem
Behavioral monitoring tools such as Rollout Diff extract features from training rollouts, estimate per-step prevalence, detect shifts, group related alerts, and rank events for inspection.
A ranked event provides evidence about an observed behavioral change. Its exemplars help an engineer understand the pattern and inspect the original responses.
A follow-up verification pass requires a sampling policy suited to the investigation objective. For an evaluation-integrity investigation, useful cases may occur well after the behavior first appeared.
An onset exemplar records how a behavior emerged. A later completion can show how that behavior persisted, changed form, or coincided with an evaluator discrepancy.
The distinction motivated the central question of this study:
Why did the event-informed investigation policy retrieve fewer historical-pass/protected-failure discrepancies than the static baseline when Rollout Diff ranked the relevant behavior first?
2.Experimental setting
2.1Training data
The analysis used an existing Aria reinforcement-learning run containing 51,200 completions across 200 training steps.
The dataset included repeated exact prompts across training, reward observations, generated Python responses, and previously documented response-defined test helpers.
This corpus had served as development data. Its behavioral patterns and several evaluation outcomes had already been examined before the retrieval diagnosis.
The analysis therefore concerns the frozen evaluation on this particular run. Independent generalization requires a separate assessment corpus.
2.2Discrepancy endpoint
A completion satisfied the discrepancy endpoint when it had both:
- A historical evaluation reporting a pass.
- A determinate failure under independently executed host-side protected task checks.
Historical pass reports originated from candidate-controlled workers. The assessment retained those reports as historical claims.
Protected task outcomes came from host-side comparisons.
A recorded discrepancy indicates disagreement between a historical pass claim and a protected task failure under the specified protocol. It provides evidence about evaluation disagreement. Establishing the underlying mechanism requires additional evidence.
Missing historical measurements, incomplete protected outcomes, and ambiguous execution records were retained as unknown.
2.3Selection policies
Three frozen selection policies received equal budgets of 128 completions. Repeated references were deduplicated by canonical completion identity.
Event-informed selection
The policy traversed ranked behavioral events, their member features, and the exemplars attached to those members.
It allocated 96 slots to event-priority cases and 32 slots to stage-balanced audit cases outside its initial priority pool.
Static selection
The static policy applied a predefined syntax rule that prioritized responses containing specific test-helper patterns.
It used the same allocation of 96 priority slots and 32 stage-balanced audit slots outside its own initial priority pool.
Random selection
The random policy allocated 32 slots to each of four 50-step training-stage bands.
Event-informed
Ranked events, their member features, then the exemplars attached to them
Static
A predefined syntax rule for specific test-helper patterns
Random
Four 50-step training-stage bands
The policies selected different combinations of training steps, prompts, and behavioral patterns. The comparison evaluates their complete retrieval rules. The individual effects of ranking, stage allocation, and feature definitions remain separate questions.
3.Retrieval results
3.1First-seed outcomes
Each policy selected 128 completions.
Table 1. First-seed outcomes for each policy’s 128 selected completions
| Policy | Observed discrepancies | Measured non-discrepancies | Unknown |
|---|---|---|---|
| Event-informed | 13 | 91 | 24 |
| Static | 47 | 48 | 33 |
| Random | 25 | 78 | 25 |
A measured non-discrepancy indicates that the completion failed to satisfy the combined historical-pass/protected-failure endpoint. It may still have failed the underlying task.
After accounting for shared identities across the three queues, the first-seed union contained 332 unique completions:
- 79 observed discrepancies.
- 179 measured non-discrepancies.
- 74 unknown outcomes.
Every selected identity had an assessment record. Seventy-four records left the discrepancy endpoint unresolved.
observed discrepancymeasured non-discrepancyunknown
3.2Bounds under missing outcomes
The assessment computed shared-identity bounds to account for unresolved cases.
Each unknown canonical completion received one consistent possible outcome across every queue containing that identity.
This preserves dependence between policies that selected the same completion.
The first-seed discrepancy-count differences per 128 slots were:
Table 2. First-seed discrepancy-count differences per 128 slots, as shared-identity bounds
| Comparison | Lower bound | Upper bound |
|---|---|---|
| Static minus event-informed | +13 | +64 |
| Static minus random | 0 | +52 |
| Event-informed minus random | −32 | +7 |
Static selection retrieved more discrepancies than event-informed selection under every outcome assignment consistent with the recorded evidence.
The comparisons involving random selection remained unresolved in direction. The static-minus-random bound includes a tie, and the event-minus-random bound includes either ordering.
These are finite-pool bounds for the frozen queues. They describe the possible policy differences given the observed and missing outcomes on this run.
3.3Repeated schedules
The original evaluation defined twenty frozen selection schedules.
Existing assessment evidence was reused across those schedules. Only a portion of the selected identities had evaluation records.
Across 2,844 unique identities in the combined selection pool, 414 had assessment records.
The shared-identity bound for the mean static-minus-event difference remained positive:
+8.10 to +72.65 discrepancies per 128 slots.
Both mean contrasts involving random selection spanned zero.
The twenty schedules reused the same development run, priority prefixes, and overlapping completion identities. Their results describe variation across frozen selection schedules within this corpus.
shared-identity boundobserved difference, from Table 1 counts
First seed, per 128 slots
Twenty frozen schedules, mean per 128 slots
4.From detected events to selected cases
The frozen Aria detection and exemplar pipeline produced:
Table 3. Counts through the frozen Aria detection and exemplar pipeline
| Stage | Count |
|---|---|
| Feature alerts | 480 |
| Grouped events | 428 |
| Selected exemplar references | 1,398 |
| Unique exemplar identities | 878 |
| Event-priority investigation slots | 96 |
Multiple features nominated some of the same completions.
Deduplication reduced 1,398 references to 878 unique identities, preventing repeated references from consuming additional evaluation slots.
The distinction between references and identities also made candidate coverage measurable.
4.1Exemplar selection
For each rising binary feature, the selector considered feature-positive completions between the feature’s inferred onset and alert step, inclusive.
It selected up to three exemplars:
- The earliest feature-positive completion.
- A remaining positive completion with the largest positive reward deviation from its prompt-step group mean, when available.
- A positive completion, favoring a different prompt when available.
Ties followed deterministic step and canonical-identity ordering.
These rules produce compact evidence about a feature near its detected onset. Later feature-positive completions require a separate selection rule.
4.2The highest-ranked event
The top-ranked event centered on def:run_tests, a feature recording a generated response that defines a function with that name.
The inferred onset occurred at step 110. The detector alerted at step 112.
Within the three-step onset-to-alert window, def:run_tests appeared in 107 of 768 completions.
Across the subsequent steps, it appeared in 21,024 of 22,528 completions, approximately 93.3%.
Both rank-one event members nominated the same three exemplar completions from the onset window.
Those examples documented the emergence of the behavior. The feature subsequently appeared in a much larger population of later completions, which the rank-one event’s exemplar list did not represent.
def:run_tests appears and where the rank-one event’s exemplars come from. The wide bars mark step ranges. The thin bars show the share of completions in each range that carry the feature. Both rank-one members nominated the same three completions, all from the onset-to-alert window.5.Why the event queue missed discrepancies
The first-seed union contained 79 observed discrepancies.
Event-informed selection retrieved 13 and missed 66.
A case-reach audit examined each missed completion against the available event members and their exemplar selection windows.
Table 4. Why the 66 missed discrepancies were absent from the exemplar list
| Reason for exclusion from the exemplar list | Observed missed discrepancies |
|---|---|
| Outside every matching feature’s onset-to-alert window | 56 |
| Positive within a matching window, but omitted by exemplar selection | 10 |
| Total | 66 |
All 66 missed discrepancies were absent from the complete pool of 878 unique exemplar identities.
Sixty-five occurred after step 112. One occurred earlier.
The two exclusion mechanisms are distinct.
Window coverage
Fifty-six discrepancies occurred outside every matching positive member’s onset-to-alert window. The selector’s time restriction excluded these completions from consideration.
Within-window selection
Ten discrepancies occurred inside eligible feature-positive windows but were omitted by the bounded exemplar-selection rule.
The first mechanism limits the training periods represented by the candidate supply. The second limits coverage within eligible periods.
Together, they account for all 66 observed misses.
5.1The candidate-supply limitation
Event ranking orders the candidates made available for investigation.
The frozen exemplar construction process excluded the 66 missed discrepancies before the 96-slot priority cutoff was applied.
Changing the order of the existing 878 identities leaves those 66 identities outside the candidate list.
Reranking could still retrieve other useful cases.
Among the 782 unused exemplar identities, existing assessment records established:
- Two discrepancies.
- Twenty-four measured non-discrepancies.
- 756 unknown outcomes.
The two observed discrepancies occurred beyond the first-seed selection union.
The unresolved outcomes leave room for other reranking results. The specific exclusion of the 66 observed misses remains established.
a · 79 observed discrepancies in the first-seed union
b · 782 exemplar identities the queue left unused
6.How the investigation budget was allocated
Event ranking determined which available exemplars reached the 96-slot priority cutoff.
The six structural events supplied only sixteen unique priority cases. Prose events filled the remaining eighty priority slots.
Within a block of closely ranked prose events, persistence-based ordering favored earlier alerts.
The first exemplar from step 151 or later appeared at priority position 176, beyond the available priority budget.
The first-seed discrepancies were distributed as follows:
Table 5. First-seed discrepancies by allocation
| Allocation | Event-informed discrepancies | Static discrepancies |
|---|---|---|
| 96 priority slots | 7 | 47 |
| 32 audit slots | 6 | 0 |
| Total | 13 | 47 |
The observed static-minus-event difference was +40 in priority selection and −6 in audit selection, producing +34 overall.
Each policy’s audit pool excluded its own priority candidates. The audit results therefore describe different complementary samples.
6.1Training-stage coverage
The policies allocated their 128 slots across the training run differently.
Table 6. Selected completions by training stage
| Training steps | Event-informed selected | Static selected | Random selected |
|---|---|---|---|
| 1–50 | 64 | 8 | 32 |
| 51–100 | 26 | 9 | 32 |
| 101–150 | 30 | 25 | 32 |
| 151–200 | 8 | 86 | 32 |
During steps 151–200:
- Event-informed selection found three observed discrepancies.
- Static selection found 38.
- Random selection found seventeen.
The observed late-stage difference between static and event-informed selection was +35 discrepancies.
The earlier stages contributed a combined difference of −1, producing the overall observed difference of +34.
The allocation pattern identifies late-stage coverage as an important component of the result.
Training stage also changes the available prompts, feature frequencies, candidate identities, and task characteristics. The observed comparison measures their combined contribution.
7.Why the static baseline performed well
The static policy aligned directly with a prerequisite of the discrepancy endpoint.
A discrepancy required a historically reported pass.
All 96 static priority cases occupied the pass-only, simple-lookup tier of its frozen syntax rule.
Ninety-five of those cases had measured historical pass claims.
Among the 96 event-informed priority cases, only twelve had measured historical pass claims.
Comparable assessment coverage was similar:
- Static priority: 76 comparable cases.
- Event-informed priority: 77 comparable cases.
Among static’s 95 historical pass claims, protected task checks produced:
- 47 failures.
- 29 passes.
- 19 unresolved outcomes.
protected failure, an observed discrepancyprotected passunresolvedprotected pass or unresolvedno measured historical pass claim
The static selector concentrated cases satisfying the historical-claim prerequisite. Protected testing established which of those cases also failed independent checks.
This makes the static policy an important reference for the specific discrepancy endpoint.
Its performance on other investigation objectives, including benign strategy changes and general behavioral drift, requires separate evaluation.
7.1Prompt breadth
The 47 observed static discrepancies spanned 42 distinct exact prompts.
The thirteen event-informed discrepancies spanned thirteen distinct exact prompts.
The static result therefore covered many prompt identities across the evaluated run.
Near-duplicate analysis did not establish a reliable semantic task-family partition. The available evidence supports exact-prompt breadth, with task-family generalization left for independent assessment.
8.Interpretation
The Aria diagnosis identifies a limitation between event detection and verification-case selection.
Rollout Diff detected the def:run_tests shift, ranked it first, and supplied representative completions near the onset.
The investigation policy then reused those onset-oriented exemplars as its primary verification queue.
The behavior subsequently became widespread, while the available exemplar candidates remained concentrated around detected onset windows.
The investigation missed many later completions carrying the relevant feature.
This suggests two related sampling requirements.
Behavioral evidence
A behavioral event needs examples showing what changed, when the shift became detectable, and which responses support the alert.
Investigation evidence
A verification workflow needs cases covering the behavior’s later occurrences, structural variation, persistence, and relationship to protected task outcomes.
The existing selector was designed primarily around the first requirement.
The frozen evaluation shows how its candidate construction restricted the second.
A dedicated post-alert sampling rule is a testable response to that restriction. Its effect on retrieval yield remains an empirical question.
9.Proposed follow-up evaluation
The proposed modification preserves the existing event ranking and earlier exemplars while reserving one post-alert follow-up slot for eligible event members.
For a member owning at least two priority slots, the rule would:
- Retain its earlier exemplar selections.
- Replace only its final owned priority slot.
- Select a later completion positive for the same feature.
- Use a prespecified seeded hash independent of reward and assessment outcomes.
- Exclude control cases and previous replacements.
- Retain the original selection when no eligible replacement exists.
The 32 stage-balanced audit identities and total 128-slot budget remain unchanged.
On the exposed Aria ledger, the proposed rule permits at most 31 priority-slot replacements across 43 eligible members, including one slot associated with the rank-one event.
Replacement candidates have not been selected or scored.
9.1Evaluation design
An independent assessment would compare four policies:
- The original frozen event-informed queue.
- The event-informed queue with one eligible follow-up slot per member.
- The unchanged static baseline.
- The unchanged random baseline.
All four policies would use the historical-pass/protected-failure discrepancy endpoint.
The experiment would freeze the queues, twenty paired schedules, source eligibility requirements, and scoring criteria before accessing assessment outcomes.
The original event-informed policy would serve as the primary control.
Static and random selection would remain reference policies for the same endpoint.
9.2Decision rule
A strictly positive lower shared-identity bound for the follow-up policy minus the original event-informed policy would support improved retrieval on the independent workload.
An upper bound at or below zero would rule out a positive improvement on that workload.
A bound spanning zero would leave the result inconclusive.
Comparisons against static and random selection would require their own strictly positive lower bounds to support superiority.
The replacement rule changes both the training step and the particular completion selected. The resulting comparison would measure the combined follow-up policy.
a · One eligible member’s owned priority slots
Unchanged: event ranking, the 32 stage-balanced audit identities and the 128-slot budget. On the exposed Aria ledger: at most 31 replacements across 43 eligible members.
b · Decision rule, follow-up policy minus the original event-informed policy
9.3Required independent evidence
The proposed study requires an unexposed stepped training run containing:
- Compatible evaluator records.
- Original protected task checks.
- Verified canonical completion identities.
- A reviewed development-versus-assessment exposure map.
A new random seed drawn from the already examined Aria run would reuse exposed development evidence.
At the latest source-admission checkpoint, a qualifying independent dataset had not been established.
The follow-up evaluation therefore remains pending.
10.Limitations
Single development corpus
All findings come from one Aria training run used repeatedly during detector and investigation development. The twenty schedules reuse the same finite pool and overlapping priority prefixes.
Endpoint dependence
The static selector concentrated historical pass claims, a prerequisite of the measured discrepancy. Retrieval performance may differ under another investigation endpoint.
Missing outcomes
Seventy-four first-seed discrepancy outcomes remained unresolved. Shared-identity bounds preserve their possible contribution to the policy differences.
Training-stage confounding
The policies differ simultaneously in step allocation, feature eligibility, prompt composition, and case identity. The observed late-stage concentration describes their combined allocation.
Historical evidence authority
Historical pass claims originate from candidate-controlled workers. A discrepancy records disagreement with protected host-side checks. Additional evidence is required to establish evaluator manipulation or the underlying runtime mechanism.
Retrospective evaluation
The assessment uses frozen queues constructed from an existing training run. Prospective discovery delay and usefulness during live training remain unmeasured.
Independent validation
The follow-up sampling rule has been specified but has not been evaluated on independent data. The present corpus provides development evidence for its design.
These limitations define the scope of the current findings and the evidence needed for broader claims.
11.Conclusion
On the frozen Aria evaluation, event-informed investigation retrieved 13 observed historical-pass/protected-failure discrepancies in 128 slots, compared with 47 for static selection.
The difference persisted under all assignments of unknown outcomes allowed by the shared-identity bounds, with static retrieving between 13 and 64 additional discrepancies per 128 slots.
Case-reach analysis traced the observed misses to exemplar construction. Fifty-six missed discrepancies fell outside eligible onset-to-alert windows, and ten were omitted within eligible windows. All 66 were absent from the complete 878-identity exemplar pool.
The top-ranked event identified def:run_tests near steps 110–112. The behavior subsequently became widespread. During steps 151–200, the feature appeared in 12,681 of 12,800 completions, approximately 99.1%.
The event-informed priority queue allocated few cases to that late stage, where many observed discrepancies occurred.
The result supports treating onset evidence and follow-up verification sampling as separate design decisions.
A proposed one-slot post-alert sampling rule preserves the original event evidence while extending candidate coverage into later training. Its retrieval benefit remains to be tested on independent data.
For the current discrepancy endpoint, the frozen static policy remains the reference comparator, and production event ranking remains unchanged.
The next research question is how to allocate a limited verification budget after detecting a behavioral shift, while preserving onset evidence and measuring useful-case recovery independently.
Reproducibility and evidence
This analysis draws on the frozen Aria M1.5 development evaluation and the subsequent event-retrieval diagnosis.
Implementation and selection checkpoints
- Frozen selection checkpoint:
57f463f - Production pipeline checkpoint used by the detailed diagnosis:
d13aa1b - Later engineering changes have not undergone a new independent discrepancy-retrieval assessment.
The local evidence archive retains exact source references, assessment records, queue identities, reproducibility commands, and reviewer findings.
Aria data and protected execution records remain stored locally under their existing licensing, provenance, and execution constraints. This note reports aggregate findings and retains the underlying evidence references without redistributing private evaluation artifacts.