Mamadou Wane

Project · RL training analysis · In development

Rollout Diff

Deterministic analysis of behavioral shifts across RL training: rollout ingestion, static feature extraction, prompt-balanced aggregation, change detection, and supporting evidence.

Python · pandas · PyArrow · Apache-2.0 · github.com/mamadou-wane/rollout-diff

Post-training dashboards track scalar metrics over time. Rollout Diff reads the completion dumps an RL trainer already writes to disk and identifies which specific behaviors changed during training. It grew out of What the reward curve hides, an analysis of four public reward-hacking runs where label-free alerts surfaced each run’s underlying exploit mechanism (selected from the alert lists after the run). Reward hacking is its first application within a broader scope.

What it does today

  1. Loads rollout dumps. load_trl reads TRL completion logs, load_verl reads verl’s per-step JSONL files, and from_table maps DataFrame columns through a columns dictionary. Every canonical row requires run, step, sample_id, prompt_id, prompt, response, and a finite reward.
  2. Extracts static features. The extractor strips complete <think> and <thinking> blocks, then parses Python fragments into structural AST features and prose into lexical n-gram features without executing code. Each tier retains a sparse vocabulary of at most 2,000 tokens, typed as binary or continuous.
  3. Builds prompt-balanced series. Each step averages completions within each unique prompt first and weights prompts equally, computing cluster-robust standard errors across sampling units.
  4. Detects changes two ways. A sequential rule freezes a 15-step baseline and alerts on 3 consecutive steps with |z| > 4 for continuous features, or z > 4 with a prevalence rise of at least 0.10 for binary features. A retrospective test selects the best single L2 mean-shift split and, by default, confirms it against a latent AR(1) noise model.
  5. Attaches supporting evidence. Reports split-half replication, event-window corroboration, persistence, within-group reward association, and prompt-mix shifts at each sequential alert, with a row for every series, including non-alerting and unscorable series.
  6. Groups alerts into ranked events. Rising binary alerts join one event when every pair’s onsets fall within 3 steps and their positive rollouts overlap with a Jaccard index of at least 0.5. Events with a rising binary member are ranked within each run by that member’s evidence: feature tier, split-half replication, persistence, signed reward association, and prompt breadth. The rest follow in onset order. Each rising binary alert carries up to three exemplar rollouts from its onset-to-alert window, stored as references to the original completions.
  7. Reports and inspects. The rolloutdiff CLI writes a versioned JSON report of ranked events, evidence, and exemplar references without raw rollout text, and renders it as a standalone HTML file. inspect resolves an event’s exemplars against the original dump and can pair each one with a feature-absent completion from the same prompt and step.
from rolloutdiff import analyze_rollouts, load_trl

rollouts = load_trl(
    "/path/to/completions", run="experiment-1", reward_column="training_passed"
)
analysis = analyze_rollouts(rollouts)
print(analysis.evidence)
print(analysis.sequential.screening)
print(analysis.event_ranking)
rolloutdiff analyze /path/to/completions --format trl --run experiment-1 \
    --reward-column training_passed --output report.json
rolloutdiff render report.json --output report.html
rolloutdiff inspect /path/to/completions --format trl --run experiment-1 \
    --reward-column training_passed --report report.json --event-rank 1

Current constraints

  • Uncalibrated error rates: Outputs are descriptive and do not establish causal effects or attribute mechanisms. Event ranks set an inspection order, and events group alerts by timing and rollout overlap without verifying a shared cause.
  • Multi-unit requirement: Scoring requires at least two sampling units per step to compute a standard error, leaving single-prompt-per-step schedules (such as the TRL run in the research note) unscorable.
  • Full-run vocabulary and single-split assumptions: Vocabulary selection scans the full dataset, and the retrospective detector fits a single step-function shift, making it sensitive to transient spikes and outliers.
  • Pending retrospective calibration: On the synthetic benchmark, the retrospective detector currently misses its target false-alert threshold, and its benchmark test fails. The cost of fewer false alerts measures what calibrating it costs in sensitivity. How to calibrate it across behavioral series is unresolved.

How it is tested

The public benchmark manifest pins four dataset revisions and 118 Parquet files by byte size and SHA-256 hash. A local synthetic benchmark fits the first 15 steps of the Aria dataset, injects shifts at step 40 of 80, and requires each detector to catch at least 95% of planted shifts within 5 steps while keeping false alerts on scorable stationary series at or below 0.5%. Pinned snapshots of rlprobe, rldoctor, and CHERRL’s RHDA run as external baselines.

The empirical numbers in What the reward curve hides come from the research prototype run prior to building this package. The note’s Reproduction section documents where the package differs.