The cost of fewer false alerts

What I tested

This investigation concerns the retrospective detector, which examines a completed series and selects at most one mean shift. The sequential alert rule is outside this comparison.

I used three fitted continuous profiles: reward from Aria, and the number of declared files and return statements from the AISI run with a KL penalty of 0.02. I call the last two files and return below. These were profiles implicated in earlier false-alert investigations.

The experiments generate Gaussian series using the saved variance and temporal-dependence parameters. A no-change series can still wander because neighboring observations are correlated. Its generating mean stays constant, so any selected changepoint counts as a false alert.

Each series has 80 observations. The planted versions add a positive shift at boundary 40, with sizes of 0.25, 0.5, or 2 rollout standard deviations. That scale comes from the fitted rollout distribution. Each unchanged trace and its three planted counterparts share the same noise and measurement uncertainty.

I measured two things: how often a no-change series was flagged, and how often a planted change was selected in the correct direction with its boundary within five observations of 40. The second measure, localization, counts success over all generated planted series. Selections outside that five-step window or with the wrong sign count as misses.

The reference false-alert ceiling was 0.5%. The ceiling applies to individual synthetic series; across the thousands of features a run screens, false alerts accumulate. The existing benchmark also requires at least 95% localization on its original planted changes. The smaller effects here were additional sensitivity measurements, with no new passing threshold.

Why removing the false alerts was difficult

The existing detector first chooses the split that best reduces squared error. It then compares constant-mean and shifted-mean models, each fitted with an AR(1) covariance model. That covariance describes the size and temporal correlation of the fluctuations around the mean.

Earlier experiments pointed to an interaction between selecting the boundary and refitting covariance. In a separate 256-realization Aria comparison, covariance-refit gain supplied the amount needed to cross the confirmation cutoff in 25 of 27 false alerts. It also supplied that amount in 185 of 226 correctly localized +0.5 SD changes.

That was an accounting of the existing fits. It gave me a reason to be cautious about suppressing the refit contribution: the same contribution appeared in many detections I wanted to keep. A modified detector would need its own evaluation.

Measurement uncertainty complicated the picture further. On 64 existing AISI null trials, replacing each step’s estimated variance with the trial’s average removed three alerts and introduced one. Replacing that average with the generating variance caused no further decision changes. Stepwise variance estimates influenced particular decisions, and pooling still left false alerts.

I wanted to test a procedure that retained covariance adjustment while accounting for uncertainty in the fitted explanation.

The comparison setup

Production was the unchanged package detector at commit 58a3f6b.

Calibrated control kept its boundary selection, screening gates, covariance fits, and confirmation score. An external wrapper changed only the final confirmation cutoff. This let me measure how much could be achieved by recalibrating the existing score.

Integrated evidence used a new research prototype. It modeled uncertainty in the reported standard errors and averaged evidence over covariance parameters and shift sizes across eligible boundaries under fixed priors. The constant-mean and shifted-mean models could favor different covariance values. It also used its own boundary estimate and omitted the production L2 screen. The comparison therefore evaluates the combined procedure as a unit, since several components changed together.

I generated 20,000 fresh no-change series per profile and scored both calibrated methods on them. Within each profile I took the recorded cutoff at ascending rank 19,940, then used the largest of the three as that method’s global cutoff. With strict exceedance, this allowed at most 0.3% calibration alerts per profile, leaving headroom below the 0.5% evaluation ceiling.

The global rule meant a detector could receive a series without knowing which fitted profile generated it.

After freezing both cutoffs, I evaluated 20,000 different base realizations per profile. Each supplied the four paired effect conditions, producing 60,000 independent evaluation base paths (240,000 traces and 720,000 outcomes across the methods). Every evaluation cell was scorable, and all models and cutoffs stayed fixed throughout evaluation.

Fewer false alerts came with many missed changes

The table puts the null cost beside localization of the +0.5 SD changes. Each rate has a denominator of 20,000.

Null alerts and +0.5 SD localization under the global cutoffs

Both calibrated methods stayed below 0.5% in all three profiles. Integrated evidence localized more +0.5 SD changes than the control under this policy. On files, the gain was 12.45 percentage points, with a paired 95% interval of 11.93–12.96 points. Its observed null rate was slightly lower as well.

That is useful evidence for the candidate. Its sensitivity was still far below production’s, and production’s higher sensitivity came with null rates above the ceiling in every profile. Because the achieved false-alert rates differed between the calibrated methods, the comparison reflects performance under a shared calibration rank.

At +2 SD, every cell reached 100% localization except the return control, at 99.895%. At +0.25 SD, integrated localization was only 0.770% for Aria, 1.365% for files, and 0.190% for return.

The large-change results alone would have concealed most of the sensitivity problem.

For the integrated candidate at +0.5 SD, selection and localization rates were almost identical. On files, it selected 22.725% and localized 22.690%. Most misses happened because the rule abstained from selecting a change.

How the shared cutoff affected sensitivity

The files profile determined the global cutoff for both calibrated methods.

For integrated evidence, its recorded calibration cutoff on files was 10.374, compared with 6.577 on Aria and 6.097 on return. All three received 10.374 in the main experiment. The control had the same ordering on its own score scale.

This explained why Aria and return were operating so far below the false-alert ceiling. I wanted to quantify the sensitivity cost of that decision.

In a post-study diagnostic, I applied each profile’s already-recorded calibration cutoff to the saved evaluation scores. Observations, scores, eligibility, directions, and estimated boundaries stayed fixed, and thresholds came strictly from the earlier calibration pass.

Here is the resulting +0.5 SD localization, with the corresponding null rates:

Post-study diagnostic: each profile’s own recorded cutoff

For Aria, integrated localization rose from 27.910% to 53.105%, recovering 25.195 percentage points through the cutoff change alone. Return recovered 13.825 points. Files stayed identical because its profile cutoff already equaled the global one.

Return also reversed the methods’ ordering: the control now localized more changes than integrated evidence, while also recording a higher null rate. The result depends on both the scoring procedure and the calibration policy.

This diagnostic used profile identities that are unavailable during normal operation, where a feature name in a new training run gives no guarantee that its noise follows one of these saved laws. The result measures a cost in this batch and leaves an application-time calibration problem to solve.

Even with per-profile cutoffs, integrated localization at +0.25 SD reached only 5.645% on Aria, 1.365% on files, and 1.615% on return. The shared cutoff explained only part of the loss.

What I take from this

Under the frozen global policy, integrated evidence localized more +0.5 SD changes than the calibrated control in all three profiles. Their observed false-alert rates differed, leaving performance at matched rates unresolved. The known-profile diagnostic showed that a substantial share of the loss on Aria and return came from imposing the files-derived cutoff. It also showed that the candidate’s advantage varied with the calibration policy.

The remaining gap is substantial, especially for smaller shifts. These results describe the procedures and operating points I tested, leaving the theoretical sensitivity limit open.

The study covers three fitted Gaussian profiles, all continuous, with a single positive change at the midpoint. The candidate’s standard-error model assumes independent, equal-weight Gaussian sampling-unit means with a common variance. Extending that model to the clustered uncertainty estimates used on training data needs separate justification. Neither calibrated method was evaluated on the complete original benchmark. False-alert rates on real training runs and performance on other change shapes remain untested.

I kept the production rule unchanged. The research prototype remains separate, and the package’s existing retrospective acceptance failure remains visible.

The question this leaves me with is how to calibrate across different behavioral series using information available from the data. The score determines how evidence is ordered, while the calibration policy determines how much of it is enough to raise an alert. In this study, both choices had a substantial effect on which changes survived.

Study details

The main comparison used calibration seeds 20270501–20270520 and evaluation seeds 20270601–20270620, with 1,000 base realizations per seed and profile. These integers identify random seeds. All priors and numerical settings, along with the cutoffs, were frozen before evaluation. All 583 scheduled numerical checks passed.

Main-study paired intervals used 4,000 bootstrap draws of whole base paths, preserving the methods and effect conditions together. Intervals are pointwise. The later known-profile diagnostic reused evaluation outcomes and reports exact intervals for nested decision changes, conditional on the recorded cutoffs. It is an exploratory follow-up to the completed study.

The full research implementation and generated study artifacts remain in my private archive. A public reproduction bundle has not been released.