TVC
The timing and telemetry foundation for a thrust-vector-control simulation. I measured the wakeup latency of the 500 Hz C++20 loop I built and traced the long tail to a timer running on another CPU.
The measured system is a Linux timing harness with a small plant stand-in, independent mitigation switches, and a telemetry ring drained outside the control thread. The harness measures the platform’s behavior before a full vehicle workload is introduced.
The repository now contains a PID controller and episode state machine with dedicated test targets, along with bounded nonblocking UDP transport foundations and typed control recording support. These components have functional evidence but are not yet the integrated simulator/controller runtime. The published timing numbers remain from the August timing-harness campaign, before these additions.
Repository snapshot 851d61c: controller · episode state machine · build targets. Measured code 0fafb7c: timing harness.
Measuring accumulated lateness
At 500 Hz, each cycle has a 2 ms period. Sleeping for 2 ms after every iteration accumulates overshoot, and in the naive L0 baseline, p99.9 lateness reached 14.9 seconds relative to the intended schedule. Measuring only against the previous wakeup would hide that drift.
The harness instead gives every cycle a deadline from one origin: origin + n × period. A late cycle still produces a sample against its original deadline, even when the next cycle is already due.
- Wakeup jitter
- Actual wakeup minus intended deadline. This measures when the loop resumes.
- Body execution
- Body completion minus wakeup. This measures the stand-in workload and the optional allocating log.
- Deadline miss
- Body completion after the next scheduled deadline. Counted separately from ordinary late wakeups.
Because the harness timestamps body completion before telemetry enqueue and statistics collection, the recorded body execution time covers only part of a cycle’s total cost.
Measurement definitions · Timestamp placement · L0 source summary.
Draining telemetry outside the control thread
The control thread writes a fixed-size record into a single-producer, single-consumer ring. A separate thread running at ordinary priority drains the ring into framed, CRC-32C checked recordings. If the ring fills, the producer counts a drop and keeps going.
- Control threadAbsolute wakeup
Run body · enqueue record - SPSC ringBounded handoff
Full ring → count drop - Drain threadOther CPU
Frame · CRC · file I/O
new and delete on the marked control path; direct C allocation is outside its scope.Measured implementation: thread setup · record and ring · drain and recording.
Investigating late wakeups
Early campaigns improved the headline result, but occasional wakeups were still hundreds of microseconds late. Disabling idle states across all 16 logical CPUs reduced the tail, but made the machine poll continuously. CPU isolation alone had not explained the behavior.
The investigation correlated late cycles with kernel noise on the isolated CPU, but those windows did not contain enough local work to explain the delay. Checking timer placement showed that the loop’s wakeup timer could be on a housekeeping CPU, where an idle-state exit delayed the wakeup.
Tracer findings and timer-placement evidence · Captured session observations.
Thread affinity and timer placement
Although thread affinity controls where the loop runs, it does not by itself fix where its wakeup timer runs. Under this machine’s isolation configuration, allowing timer migration can introduce a dependency on another CPU.
Migration enabled 1
- CPU 7Pinned loop arms its sleep timer
- Housekeeping CPUTimer may migrate here
- CPU 7Loop wakes after timer delivery
An idle-state exit on another CPU can delay the loop’s wakeup. The timer can also remain on CPU 7.
Migration disabled 0
- CPU 7Pinned loop arms its sleep timer
- CPU 7Timer stays on the isolated CPU
- CPU 7Local timer wakes the loop
The timer and loop share CPU 7. Its idle states are disabled; the housekeeping CPUs may still sleep.
| Timer migration | p99.9 jitter | Maximum |
|---|---|---|
| Enabled | 380.415 µs | 893.951 µs |
| Disabled | 16.591 µs | 27.151 µs |
This scheduling schematic explains the relationships between components without representing a trace or simulation. Both short runs used L5 with idle states disabled on CPUs 6 and 7, and only the timer-migration setting changed. These diagnostics led to the longer campaign below.
Interpretation and single-variable comparison · Diagnostic run manifest.
The qualification campaign
The follow-up campaign applied each mitigation cumulatively, with three 10-minute runs per level. The machine was an HP ProBook 465 G11 with a Ryzen 7 7735U, running Ubuntu’s 7.0.0-30-generic kernel. Timer migration was disabled; idle states were disabled only on logical CPUs 6 and 7.
- L5 p99.9 wakeup jitter
- 16.5 µs
- Maximum observed wakeup jitter
- 86 µs
- Pinned cycles measured
- 2.7 million
2026-08-29 campaign. The L5 value is the median of three run percentiles; the maximum and cycle count cover L4–L6. Published results.
Wakeup jitter at p99.9
Three individual repeats Median of run percentiles
The fixed 1–1,000 µs scale is logarithmic, so equal distances represent tenfold changes. Lines span the observed repeats without representing confidence intervals.
| Adds to the previous level | Individual repeats, µs | Median, µs |
|---|---|---|
| L1Absolute deadlines | Median331.3 µs | |
| L2Lock and prefault memory | Median407.6 µs | |
| L3FIFO priority 80 | Median400.6 µs | |
| L4Pin thread to CPU 7 | Median16.4 µs | |
| L5Remove allocating log | Median16.5 µs | |
| L6Enable telemetry | Median17.3 µs |
At L4, pinning the loop to the disciplined CPU brings all three repeat values close together. L2 and L3 do not produce a reliable improvement by themselves. The figure compares run percentiles and includes neither a chronological latency trace nor a pooled percentile.
L0 is a separate drift failure. Its median p99.9 lateness is 14.9 seconds and falls outside this axis. It remains visible here because absolute deadlines fix a correctness problem before they improve a latency number.
Across the nine pinned runs at L4–L6, there were no recorded deadline misses. L5’s median p99.9 was 16.543 µs; enabling telemetry at L6 moved it to 17.263 µs, about 4.4% higher. All three telemetry runs reported zero ring drops.
What the measurements establish
The campaign shows repeatable timing on one qualified machine under a named configuration. Compared with the earlier all-CPU polling setup, this configuration has a higher median but a much tighter observed worst case, while keeping only the isolated pair awake.
The observed 86 µs maximum is not a worst-case execution-time bound. A stock Linux kernel, firmware behavior, and a finite campaign do not establish a hard real-time guarantee. The new controller and episode code need their own integration and measurement; the pinned-timer comparison with PREEMPT_RT also remains open.
Tradeoffs and open measurement question · Firmware qualification record.
Evidence and maintenance
Functional checks and timing qualification have different jobs. Hosted CI can check correctness; timing regression runs belong on the measurement machine. The environment record includes power settings, idle-state disable counts, and timer migration so a lapse in setup can be distinguished from a code regression.
- How the harness measures Deadlines, allocation guard, and measurement scope.
- Qualified platform record Topology, isolation, firmware, and power discipline.
- Campaign history and findings Null results, setup mistakes, timer investigation, and remaining questions.
- Timing regression gate Checks L5 run percentiles against the committed baseline.
The transport and typed recording foundations are recorded at dd32b14. Their functional checks do not extend the timing campaign’s qualification.
Historical timing sources frozen at 851d61c (2026-09-06). Campaign code revision: 0fafb7c. No new timing campaign was run for this case study.