Preregistration — Stage 1 of the Bullettime deadline-miss validation programme
- Written: 26 August 2026, before any result existed. The analysis below was fixed first and the harness was then run once against it.
- Registers:
research/lab/(harness) andresearch/results/stage1.json(output). - Supersedes nothing. This is the first entry in the programme.
1. The claim under test
PRD §12 records the product's central claim as unestablished:
No study compares late-packet rate against latency, jitter or conventional loss as a predictor of user-reported problems using the same traces.
The full claim is about user-reported problems and cannot be settled without human subjects. Stage 1 tests the weaker claim that must hold first:
H1. Deadline-miss rate — packets arriving after the consuming application's deadline, plus packets that never arrive — predicts simulated application-level damage better than mean one-way delay, RFC 3550 interarrival jitter, RFC 5481 packet delay variation, or conventional packet loss rate, when every metric is computed from the same packet trace.
What Stage 1 can establish: that the metric has, or does not have, discriminative power over the alternatives against a mechanistic model of what the application actually does with the packets.
What Stage 1 cannot establish: anything about human perception. A win here is a necessary condition for the product's claim and not a sufficient one. A loss here refutes the product's claim cheaply, before a subjective panel is paid for. That asymmetry is the entire reason this stage exists and is run first.
2. The circularity hazard, and the design that answers it
The obvious oracle for a deadline-based metric is a jitter buffer, which is itself deadline-based. Scoring deadline-miss rate against jitter-buffer discards measures the same quantity twice and returns a perfect result that means nothing.
Primary oracle is therefore non-circular by construction. It is a client-side game simulation whose damage measure contains no deadline, no threshold and no notion of lateness:
- A server ticks at 60 Hz and emits authoritative entity positions.
- A client renders at 120 Hz, interpolating between the two snapshots bracketing
now − interpolation_delay, and extrapolating from the last known velocity when starved. - Damage = time-weighted RMS positional error, in world units (metres), between what the client drew and where the entity authoritatively was, plus the count of shots whose aim disagreement exceeded a hitbox radius.
Positional error is an application outcome measured in metres. It is not a restatement of the candidate metric. The link between them is real but not tautological, because extrapolation absorbs a late packet with zero error when motion is linear, and compounds superlinearly when it is not; because a packet arriving late but before the next render may still be consumed; and because a packet whose successor already arrived is discarded whatever its own delay was.
A second oracle (voice) is reported separately and is explicitly flagged as partially circular, because an adaptive jitter buffer shares mechanism with the candidate. It is retained because its damage measure adds two components the candidate cannot see — added conversational delay from a buffer that grew, and the non-linear perceptual cost of consecutive concealment — but no conclusion rests on it alone.
3. The design's discriminating region — the point most easily got wrong
If sessions are sampled only from conditions where loss and lateness co-vary, no analysis can separate the predictors, and a study that samples that way will report that everything works equally well regardless of the truth.
The parameter space is therefore sampled so that loss severity and delay-variation severity are drawn independently, deliberately populating all four quadrants:
| low delay variation | high delay variation | |
|---|---|---|
| low loss | clean | the region that separates the metrics |
| high loss | the region that separates the metrics | both |
The two off-diagonal cells are where the hypothesis lives or dies. This is a stated design choice, not a claim that those conditions are equally common in the field.
4. Predictors, fixed in advance
All computed from the same trace, using only what a real measurement client can observe (no oracle state, no synchronised clocks — delay is handled as RFC 5481 PDV relative to the sample minimum).
| # | Predictor | Note |
|---|---|---|
| P1 | loss_pct | conventional packet loss rate |
| P2 | owd_mean | mean one-way delay |
| P3 | owd_p95 | 95th percentile one-way delay |
| P4 | jitter_rfc3550 | RFC 3550 interarrival jitter, 1/16 gain |
| P5 | pdv_p99 | RFC 5481 PDV, 99th percentile |
| P6 | pdv_p999_range | RFC 5481's own summary, D(99.9) − D(min) |
| C1 | deadline_miss_app | the candidate. Late + never-arrived at the application's own deadline |
| C2 | deadline_miss_100 | control: the same metric at a fixed 100 ms for every application |
C2 is a required control. C1 is allowed to know the application's deadline and P1–P6 are not. That asymmetry is the product's claim ("the deadline is the part that varies by application"), so C2 exists to measure how much of any C1 advantage comes from the deadline concept itself rather than from being handed the application's number.
5. Analysis, fixed in advance
- Primary: Spearman rank correlation ρ between each predictor and continuous oracle damage. Rank-based, so no functional form is assumed for any predictor.
- Primary test: bootstrap over sessions, 10,000 resamples, of the paired difference
|ρ(C1)| − |ρ(P_i)|for each alternative. 95% percentile CI. - Secondary: AUC for the binary outcome damage above the design median, with **DeLong's test for two correlated ROC curves** (same sessions, paired).
- Multiplicity: Holm–Bonferroni across the six alternatives, per oracle.
- Effect direction:
|ρ|is used because a predictor is useful whichever sign it carries.
Falsification criterion, stated before the result
H1 is NOT supported at Stage 1 if, for the primary game oracle, the bootstrap 95% CI for |ρ(C1)| − |ρ(P_best)| against the strongest alternative includes zero or lies below it.
If that happens, the honest response is to say so on the marketing page and in the PRD, and not to run a subjective panel until the mechanism is better understood.
Additional pre-registered reporting requirements
- Base rate of the binary outcome is reported; a rate outside 5–95% invalidates the secondary AUC.
- If C1 beats P1–P6 but not C2, the finding is reported as *"a deadline metric wins, and knowing the application's specific deadline adds nothing measurable"* — which contradicts a material part of the product's positioning and must be published as such.
- The number of sessions, the seed, and the full parameter grid are recorded in the output.
- Any deviation from this document is listed in the results under
deviations, with its reason.
6. Known limits of Stage 1, recorded in advance
- Simulated impairment, not measured impairment. Stage 2 replays the same profiles through Linux
tc-netemin a container and checks the trace statistics agree; until then, internal validity rests on a model. - One simulated game and one simulated codec pipeline. Neither is a real title or a real product.
- No human subjects. No claim about perception may be drawn from this stage in any form.
- The oracle is a model of an application and inherits whatever that model gets wrong.
Harness, seeds and raw JSON live beside these documents in the repository. Every figure here comes from simulated impairment; no human subjects were involved and no perception claim is made. Back to Bullettime.