The step oracle

Abstract

A CPU with no GPU can state the bytes a GPU training step will produce — the loss, every parameter, every optimizer tensor — before the card runs it, and verify them afterwards. Decided on four cards across three architectures, 1,486 of 1,486 stated outputs each; a 300-step GPT-2 small run committed as a 119 KB document and confirmed on an L40, an H100 and an A100 by recordings made or sealed apart from the commit. The H100’s undocumented split-K plan was recovered from the card’s digests alone.

The instrument is the step oracle: a PyTorch training step — forward, backward and optimizer update, every kernel the dispatcher launched — re-executed on a CPU from a specification of the kernels’ arithmetic, to the same bytes. It replays PyTorch 2.13’s default fp32 steps on Turing and Ada, the pinned fp32 step on Turing, Ampere, Ada and Hopper, and bf16 autocast at 124 M, where the runs of an RTX A6000, an L40 and an H100 in both of PyTorch’s reduction settings are documents stated 300 steps ahead and decided by sealed recordings. Under each card’s family table a T4, an L4, an RTX A6000, an L40 and an H100 compute one 124 M trajectory byte for byte, and one document verifies it on x86 and aarch64. With one checkpoint in a hundred kept and the rest deleted, all 595 tensors of every deleted state return identical, across instruction sets. Of seventeen edits to run documents across two programs, five are refused before any float is computed, nine are caught at the step whose bytes they change, and three verify because they change no byte. The price is 40–50 s per 124 M step on 16 CPU threads, 270–480× the GPU’s — against a published route that audits at 1.2× but modifies the run and grows its artefact with every step. The claim is exactness by specifying the vendor’s arithmetic rather than logging it, over an unmodified stock-precision run, with a constant-size artefact, verifiable on a CPU.

1. The claim

A PyTorch training step — forward, backward and optimizer update, every kernel the dispatcher launched — can be stated on a CPU that has no GPU, from a specification of the kernels’ arithmetic, and the GPU then produces the same bytes: the loss, every parameter, every optimizer tensor, and every intermediate of every op. We call the program that does this the step oracle. Run backwards over a recording it names the kernel and the cause of every mismatch; run forwards it states the step of a card that has not run it. It exists today for PyTorch 2.13.0’s default fp32 steps as the dispatcher runs them — stock cuBLASLt GEMMs including the ones split over K, the dispatcher’s own attention with its in-kernel dropout, the random kernels, the fused and foreach optimizers — on Turing and Ada under both the deterministic and the default rung; for the pinned fp32 step on Turing, Ampere, Ada and Hopper, where the four cards produce one set of bytes; and for the bf16 autocast step on one Ada card, six steps at 124 M parameters. The bf16 124 M runs themselves are documents on three cards — the RTX A6000 in both of PyTorch’s reduction settings, the L40 in both, the H100 whose two settings are one run — each stated 300 steps ahead and decided by a recording sealed before its recipe was committed (§4c). Every recorded step on every card we own replays; one op — the fused embedding backward above 3,072 indices under the default rung, a race of up to 62 atomic partial sums — is carried from the recording rather than computed, and the deterministic flag removes it.

Four cards were stated before they ran. Given a card’s three numbers (SM count, threads per SM, compute capability), its arithmetic class and its GEMM plan table, the oracle wrote each card’s default bf16 step to a file before the card had run it, and the recording then agreed on every one of 1,486 stated outputs, the loss, 53 of 53 parameters and 159 of 159 optimizer tensors: an L40S and an L40 — two Ada products that reject each other’s file at the predicted op with the predicted counts — an A100 and an H100 PCIe, whose plans came from heuristic queries and witnesses and never from a recording of the step (§4, §16.38). Four cards, three architectures, ≈ 1.7 rented GPU-hours.

Two 300-step runs were committed before their cards ran. At GPT-2 small — 124 M parameters, 3,430 ops a step, 50,304-wide GEMMs the vendor splits over K in ways no log states — the oracle walks each card’s stock run from the seed and writes it as a 119 KB document of hashes and literals promising the state after eleven steps. The L40’s was committed at 09:27 UTC and a second L40 instance, started at 09:34, matched the promise after step 299 and all 300 loss bits (§16.47.1); the H100’s was committed at 10:28 UTC and a recording sealed at 06:01 UTC, its hash recorded at fetch and unreadable to the walk, matched when opened (§16.48.2) — its six undocumented split-K plans recovered from the card’s digests alone. The A100’s, committed at 13:11 UTC, was decided by a recording made the next day: promise 299 equal and all 300 loss bits (§16.47.3). Under each card’s family table the same 124 M program is byte-identical on a T4, an L4, an RTX A6000, an L40 and an H100 — Turing, Ada, Ampere and Hopper — and with one checkpoint in a hundred kept, every deleted state of the pinned run came back identical — 595 of 595 tensors, across instruction sets (§4).

The pinned 3.2 M-parameter trajectory — 300 steps, eleven checkpoints, 425.8 MB of parameters and optimizer state — is a 34,888-byte recipe of hashes and literals, and verifying that document reproduces all eleven promises on a machine holding no tensor at all (§16.35); its initialisation is computed from the seed on a machine that cannot draw it (§16.34).

Two failures of the method’s own discipline are recorded with these results and bound what they may be read to claim: a session summary script walked a sealed recording and leaked its 300 loss values before the prediction was committed, so the pinned 300-step prediction may be judged only on the state promise (§16.50.1); and the walk’s step-0 comparison reported 176 in-place optimizer outputs as different where card-against-card they were identical — an instrument fault, registered open and then found (§16.49.2–3, §7). No claim was judged on step 0 at any point.

Seventeen deliberate edits across the 3.2 M and 124 M programs — five refused before any float is computed, nine caught at the step whose bytes they change, three that verify because they change no byte (§16.36, §16.49). The claim is falsifiable byte by byte and was tested that way: each kernel family was registered with a prediction, run, and either matched every element of every output or was reported as a mismatch with its cause (§4). Every GPU session of this programme was one rented instance — mostly L4s and T4s at $0.63–0.97 an hour, one A100 hour and one H100 hour — backed up in full and deleted after use; the CPU side ran on the project’s 8-core aarch64 controller, two fleet x86 hosts and the fleet’s Mac.

2. Method: record, re-execute, compare — and predict

Record. One training step is recorded on the GPU with every op the dispatcher launched — its arguments, outputs and storages — together with the card’s SM count and threads per SM, the device generator’s state before and after, and the parameters and optimizer state before and after. A 10 M-parameter transformer step is 1,333 ops and 848 MB; GPT-2 small (124 M) is 3,430 ops, kept at 3.5 GB by recording large tensors as digests only.

Re-execute. The oracle computes each op on the CPU from the recorded inputs, with the arithmetic of the kernel PyTorch dispatched: GEMMs as their arithmetic class (§3.4); reductions, normalisation and softmax as the trees their open-source kernels describe (§3.2); transcendentals through the hardware’s approximation instructions (§3.1, §3.3); random kernels through the generator’s stream (§3.6); attention from the kernels’ sources (§3.7). Where the source leaves a choice open — which multiply-add the compiler fused — every choice is computed and the recording says which one the GPU took; a rule is “fitted” only in that sense.

Compare, then replay. In op-local mode each output is compared with the GPU’s bytes (bit equality; a mismatch names one kernel). In replay mode each op is fed the CPU’s own previous outputs, so a single wrong bit anywhere propagates to the end; “no divergence” means the loss, the updated parameters and the optimizer state equal the GPU’s after the whole step.

Predict. The same program run for a card that never executed the step: its SM count and threads per SM, its arithmetic class, and its GEMM plans as cuBLASLt’s own heuristic reports them on that card; the contraction choices taken from a card that did run it, since they belong to the kernels; every output digested when produced and the whole predicted step written in the recorder’s format. Nothing is compared until a recording exists. The instrument is tested by predicting a card from its own parameters — the L4 so predicted equals its recording on 1,486 of 1,486 predicted outputs — and decided across cards on the L4/T4 pairs: each card’s step from the other’s program, four of four (§4). A step’s inputs after step 0 are card-dependent (the parameters carry the card’s dropout masks), so a prediction takes the target’s own inputs, and a recording made to test one starts from the predicted step’s pre-state.

3. The kernel families

3.1 The hardware’s approximation instructions

libdevice’s expf, exp2f, rsqrtf, tanhf, erff and the fast logarithm end in MUFU instructions — ex2.approx, rcp.approx, rsqrt.approx, sqrt.approx, lg2.approx — whose results the PTX ISA bounds but does not specify. All 2³² inputs of ten such instructions were swept on an A100, an L40S, an L4 and a T4 (§13 (k), §16.2, §16.8, §16.11): the tables are identical on every card, and each instruction reduces to a compact table and a rule that reconstruct every output exactly. Two of the rules are findings: ex2.approx needs a second rule for negative arguments (the natural one fails on a large share of them, the right one on none), and lg2.approx — found in §16.17 through the flash-attention logsumexp — composes exponent and mantissa by a rule that truncates toward zero (the nearest-even composition misses half of the e ≠ 0 inputs by one ulp). sin, cos and tanh.approx have no compact rule among those tried; PyTorch’s step does not use them.

3.2 Reductions, normalisation, softmax

ATen’s generic reduction is a function of the row length alone: one warp per row, four sequential accumulators combined as ((a₀ + a₁) + a₂) + a₃, a shuffle tree with decreasing offsets 16…1, and a warp split when values per thread exceed 256. A CPU reproduces every row of every dump — 2048 × 768, 2048 × 1024, 64 × 4096, the warp-split 16 × 50304, and the mean as the sum times fp32(1/N) (§16.4). Full-tensor sums are the exception: their block count follows the SM count (ctas per output = min(3 · SMs, values/16)), so the oracle routes them through GEMM (the bias gradients) and specifies the rest per SM count. Welford variance and the vectorised layer-norm kernel (128 threads, online sums with a reciprocal multiply, rsqrt.approx from its table) match on every row once two compiler contractions were read off the rows (§16.5, §16.10). Softmax changes kernel with the row length — a warp kernel to 2048 columns, a register kernel to ten blocks, an ILP kernel beyond — and cross-entropy’s nll_loss tree uses clamp(2^round(log₂(rows/16)), 32, 1024) threads; all three kernels and the loss match bit for bit on both cards’ dumps at the first comparison (§16.9).

3.3 The math library

libdevice’s expf, logf, tanhf, erff, rsqrtf and sqrtf are re-executed from the bitcode, with exact fused multiply-adds and the tables of §3.1. Validated on the L4 against seven elementwise dumps — exp, log, exp − rowmax, rsqrt, tanh, sigmoid, gelu, 6.5 M elements — bit-identical (§16.8). One correction became a finding: gelu matched only with erff’s tail coefficients from the CUDA 12.9.86 toolkit’s libdevice, not Triton 3.7.1’s bundled copy; the two differ in 17 of 571 functions, so eager PyTorch and torch.compile carry different math libraries and the oracle is keyed by the libdevice version.

3.4 GEMMs

fp32. Chapter 1 found that every no-split-K cuBLASLt configuration of algorithms 20 and 0 gives one digest across Ampere, Ada and Hopper, and §14.1 named what it computes: the textbook dot product, acc ← fma(a, b, acc) over k in order in binary32. The oracle computes exactly that (with the midpoint tie rule a naive double emulation gets wrong on 6 of 22 shapes) and reproduces the A100’s witness bytes for 7 of 7 such shapes while 34 controls differ. The family learner tests a card’s configurations against that CPU digest and keeps the fastest member per shape: 14–15 members of 61–62 configurations for every one of the 22 shapes on the L4 and A100, 22–23 on the H100, 6–7 on the T4, at 0.91–1.11× the stock speed; the stock heuristic is outside the class for 11–15 of 22 shapes on every card (§16.10.3, §16.11.1). Under the family table the GEMM layer is the class. The stock kernels outside it are the class too, chunked (§16.26.1): cuBLASLt’s fp32 split-K kernels compute the chain per contiguous K-chunk with a kernel-specific alignment and sum the partials in fp32 in split order, the bias first on addmm; one Turing kernel keeps four sliced accumulators per chunk and combines them pairwise. Read off the bytes for 10 L4 shapes, 9 batch-32 shapes and 9 T4 shapes (263 GEMM ops over eight traces), the plans make the DEFAULT fp32 step — stock heuristics, no pinning — replay end to end on both cards (§4).

bf16. The tensor core is not IEEE-sequential. Its arithmetic is MMA-Sim’s model (Xie et al., arXiv:2511.10909; report R7): exact products, groups of L = 8 aligned to the largest exponent with the accumulator, truncation to F = 24 fractional bits, one round-toward-zero to fp32 — the same on Turing, Ampere and Ada; L = 16, F = 25 on Hopper and Blackwell. Chained over k the way the kernel walks it (sequential, or sliced-K in two partitions), it reproduced the A100’s bf16 GEMM witnesses (§18.1) and 34 of the 51 stock bf16 GEMMs of the recorded bf16 step. The other 17 are split over K; §16.21.1 recovered their five plans from the bytes — each chunk a fresh chain on the same adder, reduced as the kernel’s documented scheme reduces — and with them all 51 match (§5.1).

Three cards’ stock fp32 plan tables for one program (the 124 M nanoGPT, batch 2 × 1024, vocab 50,304; §16.42–16.48) were read from each card’s own cuBLASLt log and fixed by its bytes or digests where the log alone was wrong. Ada splits little and splits contiguously; Ampere sends two shapes to a legacy kernel whose K-order is an interleaved sliced chain; Hopper’s nvjet splits six shapes and deals K round-robin in 8-wide tiles. Per-shape tables for the three cards are available to reviewers.

3.5 Elementwise and optimizer kernels

The addcmul/addcdiv and lerp kernels, the eight _foreach_* kernels of torch.optim.AdamW, the fused AdamW kernel’s fp32 opmath with a correctly rounded powf, the α-add, division by a CPU scalar as a multiply by the reciprocal and gelu_backward’s erf form: each was read from source, offered with its contraction candidates, and matched (§16.10.1, §16.12.1, §16.13.1).

3.6 The random kernels

Dropout is Dropout.cu’s fused kernel: a grid of min(SMs · threads-per-SM / 256, ⌈n/256⌉) blocks, each thread a Philox4x32-10 subsequence, four (eight for bf16) consecutive elements per curand_uniform4, the uniform conversion as one fused operation, and a generator advance of ⌈n / (256 · grid · 4)⌉ · 4. The mask is therefore a function of (seed, offset, n, SMs, threads per SM) — the card is in the bytes — and the oracle computes every mask of ten traces on the L4 and the T4 exactly, predicts the generator’s offset after the step to the number, and replays the stock step on both cards end to end although the two steps differ in every byte (§16.13.1). nn.Embedding’s backward (the feature kernel below 3,072 indices, the sorted kernel above) matched from source on both cards.

3.7 Attention

Three kernels, three arithmetics, all from source. Turing, fp32 (_scaled_dot_product_efficient_attention, cutlass FMHA on OpClassSimt): forward and backward computed from the kernel’s source — all eight calls bit for bit, and the T4’s default step replays end to end (§16.14.1). Ampere/Ada, fp32: the same kernel on tensor cores through cutlass’s 3×TF32 split over §3.4’s adder in its tf32 form; bit for bit at 128 and 256 keys, and the L4’s step replays end to end (§16.15.1). bf16, FlashAttention-2 (_scaled_dot_product_flash_attention): from source, with one compiler contraction located by the fact that only two-block rows missed; P is dropped by a seven-round Philox keyed by tile position with keep iff byte ≤ ⌊float(1 − p) · 255⌋ — a keep rate of 230/256 for p = 0.1, rescaled by 1/0.9; forward, logsumexp and dq/dk/dv bit for bit on all eight calls (§16.17.1).

The attention’s dropout, and the default rung’s order (§16.27.1). The memory-efficient kernel draws its own dropout — one Philox state per (batch, head) at the generator’s offset, keep iff u > p, the kept values scaled once by float(1/(1 − p)) — and the backward rebuilds the mask from the forward’s saved seed and offset. Read from source, it matched all 8 attention ops of seven stock fp32 + dropout traces on the L4 and the T4 — and exposed the default rung’s arithmetic: without the deterministic flag the backward runs two key-split CTAs that continue each query block’s dQ accumulator through a lock in arrival order, and under the causal mask the later key block’s CTA arrives first. With that order the two G0 recordings match bit for bit. At 256 keys the default kernel runs four key-split CTAs (fp32 has no cap on the split count), a genuine race for each tile’s lock — and the realized order is still readable from the bytes: every one of the 1,024 (batch-head, query block) tiles of four recordings on the L4 and the T4 is one of its ≤ 24 admissible orders (§16.29.1), overwhelmingly key blocks descending; the two runs of a card differ on 6 of 768 inferable tiles on the T4 and on none on the L4. The same recordings hold the other kind of race: at 4,096 indices the token-embedding backward takes a fused atomic kernel whose partial sums land in an order with up to 62! possibilities per element — readable only for rows with ≤ 7 partials (11,520 of 11,520 elements), and reported as unmodelled otherwise.

4. Results

(a) Predicted, then recorded — and the runs committed before their cards ran.

trace (card)	step	ops	op-local: match / mismatch / unmodelled	end-to-end replay
A100 — predicted, then recorded	the same default bf16 step, predicted from the L4’s inputs, the A100’s own heuristic plans, 108 SMs and the Ampere class (§16.22); recorded on an A100 PCIe (§16.38.3)	799	689 computed; 1,486 of 1,495 outputs stated	DECIDED: 1,486 of 1,486, the loss, 53/53, 159/159, the offset — the first decision across an architecture (Ada inputs, an Ampere card), from a heuristic query and witnesses, with no A100 recording of the step in existence beforehand
H100 PCIe — predicted, then recorded	the same step, predicted on the Hopper adder with the H100’s own heuristic (nvjet sequential; algorithm 24’s 8-wide groups and split-K, digest-exact on the H100’s witnesses), flash under Hopper, 114 SMs (§16.28); recorded on an H100 PCIe (§16.38.4)	799	689 computed; 1,486 outputs stated	DECIDED: 1,486 of 1,486, the loss, 53/53, 159/159, the offset — the hard card: a different adder, kernels the source card never runs, flash attention under Hopper untested on any recording; none of the four ranked risks fired
L40S — predicted, then recorded	the same step, predicted from the L40S’s own logged choices (four split shapes), the Ada class, 142 SMs (§16.31.2); recorded on an L40S six hours later (§16.38.1)	799	689 computed; 1,486 outputs stated	DECIDED: 1,486 of 1,486 predicted outputs equal, the loss, 53/53, 159/159, the generator’s offset. The control — the L40’s file against the same recording — fails at op 247 with 477/1,486, exactly the counts the two files’ diff gave before the card ran; the card’s own cuBLASLt log reproduces the committed plan table on 15/15 shapes; two runs byte-identical
L40 — predicted, then recorded	the same step on the other Ada 142-SM card, from the L40’s own logged choices (§16.31.2); recorded on an L40 (§16.38.2)	799	689 computed; 1,486 outputs stated	DECIDED: 1,486 of 1,486, the loss, 53/53, 159/159, the offset. The pair is the sharpest test on file: same architecture, SM count and class, and each card’s recording rejects the other card’s file at op 247 with 477/1,486 — the plan table, not the card block, is what separates them
T4 ← L4’s program, L4 ← T4’s program (§16.23)	pinned-dropout (1,365 ops) and attn-pinned-128 (1,197 ops), each card predicted from the other card’s op list with only its own inputs and (SMs, threads/SM, compute capability)	4 runs	1,361/1,361 and 1,189/1,189 predicted outputs equal to the target’s recording, each direction	loss, 53/53, 159/159 and the generator’s offset equal in all four; the attention arithmetic (SIMT / 3×TF32) selected by compute capability alone
T4 ← L4’s program, L4 ← T4’s program, STOCK (§16.30)	stock-g1-a (733 ops) and stock-fused (619 ops): stock GEMMs under the target’s plan table, the dispatcher’s attention with dropout, native_dropout, foreach / fused AdamW	4 runs + 1 control	1,425/1,425 and 999/999 each direction; the control with the source card’s plan table diverges at the first differently-split GEMM	loss, 53/53, 159/159 and the generator’s offset equal in all four
the fleet’s 300-step run (§16.32)	the pinned fp32 program walked forward on the CPU: the recorded step as the program, the state carried, the batch from the same generator, the two AdamW host scalars recomputed per step	300 steps	—	10 of 10 checkpoint digests equal to the fleet’s (steps 0, 1, 2, 3, 5, 10, 20, 50, 100, 200, 300) from the seeded init; 9 of 9 from the step-1 checkpoint; 1,505 s on two CPU cores
the fleet’s 300-step run from the SEED (§16.34)	the same walk with the initial weights computed rather than captured: normal_’s AVX2 Cephes kernels and uniform_’s contraction, from the seed	300 steps	53/53 initial tensors bit-identical	10 of 10 digests with no captured tensor anywhere; 1,450 s
the L4’s 300-step STOCK run (§16.33)	the same walk with cuBLASLt’s own kernels and split-K plans (the L4’s table, read off its bytes), math attention, 58 SMs	300 steps	—	3 of 3 digests and losses equal to the L4’s September trajectory (step 300 rcpt2:fd8c24b3e517…); 1,689 s
the T4’s 300-step STOCK run, from the L4’s program (§16.33)	the L4’s op list, the T4’s plan table and (40 SMs, 1,024 threads/SM) — the T4 never ran this program	300 steps	—	3 of 3 digests and losses equal to the T4’s September trajectory (step 300 rcpt2:872dbfbf2bd7…). The three runs above are pairwise distinct at every step: the cards’ split choices move the trajectory, and the pin moves it again
the L40’s and the A100’s STOCK 124 M runs from the SEED (§16.43.1, §16.44.3)	GPT-2 small under default cuBLAS on each card, six steps: the seed, the op list, the card’s own kernel log (the L40: one split-K chunk fitted from bytes; the A100: two legacy algo 1 kernels fitted — an interleaved sliced chain), its SM count, and the stock candidate reference (the L40’s, serving both)	3,430 × 6	step 0 replayed end to end: 3,722 / 3,722 on each card, 148/148, 444/444	steps 1–5 each 3,246 / 3,246 op digests, 148/148 parameters, 296/296 optimizer tensors, every loss bit against each card’s own streamed recordings; two sessions per card agree with each other (3,214/3,214)
the L40’s, H100’s and A100’s 300-step STOCK runs, stated before the card (§16.47.1, §16.48.2, §16.47.2)	124 M, from the seed, each card’s fitted table: promises after 1–5 (recorded) and 9, 19, 49, 99, 199, 299 (predicted), 300 loss bits; 119 KB documents committed before any card ran them	300 steps each, 3 h 38 min–3 h 55 min on 6–16 CPU threads, 10.7–13.3 GB	control 3,246/3,246 at each of steps 1–5 on all three	DECIDED on the L40 by a second instance run 7 min after the commit: promise 299 EQUAL, 300/300 loss bits; DECIDED on the H100 by a recording sealed 4½ h before the commit (hash a281076c…): promise 299 EQUAL, 300/300 loss bits; DECIDED on the A100 by a recording made the day after the commit: promise 299 EQUAL, 300/300 loss bits (§16.47.3)
the H100 by digests alone (§16.48.1)	six digest-only recordings (steps 0–5, 1 MB each); at each split-K GEMM the walk digests 22–30 candidate plans and takes the one the card’s step-0 digest names	3,430	all six nvjet algo-55 shapes = P3 t=8 F1 (round-robin 8-wide K-tiles, partials summed in split order); two H100 sessions agree 3,214/3,214	under the fitted table: 3,246/3,246 at each of steps 1–5, 148/148, 296/296, every loss bit — the third card, decided without a byte of its tensors
five cards, four architectures, one trajectory (§16.50.1, §16.50.3, §16.50.4)	the PINNED 124 M program under each card’s own family table (24/24 shapes in the identity class on each; median price 0.95× L40, 0.95× H100, 0.99× A6000, 0.99× T4): T4 (Turing, 40 SMs), L4 (Ada, 58 SMs), RTX A6000 (Ampere GA102, 84 SMs), L40 (Ada, 142), H100 (Hopper, 114)	3,874 / 3,430	card vs card: 3,848/3,848 at step 0, 3,404/3,404 at each of steps 1–5, 148/148, 296/296, every loss bit, on all five; the L4’s 2026-09-12 recording agrees on all 3,404 at step 1	the CPU walk with no plan table equals every stream at steps 1–5 (3,246/3,246, 148/148, 296/296) with the L40’s card numbers and, unchanged, with the H100’s; the T4’s sealed 300-step recording equals the committed no-plan-table recipe’s promise 299 with 300/300 loss bits; the same cards’ stock runs share 225/3,404 at step 5 and no parameter

(b) Op-local re-execution and end-to-end replay, per recorded step.

trace (card)	step	ops	op-local: match / mismatch / unmodelled	end-to-end replay
l4-5 (L4)	stock fp32, math attention	1,333	1,242 / 34 / 0 before the plans; 1,276 / 0 / 0 with the fp32 plan table (§16.26)	no divergence with the plans: loss, 53/53, 106/159 + 53 step counters op-locally (pre-fix recorder); the default step needs no pinning
l4-7 (L4)	fp32 under the family table	1,333	1,276 / 0 / 0	no divergence: loss, 53/53 parameters, 159/159 optimizer tensors
a100-8, h100-11, t4-5	the same program, own family tables	1,333 each	1,276 / 0 / 0 each	no divergence on each; all 1,333 op outputs identical across the four cards
l4-8 (L4)	GPT-2 small, streamed	3,430	3,278 / 0 / 0 (152 skipped)	no divergence: loss, 148/148, 444/444; 200 s on 8 CPU cores
l4-9, t4-6	stock fp32 + dropout + foreach AdamW, pinned GEMMs	1,365 each	0 / 0 unmodelled on each	no divergence on each; the two steps differ in every byte and both replay
l4-9 ×5, t4-6 ×2	stock fp32 + dropout, stock GEMMs under the cards’ fp32 plan tables (§16.26), the attention’s dropout and the default rung’s key-split order (§16.27)	733 / 619 each	0 mismatch, 0 unmodelled on all seven (G1, G0, batch 32, fused AdamW; Turing and Ada)	no divergence on all seven
t4-7, l4-10	default fp32 with the dispatcher’s attention	1,197 each	0 mismatch, 0 unmodelled	no divergence on each — no kernel outside the recipe on Turing or Ada
l4-10 bf16	default bf16 autocast, flash attention; 17 GEMMs split over K and reduced in bf16 under the permissive default	799	672 / 17 / 0 before the plans; 689 / 0 / 0 with them (110 skipped)	no divergence: loss, 53/53 parameters, 159/159 optimizer tensors — the default bf16 step, with the five split-K plans of §5.1 recovered from its bytes

(c) The run as a document: recipes, the audit, the burial, bf16.

trace (card)	step	ops	op-local: match / mismatch / unmodelled	end-to-end replay
the 124 M runs as recipes (§16.45.1, §16.46.1)	each run a document: one 79 KB program blob shared by both cards, the card block, the plan table, 60 contraction indices, five state promises (a digest over 148 + 296 tensor digests and the loss)	111 / 121 KB	—	5 / 5 promises each from the document and the corpus alone (≈ 5 min, init from the seed in 10 s), 7.47 GB of kept state per card; the card-swapped control refused after step 1; a sampled audit of one step from the named checkpoint in 50 s, a one-ulp tamper refused in 3 s
the sampled audit and the red team at 124 M (§16.46.1, §16.49.1)	one step from the checkpoint the recipe names (1.49 GB, hashed in 3 s); seven edits to the L40’s document	1 step	the checkpoint’s own promise checked before any step; the tampered copy refused in 3 s	one step verifies in 50 s; edits: 2 refused before computation, 4 fail the promise after step 1, the SM count 142 → 108 passes (not load-bearing at these shapes; it was at 3.2 M)
the first burial (§16.51.3–5)	the PINNED 124 M run, 300 steps, stated on the x86 VM with no plan table (111.9 KB, 11 promises); checkpoints after 99, 199, 299 written; 99 kept, 199 and 299 retained as buried copies the walk cannot see; resurrection 99 → 199 from the kept state, 199 → 299 from the resurrected state; the same 99 → 199 on an aarch64 CPU from the x86 checkpoint	300 steps; 2 × 100 resurrected on x86, 100 on aarch64	three sealed recordings (L40, H100, A6000): promise 299 EQUAL on all three; resurrections 34.6 s/step (x86, 16 threads), 35.8 (aarch64, 6 threads)	every buried byte came back: 595/595 tensors identical to the buried copy at 199 and at 299, and 595/595 across instruction sets — parameters, both moments, step counters, loss, generator; 1.49 GB + 112 KB kept in place of 16.4 GB promised — a saving at 124 M; at frontier scale the keep-interval economics invert and the recipe’s value is the proof, not the storage (§7.1)
the bf16 autocast 124 M step, from the seed (§16.53.1–2)	the pinned program under bf16 autocast (147 tensor-core GEMMs, autocast casts, bf16 gelu, the 50,304-wide bf16 log-softmax), recorded digest-only at steps 0–5 on an RTX A6000; the CPU walk with the compiled T-FDPA adder, no reference report, no plan table — the card’s digests choose	3,808	2,292/2,292 at step 0 (aligned prefix), 3,624/3,624 at each of steps 1–5; 0 unmatched selections; the only choices: 12 gelu_backward contractions	no divergence at any of six steps: 148/148, 296/296, every loss bit; 36 min per step on 16 x86 threads (both plans per GEMM); the run written as a 115.6 KB recipe with bf16_reduction: exact
the bf16 run stated 300 steps ahead (§16.55.1)	the exact-setting bf16 program walked 300 steps from the seed under the single-plan table with the twelve digest-chosen contractions carried; a 122 KB recipe of eleven promises (1–5 recorded, 9/19/49/99/199/299 predicted) committed before the sealed A6000 recording was opened; the seal’s hash verified at fetch and at opening	300 steps, 77 s each on 16 x86 threads (6 h 26 min)	steps 1–5: 3,624/3,624 each against the six streams	promise 299 EQUAL, 300/300 loss bits equal — the first bf16 run stated ahead; checkpoints after 99, 199, 299 kept as a bf16 burial corpus
the reduced-precision bf16 run as a recipe (§16.56.1)	the same program under PyTorch’s default reduction, its GEMM plans read from the card’s digests by candidate search (single-pass plans and the split-K vocabulary of §5.1, each computed only when the selection asks)	6 steps	13 of 147 GEMMs split over K, all reduced in bf16 (INPLACE): the twelve K = 3072 input gradients of the MLP’s first layer in chunks of 1024, the K = 50,304 lm_head input gradient in chunks of 6336; 134 single-pass; 0 unmatched	no divergence at any of six steps: 2,292/2,292 prefix, 3,624/3,624 ×5, 148/148, 296/296, every loss bit; the recipe carries bf16_reduction: reduced and the fitted table
the reduced-precision bf16 run stated 300 steps ahead; the exact recipe on a second instance (§16.56.4)	the reduced-setting recipe (commit 7c43e10, 00:49 UTC) with its thirteen bf16-reduced split plans read from digests; a second A6000 instance recorded the reduced 300 after the commit (01:22–01:30 UTC), and the exact 300 again	300 steps	—	promise 299 EQUAL, 300/300 loss bits on the reduced recording; the exact recipe of §16.55.1 holds on the second instance too — one document, two instances
the exact-reduction flag removes the reduction, not the split: the second Ada card is its own recipe (§16.58.1–2)	the L40 (142 SMs, cc 8.9) under the same exact-setting bf16 program blob as the A6000 (84 SMs, cc 8.6): same adder class	step 0	1,375 of 4,226 comparable outputs equal; first difference at op 73, the forward 2048×3072→768 addmm (K = 3072), which the L40 splits over K with an fp32 reduction and the A6000 runs in one pass	a stock bf16 recipe carries the adder class and the card’s plan table; the L40’s op-73 kernel matched none of 936 digest-searched plans — cuBLASLt algorithm 30, a kernel outside the model until the next row
the H100’s exact and reduced bf16 runs are one run, stated 300 steps ahead (§16.58.3–5)	the H100’s exact-setting step read from its digests — the Hopper adder (F = 25, 16-wide groups), 146 sequential GEMMs and one fp32-reduced split (the lm_head input gradient, chunk 10,112 × 5); the two settings’ streams identical, PyTorch’s flag a no-op on Hopper; one recipe (commit 83d7e70) against both sealed 300-step recordings (e530f6b8…, 1cf17dd1…)	300 steps, 73.3 s each on 16 x86 threads	2,292/2,292 at step 0; 3,624/3,624 at each of steps 1–5	promise 299 EQUAL and 300/300 loss bits against both seals — over 300 steps the flag left the H100’s bytes untouched
algorithm 30, a kernel outside the model, fitted from bytes (§16.59.1–4)	the L40’s step 0 recorded in bytes (tensors to 32 MB) under both settings with cuBLASLt’s own log, which names algorithm 30 (64×64, single pass) for the twelve K = 3072 projections; from op 73’s bytes its arithmetic is four sliced accumulators over 16-wide K-tiles, pair-reduced (s0+s2)+(s1+s3), the bias added into slice 0 before the reduce, fp32 epilogue — exact on 1,572,864 of 1,572,864 outputs; at 128×64 two slices over 32-wide tiles; its split-K chunks round to 64	4,226	step 0 replays whole in both settings (4,100 / 0 / 0; 148/148; 444/444; the loss)	the first fit put the bias after the reduce and the six-step walk parted at step 1 on op 145, the layer-1 instance of the same shape; the card’s own log for steps 0–1 showed algorithm 30 for all 24 calls — the walk was wrong, not the card; with the bias in slice 0, op 145 is exact under 1 of 139 associations
the L40’s bf16 run stated 300 steps ahead, both settings (§16.59.5–6)	the log-derived tables (exact: algorithm 30 sliced, one fp32 two-way split; reduced: plus thirteen bf16-reduced splits, chunk 10,112 for the lm_head gradient); recipes committed on 2026-09-17 at 13:08 UTC (exact, 68e4b19) and 13:37 UTC (reduced, d4afa33); recordings sealed on 2026-09-16 opened after each commit	300 steps: 77.8 s each on 16 x86 threads (exact), 132.1 s each on 6 aarch64 threads (reduced)	steps 0–5: 2,292/2,292 on the prefix, 3,624/3,624 ×5, in both settings	promise 299 EQUAL, 300/300 loss bits in both settings — with the A6000 and the H100, four cards’ bf16 124 M runs are documents, all stated 300 steps ahead
the 4-rank all-reduce, derived (§16.57.1)	DDP’s rebuilt bucket assignment (torch’s own function over the gradient-ready order: a 1 MiB first bucket, then 25 MiB) and NCCL’s ring rule read from all_reduce.h (524,288-element chunks in loops of four, 16-byte-aligned quarter tails, chunk c folded left from the ring position after c), applied to the four ranks’ pre-reduction gradients with no reduced gradient in hand	2 steps × 3 nodes	—	101/101 tensors, 56,902,656 of 56,902,656 elements bit-exact on a 4×L40S, a 4×A100 PCIe and an 8×A100 NVLink node (four ranks), at steps the derivation never saw; the initial-reversed control 12/101 on each — the recipe’s all-reduce term is four literals and one rule
a DDP rank’s step as a program (§16.60.1)	the recorder under torchrun: the reducer’s slot writes (mul.out by 1/W), the collective (c10d.allreduce_ on the flat bucket, waited on) and the reads back, each annotated with its bucket; the oracle folds the W ranks’ assembled buckets by the ring rule, the peers’ buckets from the same program walked on their rows of the batch	a tiny model on a CPU with gloo, W = 2 and 4	identical op lists and annotations across ranks; the two-rank collective exact; mul.out 29/29, copy_ 31/31, mm 21/21	the walk runs with shadow ranks at both world sizes; at W = 4 the NCCL rule differs from gloo’s reduce on 23,078 of 116,608 elements — gloo’s association is not NCCL’s, and the card decides (Part B: the next row)
the four-rank step on the card (§16.60.4–5)	four L40S under NCCL Ring / Simple / one channel, NVLS and CollNet off; the stock fp32 124 M step recorded on every rank in one pass (3,889 ops each, thirteen collectives at the same indices), the peers and the streams lean (pre-state by digest — their digests equal rank 0’s on every tensor); the walk from step 2’s pre-state with the peers’ rows as shadow ranks under rank 0’s digest-chosen candidates, then from the seed through the run’s three programs	a 4×L40S node for 0.8 h (≤ $4.22); the x86 VM, 16 threads, 234–272 s per four-rank step	3,703 of 3,703 op outputs at each of steps 2–5, the thirteen all-reduce outputs among them (ring 0→1→2→3, chunk 524,288, one channel); 148/148 parameters, 296/296 moments, the loss bits at every step; each peer’s own recorder 148/148 slots; from the seed: 4,135/4,135 at step 0, the arithmetic of step 1 whole (171 rebuild-bookkeeping outputs not comparable), 3,703/3,703 at 2–5 to the same promises	the first walk’s shadows had run before rank 0 chose its GEMM candidates (3/148 slots, the first collective different) — a selection pre-pass fixed it; one node, one fabric, one NCCL version, world size 4, the pinned arm
the four-rank run as a document (§16.62.1)	run_recipe write with the run’s three programs (bucket annotations kept in the blobs) and a ddp block of four literals — world, ring, chunk, channels — plus the card’s plan table and six recorded promises; verify reads the file, the corpus and a cached initialisation and nothing else	the x86 VM, 16 threads	VERIFIED 6/6 promises from a 366 KB document (157–178 s per four-rank step, 17 min; 8.96 GB of state replaced); of five edits, a second channel and a tampered blob are refused before any float, ring and chunk fail at the first promised step, world 2 is refused because the program’s shapes fix the per-rank batch	the loss bits of step 0 survive a ring edit (the loss precedes the collective) — the promise is a state digest for this reason; the refusal of a tampered blob is an uncaught assertion, not the tool’s REFUSED line

Rows are settled in PREREG §16.10.1, §16.10.3, §16.11.1, §16.12.1, §16.13.1, §16.14.1, §16.15.1, §16.17.1, §16.21.1, §16.43.1, §16.44.3, §16.45.1, §16.46.1, §16.47.1–2, §16.48.1–2, §16.49.1, §16.50.1/.3, §16.51.3–5, §16.53.1–2, §16.55.1, §16.56.1/.4, §16.57.1, §16.58.1–5, §16.59.1–6, §16.60.1, §16.60.4–5 and §16.62.1; result files under python/recipe_checkpoints/results/. The “skipped” ops are scalar reads and profiler markers.

5. What only an exact oracle can see

1. A PyTorch default makes a third of the bf16 step’s GEMMs inexact (§16.16.1). torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction defaults to True; cuBLASLt then reduces split-K partial sums in bf16. The 17 affected GEMMs of the recorded step — the weight gradients with K = 2048 and one input gradient — differ from the exact bf16 product in 18–36 % of their elements, with 99th-percentile relative errors of 0.6–4.7 %. Run-to-run comparison cannot see it (two runs agree byte for byte); training curves cannot; the oracle does. The precision fix is the flag. Reproducibility does not need it: §16.21.1 recovered the five kernels’ plans from the bytes alone — a search over K-partition rules and cuBLASLt’s documented reduction semantics, judged by the class adder. INPLACE kernels add the parts up in bf16 in split order; OUTPUT_TYPE kernels round each partial to bf16 in the workspace and sum them in fp32. Every recovered (numSplitsK, scheme) equals the L40S heuristic table’s row at smCountTarget = 58, and the same chunk rule is digest-exact on the A100’s witnessed COMPUTE_TYPE entries. With the plans in the oracle the default bf16 step replays end to end (§4’s last row). So the default is inexact and predictable at once — two properties, two fixes: the flag for the values, the plan pin for the bytes. Upstream knows the mechanism as a trade-off, not its magnitude: pytorch#153095 (2025-05) made cuBLASLt respect the flag at all, noting that split-K with reduced-precision reductions “were unlikely to happen” under the old 1 MiB default workspace and that the default was being raised; pytorch#164766 (2025-10) added a separate allow_splitk control. Our step ran with CUBLAS_WORKSPACE_CONFIG=:4096:8 — the setting cuBLAS documents for run-to-run determinism — which is a 4 MiB workspace: the deterministic rung is also the setting under which the reduced-precision split-K kernels become eligible.

2. PyTorch’s Arm CPU wheel rounds bf16 partial sums to bf16 between K blocks (§16.19.1). Arm Compute Library v52.6.0, pinned by PyTorch 2.13, splits K into ⌈K/680⌉ blocks and merges them through the bf16 destination; 26.3 % of the recorded step’s K ≥ 768 GEMM outputs on this controller differ from the exact single-pass result. ACL v52.8.0 (2026-01-23) disables the path.

3. The card is in the bytes through the random generator and the full-tensor sums (§16.13.1, §16.4): dropout masks and the generator’s state depend on the SM count by one std::min; ATen’s grid-strided sums by their CTA split.

4. The hardware logarithm rounds by truncation (§16.17.1), closing §16.2’s open item: lg2.approx composes exponent and mantissa table by truncation toward zero, exact on 32,768 rows.

5. FlashAttention-2 keeps 230/256 of the entries at p = 0.1, not 0.9 (§16.17.1): a uint8 threshold of ⌊0.9 · 255⌋ = 229, rescaled by 1/0.9 — every dropout-trained FA2 output carries a factor 0.9983.

6. nvcc’s contractions are part of the arithmetic: four of them — in layer norm’s backward, the flash rescale, Welford’s update and dropout’s uniform conversion — decide bytes (§16.5, §16.10, §16.13, §16.17). None is documented; each was located by rows that missed.

7. libdevice is versioned (§16.8): Triton’s bundled copy and the toolkit’s differ in 17 of 571 functions; gelu picks the difference up.

8. oneDNN’s K-chunking is a selector on x86 CPUs (§16.20.1): PyTorch’s bf16 GEMM there merges K-chunks in fp32, but which chunks depends on the shape and, on AMX, on the thread count — torch.get_num_threads() is part of a CPU recipe as the SM count is of a GPU’s.

9. cuBLASLt’s split-K plans are readable from the bytes, and the default step needs no pinning (§16.21.1, §16.26.1). A candidate search against the card’s arithmetic class recovers each split kernel’s plan in seconds — 17 bf16 and 263 fp32 GEMM ops on the L4 and the T4, every one exact, the two cards’ heuristics splitting every shape differently. “Pin, or lose exactness” (§3.4) is now “pin, or record the plan”. The candidate space per split shape is 22–30 plans and the deciding digest is 256 bits, so a spurious match has probability ≤ 30 · 2⁻²⁵⁶ per shape; two plans that compute identical bytes on a step are a tie, not a match, and §5.12 says how a run breaks ties a step cannot.

10. Two kinds of non-determinism, not one (§16.27.1, §16.29.1). The default rung’s attention backward races its key-split CTAs for a lock, yet its outcome is one of at most 24 orders per tile and is read back from the recording every time; the default rung’s fused embedding backward races up to 62 atomic partial sums per element and cannot be. A certificate can carry the first; only the deterministic flag removes the second.

11. The run composes over steps and over cards, unpinned (§16.33.1). The plan table is a property of (card, shape), not of a step: 300 steps of reuse on two cards, with the vendor’s own kernels, land on digests recorded before the tables existed. And a card’s reductions follow from (SM count, threads per SM) for a whole run, not just a step — so a verifier that owns neither card can state either card’s run.

12. A run is a stronger instrument than a step (§16.32.1). A recording pins an arithmetic form only where its data separates the candidates, and a single step’s data often does not. CUDA’s division by a host scalar forms the reciprocal from the scalar as the host holds it — a Python float, i.e. a double — and the oracle’s 1/f32(y) agrees with it for AdamW’s bias correction at t = 1 and t = 2 and first differs at t = 3: every step recorded in this programme was blind to the difference, and the run found it at the third step. A ties report lists what a recording leaves open (on the pinned step: 53 scalar divisions, 9 addcdiv_, 9 layer-norm backwards, three reduction shapes), so the open forms are listed rather than assumed.

13. The vectorised kernel is the reproducible one; the scalar kernel is not (§16.34.1). PyTorch’s normal_ on 16 or more floats uses, on AVX2, Cephes polynomials compiled into the binary with no library call — so it can be specified from source and computed anywhere. The scalar fallback calls the host’s libm, whose float log, sin and cos are not correctly rounded and differ by build, so it cannot. The reproducible path is the SIMD one, which is the opposite of the expectation the GPU work sets up. Getting there also required a rule: the source is not the kernel. The last 1.1 % of elements was decided by which multiply-add pairs the compiler contracts, including one association no reader would guess — one association no reader would guess, deciding 12 of 49,408 elements.

14. The seeded initialisation is not portable across host ISAs (§16.32.1). torch.manual_seed(31), same version, same seed: an x86 host (AVX2 and AVX512 alike) and this aarch64 controller produce different model weights — normal_ takes an AVX2-only path at 16 elements and above, and uniform_ over a non-power-of-two range is FMA-contracted on AVX2 and aarch64 but not on baseline x86. Every GPU host in the fleet was x86, which is why fifteen runs on four GPU products share an init this controller cannot draw from the seed. A recipe that starts from a seed must name the CPU dispatch that drew it, or carry the init bytes — a host term in a GPU recipe.

15. What a run recipe must name, measured by attacking it (§16.36.1). Of the things a recipe could carry, the red team says which are load-bearing for a given program: the corpus, the program, the optimizer’s literals, the host dispatch that drew the initialisation, and the GEMM plan table all change the bytes and are caught. The card’s SM count and threads per SM do not, for this program — its reductions are too small for the CTA split to bind, and its four recording cards (108 to 142 SMs) agree — so that block is provenance here and a parameter only where it binds. And a contraction index that one recorded step cannot pin is pinned by the run: the tie breaks at step 3.

16. Metal computes the recipe’s GEMMs and elementwise kernels, and does not contract (§16.24.1, §16.25.1). Apple’s GPU through MPS is the fp32 chain in fp32 and in bf16 — the first GPU in the identity class without forcing — and a pinned step recorded on it matches this oracle on 1,089 of 1,192 ops; its optimizer fusions match only unfused, and its six reduction families are its own.

17. Where the price of exact verification goes (§16.39.1). Profiled per op at both sizes, the oracle’s time is emulated arithmetic, not interpretation. At 124 M, 51.5 % of the 333 s step is the elementwise transcendental chain — softmax, log-softmax, gelu and their backwards — in numpy at 200–300 ns per element: a few fmas and one table gather per element, each fma an exact float64 emulation with a midpoint correction that costs a dozen array passes. The compiled GEMM chain is 26 % (19.8 GFLOP/s on 4 threads, 38.8 on 16), the optimizer 3 %, views and metadata 0.0 %. At 3.2 M one handler is 48 % of the step by itself: layer norm’s backward enumerates 48 candidate contractions although the walk uses the one the recorded step pinned. Only the chain scales with threads (4 → 16: the step 1.16× faster). Hardware fmaf is the operation the emulation reproduces, so this family’s price is compiled away, not inherent — done in §16.40–16.41: the C ports agree with the numpy originals on all 2³² inputs (five functions, two hosts), and the 124 M step costs 47 s at 4 threads and 39 s at 16 (from 335 and 290), every digest unchanged.

18. A stock 124 M run is bit-exact from the seed on one card, and the two things the log did not say are in one recorded step (§16.43.1). The L40’s default-cuBLAS nanoGPT step 0, recorded in bytes, mismatched the log-derived plan table on one op of 3,430 — the K = 50,304 lm_head backward, whose split-K chunk rounds to 32 where the same kernel rounds another shape to 4 — and the walk from the seed then mismatched on one candidate — the bias-gradient sum, a ones-GEMM in the pinned program and ATen’s reduction tree in the stock one. Both fitted from that step, the L40’s stock run reproduces on a CPU for six steps with 3,246 of 3,246 op outputs, 148/148 parameters, 296/296 optimizer tensors and every loss bit equal at each step; the A100 and H100 agree in the loss and differ in the bits until their own bytes are read. The A100’s bytes (§16.44.1) name the residual: two weight-gradient shapes run on the legacy algo 1 64x32 kernel, an interleaved sliced chain rather than the chain — fitted for one shape (0 wrong of 2.36 M); the other was first reported outside the plan space and was the fitter’s error (finding 20).

19. Two 124 M runs as documents (§16.45.1). The L40’s and the A100’s stock six-step runs written as recipes — the seed, the corpus by hash, one 79 KB program blob shared byte for byte by both cards (slot-renamed, lineage and all), each card’s block and plan table, 60 contraction indices, and five state promises, each one digest over the per-tensor digests of 148 parameters, 296 moments and the loss — are 111 and 121 KB, and each verifies 5 of 5 promises from the document and the corpus alone in five minutes on 16 EPYC threads, the initialisation computed from the seed in 10 s: 7.47 GB of kept state per card, a ratio above 60,000×. A control with the other card’s block is refused after step 1, carrying that card’s loss bits.

20. A promise is an address: the sampled audit (§16.46.1). The checkpoint after step 3 of the L40’s run, written by a prover’s partial walk on an aarch64 host, has as its own digest the recipe’s promise 3; an auditor checks that, resumes, walks step 4 alone and reproduces promise 4 in 50 s with 1.5 GB read — the run never replayed. A one-ulp change to one weight element is refused before any step in 3 s, or, with the refusal waived, fails promise 4 after the step (125 of 148 parameter digests differ). Promises agree across aarch64, x86 and the two cards; the 124 M verifier runs in 12.5 GB.

21. The third card, decided by its own digests; Hopper’s split-K deals K round-robin (§16.48.1). Twenty minutes of an H100 recorded the stock 124 M program’s steps 0–5 as digests only. Offered candidate lists per split-K shape (22–30 plans: the derived chunk, then interleaved and balanced partitions under two reduce orders), the walk let the card’s step-0 digest choose, and it chose the same plan for every nvjet algo-55 split-K shape: P3 t=8 F1, K dealt round-robin in 8-wide tiles across the splits, fp32 partials summed in split order — no contiguous chunk of any rounding fits. Under that table the walk from the seed matches all five H100 streams, 3,246 of 3,246 op digests, 148/148, 296/296, every loss bit; two H100 sessions agree with each other 3,214 of 3,214; the recipe is 111 KB and verifies 5 of 5. Three cards, three trajectories differing in the low bit of the loss, each reproduced exactly. A digest is enough to fit a kernel’s plan when the candidate space contains it.

22. The red team at 124 M: no hole, and a card-block field that does not reach the bytes (§16.49.1). Seven edits to the L40’s 124 M recipe: a corpus byte and a deleted op are refused before any computation by their hashes; a contraction index, the fitted split-K chunk, the initialisation’s ISA and an altered promise each fail the first promise; the SM count moved from 142 to 108 verifies clean because at this program’s shapes no SM-dependent kernel changes a bit (at 3.2 M the same edit failed at step 1). The verifier recomputes and compares; a field whose value changes nothing changes nothing.

23. The exact-reduction flag removes the bf16 reduction, not the split (§16.58.1). With allow_bf16_reduced_precision_reduction off, the RTX A6000 (84 SMs) runs every bf16 GEMM of the 124 M program in one pass and the L40 (142 SMs, the same tensor-core adder) splits the K = 3072 forward projection over K with an fp32 reduction — their exact-setting bytes part at that op. The SM-count key of the fp32 heuristic is alive in bf16, so a stock bf16 recipe is a plan table plus the adder class; the class alone holds only under a table that pins the plans.

24. Under PyTorch’s default, the A6000 reduces thirteen GEMMs of the 124 M step in bf16 (§16.56.1): the twelve K = 3072 input gradients of the MLP’s first layer (three chunks of 1024) and the K = 50,304 input gradient of the lm_head (eight chunks of 6336), all INPLACE, all read from the card’s digests by candidate search over the vocabulary §5.1 fitted at 3.2 M; the other 134 GEMMs are single-pass. The reduced run differs from the exact run in those thirteen GEMMs and nowhere else, and both are recipes.

25. The four-rank all-reduce is derived, not fitted (§16.57.1). DDP rebuilds its buckets after the first backward in the order the gradients became ready, closing a bucket past 25 MiB after a 1 MiB first bucket — one bucket per transformer layer for the program of §13.5 — and NCCL’s ring reduces each bucket in 524,288-element chunks, the remainder in four 16-byte-aligned pieces, each chunk as the left fold of the ranks from the ring position after the chunk’s owner. Built from torch’s own bucket assignment and all_reduce.h, the map reproduces the reduced gradient bit for bit on a 4×L40S node and, at steps it never saw, on a 4×A100 PCIe node and an 8×A100 NVLink node (four ranks): 101 of 101 tensors, 56,902,656 elements, each time. The recipe’s all-reduce term is four literals — world size, ring order, chunk, channel count — and one rule. Recorded and settled on a card (PREREG §16.60.4–5): four L40S under the pinned arm, rank 0’s step walked on a CPU with the peers’ rows as shadow ranks, 3,703 of 3,703 op outputs at each of steps 2–5 — the thirteen all-reduce outputs computed by the rule — and, from the seed, the run’s three programs to the same promises; written as one 366 KB document, the run verifies 6/6 from the seed and every edit to its literals is refused or fails at the first promise (§16.62.1).

6. The classes: what the same recipe computes elsewhere

MMA-Sim’s published cards for every NVIDIA and AMD matrix unit, Intel’s pseudocode for AMX and AVX512-BF16, Arm’s non-EBF16 BFDOT, and the fp32 chain were run over the L4’s recorded bf16 step (R12; §16.18–16.20). On 34 in-class GEMMs (33.2 M bf16 outputs) the units disagree in 49–87 % of their fp32 accumulators and in 4·10⁻⁵ (Hopper) to 1.7·10⁻⁴ (all nearest-rounding units) of bf16 outputs, 94 % of those by one bf16 ulp — bf16 identity across units does not exist; near-identity is the norm, and the rate explains why cross-card trajectories agree for a while and then split (Chapter 1: A100 ≠ H100 at step 100, not step 1). Three vendors’ units are now pinned bit for bit — NVIDIA’s by the validated model, Arm’s (BFDOT/BFMMLA) and x86’s (AMX and VDPBF16PS, byte-identical on Intel and AMD) by probes on our own machines. The class card — products, fused width, fraction bits, summand and output rounding, zero exponent, plus the kernel’s plan — is what a recipe’s GEMM field must record; a vendor name is neither enough (Ampere ≠ Hopper) nor necessary (a CPU verifies any class it can model). Byte-exact predictions for an H100 (forward out differing from the L4’s in 17, 24, 12 and 9 elements on the four flash calls; the Hopper-adder oracle matching it), an A100, a B200 and an MI300X are registered in §16.18 and await hardware.

A twelfth unit was measured on the fleet’s Mac (§16.24.1): Apple’s M4 Pro GPU through torch MPS computes the sequential FMA chain in fp32 — 19 of 19 GEMM witness shapes digest-equal to the oracle’s chain, the first GPU in the fp32 identity class with no forcing — and in bf16 the same chain with one rounding at the end (19 of 19 witnesses; all 37.4 M elements of the L4 step’s 51 bf16 GEMMs re-run from the recorded inputs), run-to-run deterministic. Its GEMV kernel is a different plan (strided lanes, simdgroup trees; the lane count a per-shape selector) and its transcendentals a third table (51–92 % correctly rounded, ≤ 2 ulp, unequal to libdevice on 18–51 % of inputs). Against the L4’s in-class bf16 GEMMs the Mac differs on 1–5·10⁻⁴ of elements, one ulp — the chain-vs-tensor-core gap the fma_chain row of R12 predicted.

The recipe itself was then held against Metal (§16.25.1): the pinned fp32 step recorded on the Mac’s GPU and this oracle run on it unchanged. 1,089 of 1,192 ops match — every GEMM (MPS fuses nn.Linear into aten.linear; its two backward GEMMs and the attention bmms are chains) and every IEEE elementwise kernel — and the optimizer’s addcmul_/addcdiv_ match on an unfused candidate: Metal’s compiler does not contract the multiply-add nvcc fuses. The 46 that differ are the reductions, normalisations and scans (layer norm by up to 8,192 ulp in rstd — a different variance computation; softmax within 8 ulp; the bias column sums; a 2,048-row cumsum), which a Metal chapter would fit from bytes as §3.2 fitted NVIDIA’s. The replay diverges at op 6, the first layer norm.

7. Limits, residuals, open

• The bf16 step’s 17 reduced-precision GEMMs are modelled (§16.21.1); what stays open is the OUTPUT_TYPE reduce’s summation order — ascending, descending and pairwise all reproduce the bytes here — and the 128x64 / 64x3 kernel’s tail-first walk was fitted on 9 elements after looking (then exact on 262,144). The A100’s default bf16 step is predicted, not yet recorded (§16.22): one ≈ $1 recording decides 1,486 output digests at once.
• Flash attention was modelled at 128 keys, one key block; at ≥ 384 keys the non-deterministic backward’s atomics can race (registered; cf. the efficient kernel at 256 keys, §16.14.1). The H100/A100 predictions of §16.18 are untested.
• Two races under the default rung are reported, not modelled: the attention backward’s key-split arrival order at three or more blocks (read back per tile from the recording, §16.29.1 — a certificate can carry it) and the fused embedding backward’s atomic partial sums above 3,072 indices (beyond enumeration; the replay carries the recorded tensor).
• The price, measured (§16.37.1, §16.39.1, §16.40–16.41, §16.51). 10.2 s per step of the 3.2 M-parameter program and 333 s per step of the 124 M one on one x86 core when first measured — 2.7–3.2 µs per parameter-step, roughly linear in the model, and 260–530× the L4’s own step at 3.2 M. Profiled, that price was 51.5 % emulated transcendentals at 200–300 ns per element, 26 % the compiled GEMM chain, 10 % layer norm’s 48-candidate backward and 7 % copies into the storage model. With the chain in C on hardware fmaf, a vector GEMM kernel and the pinned candidate the 124 M step is 47 s at 4 threads and 39 s at 16, and a 300-step walk holds 41.7–46.6 s per step (medians, 16 x86 and 6 aarch64 threads). At 124 M a six-step run recipe verifies in ≈ 5 min on 16 threads, initialisation included (§16.45.1); one sampled step costs 40–50 s. The bf16 step was 36 min until the tensor-core adder was laid out eight output columns per vector (§16.54: bit-identical to the model, 0.046 ns per fused product on 16 x86 threads, 28× the scalar kernel) and is 77 s on the same threads (§16.55.1).
• The verifier’s memory, decomposed — and a correction to this section. This section previously reported 21.8 GB for a 124 M step and extrapolated ~180 GB for a 1 B model, concluding that “the verifier’s memory, not its time, is the first thing to engineer”. Both numbers are now measured and the conclusion is withdrawn. Instrumented (the live store’s byte total after every op), the 124 M step’s live store peaks at 3.87 GB across 317 buffers at op 709, with peak resident 8.25 GB — the 21.8 GB predates §16.40–16.41. The peak is not made of parameters. Its largest terms are the logits (batch × sequence × vocabulary = 412 MB), the tied embedding family (vocabulary × width, four of them: the parameter, its gradient and both moments, 154 MB each) and the attention scores (batch × heads × sequence², seven live at once, 101 MB each) — all of which scale with vocabulary and sequence length, not with parameter count. Only the state (3 × 4 bytes × parameters) is strictly parameter-proportional. Re-extrapolating to 1 B at GPT-2 XL shapes (1600 wide, 25 heads, the same vocabulary and batch): 18.7 GB of state, 0.41 GB of logits, 1.3 GB of embedding family, 1.5 GB of scores and ≈ 4.5 GB of the remaining live set, i.e. ≈ 29 GB, not 180 GB — one commodity server, not a special machine. So the wall at scale is time (≈ 10 min per 1 B step at the measured per-parameter rate), which is what the sampled protocol of docs/04 §3.1 is for, and what a GPU-side verifier under the family table would remove.
• The recorder’s lineage is by address, and the replay trusted it (§16.44.2). In streamed recordings view_of is derived from the last output on a (data_ptr, nbytes) block; a fresh tensor on a reused block inherits a dead tensor’s digest (201 of 3,874 outputs in the A100’s step 0), and the replay’s write-through mutated buffers keyed by digest — shared by every tensor with the same bytes. Fixed at both ends (liveness in the recorder; in-place-only, copy-on-write write-through in the replay; a counter for reads through the fallback) and re-verified on the L40’s full-bytes recording (unchanged, 3,722/3,722). The recordings on file keep their false lineage; the replay no longer acts on it.
• The program blob is the recorder’s document, not the verifier’s (§16.48.1). Three attempts to strip fields the verifier “does not consume” each broke a verified recipe — input storage names (the step counters are distinct only by storage), group-only storages (the sorted-embedding constants), input lineage (a view of a mutated base is read through it; on one card the walk ran to the end with 0 of 5 promises rather than failing). The blob keeps everything the recorder wrote; two cards recorded by the same recorder share one blob byte for byte, and all three do once the interim recorder’s kept inputs and the lineage are normalised. A stripping rule may only follow the walk’s instrumented reads.
• A recipe that starts from a seed must name the host dispatch that drew it (§5.14): the same seed gives different weights on x86 and aarch64 hosts, so the initialisation carries a host term or the init bytes.
• Multi-GPU training is traced for one case — four L40S on one PCIe node under NCCL’s pinned arm (Ring, Simple, one channel), the stock fp32 step, six steps (§5.25, PREREG §16.60.4–5) — and not for other fabrics, NCCL versions, world sizes or NCCL’s own algorithm choice; and torch.compile, cuDNN convolutions and FP8 remain outside.
• Residuals inside the model: the non-ftz ex2.approx’s subnormal results (100,385 of 2³² inputs) are not tabled — immaterial to the kernels traced; lg2.approx‘s truncation is confirmed through lse only; AMD kernels’ K tails and oneDNN’s chunking heuristic are fitted from bytes, not derived from source.
• Two settlement headers of 2026-09-13 were first written with estimated times ahead of the machine clock; they are annotated with the commit times. Times in this section are the clock’s.
• One published table was built from the wrong card’s log and is corrected in place (§16.31.2). The cuBLASLt tables carry a trace column that looks per-card and is not: one captured log is shipped to every host, so it is byte-identical in all ten tables, including 14-SM MIG slices. The L40S’s bf16 plan table was read from it and was the L40’s; four of fifteen shapes’ plans differ between those two cards, and the prediction is re-stated from the L40S’s own log. The A100 and H100 tables rest on by_sm_target queries made on those cards and on their own witnesses, and stand. Provenance is a measurement: a field claiming to be per-card must differ across cards, and a derived table must be regenerable by a command — both are now enforced by the tooling.

7.0 What the instrument got wrong, and the rule each bought

These are faults of the verifier, not findings about GPUs; each is recorded with the rule it left behind.

1. A verifier that can read the answer will; the loss is not the state (§16.42.1). The 124 M recordings name big tensors by digest and give the after-state storage keys no op produces; the walk’s read of a storage nobody wrote fell back to the recorded bytes, so 150 of the 444 tensors “reproduced from the pre-state” had been read from the recording. Undeclared reads are now refused; re-measured, the pinned 124 M run reaches its recorded step from the seed with 148/148 + 296/296 identical. And the three cards’ stock 124 M runs from the seed agree with the CPU in the loss’s bits on 16 of 18 steps while their step-5 tensors differ from the first op: a mean over 10⁸ logits does not see last-bit differences in 10⁸ parameters. Only the bytes decide.
2. Two cards, and the verifier’s own defects were the last residual (§16.44.2). The A100’s “unfittable” kernel was the fitting tool reading a digest-named input through the wrong geometry: a streamed recording names inputs by the digest of their logical bytes, the first output carrying that digest was a transposed view over another tensor’s storage, and the tool applied the input’s contiguous geometry to that storage. Read correctly, the kernel at N = 768 is the same interleaved sliced chain as at N = 3072 (0 of 589,824 wrong). Behind it, two replay defects with one root: the recorder keys lineage by (address, size), so a fresh tensor on an allocator-reused block inherits a dead tensor’s digest as its base — 201 such outputs in the A100’s step 0 — and the replay wrote through that lineage, once zeroing wpe’s gradient (the optimizer’s zeros for its second moment landed on the block of the batch-summed gradient) and once writing the first AdamW moment into the buffer all zero tensors share by digest, from which the second moment was read. Write-through is now confined to in-place ops and copies the base under its new digest, a liveness check removes the false lineage at the source, and reads through the base fallback are counted (0). With that, the A100’s stock step 0 replays 3,722 of 3,722 and its six-step run from the seed matches its own recordings at every step — 3,246 of 3,246 op outputs, 148/148, 296/296, every loss bit — with the candidate choices taken from the L40’s recorded step: the second card, and the choices are the program’s, not the card’s.
3. A literal the verifier never read. The first 124 M recipes carried the architecture as a literal the verifier did not consume; a control that changed it verified clean. The verifier now consumes every literal the document names (§16.45.1).
4. One published table was built from the wrong card’s log (§16.31.2, below): a field claiming to be per-card must differ across cards, and a derived table must be regenerable by a command.
5. A helper deleted with its call sites left, and a helper that called itself (§16.60.1). One commit removed the stock fp32 GEMM’s split-K helper and kept both call sites, so for seventeen hours every fp32 mm/addmm raised, and nothing exercised the path — the bf16 walks in flight never reach it. A blanket edit placed the adder-class line inside the very function it was meant to call, and verify, audit and resurrect recursed to death for thirteen hours while check, which the night’s seals used, never calls it. Neither sat behind a settled claim (the results committed in between name no file made by the broken paths). Both were found by the two-rank CPU recording of §16.60 and fixed; the L40 stock recipe re-verified through step 1 (1 of 1 promises), and that verification now runs before any commit to the oracle. The rule: a helper is deleted with its call sites, a blanket replacement is read at every site it hits — and a walk of the same program with candidate 0 everywhere, which gives the card’s loss bits at both steps and 95 of 3,246 op digests, is item 1 again: the loss is not the state.
6. A collective recorded before it completed (§16.60.1). The first two-rank recording’s reduce output equalled its input, digest for digest: the collective returns a handle and finishes later — at once on gloo, on NCCL’s own stream on a card — and the recorder had read the buffer as the op returned. The recorder now waits on the handle: an op’s output is what it is when the op is done, the rule §16.29 stated for races.
7. The first sliced-K fit put the bias in the wrong place (§16.59.4). One recorded GEMM (op 73) could not tell bias-in-slice-0 from bias-after-reduce — its data tied both forms — and the six-step walk parted at step 1 on the next instance of the same shape. Before blaming the card, its own log for steps 0–1 was read: algorithm 30 for all 24 calls; the walk was wrong. §16.32.1’s rule again: one recorded GEMM may not separate candidates; the step that does is the instrument.
8. Shadow ranks that had not chosen their candidates (§16.60.4). The first four-rank walk had rank 0’s 732 ops before the first collective exact, then the collective different and the peers’ bucket slots 3 of 148. The peers’ shadow executions run before rank 0’s, and a GEMM’s candidate is chosen by the card’s digest during rank 0’s execution — so the shadows had taken candidate 0 everywhere. A selection pre-pass of rank 0’s program (collectives skipped, nothing written back) now precedes the shadows; the rerun was exact at every op. No CPU dry run could have shown it: the tiny CPU recordings are not the oracle’s arithmetic past op 7.

7.1 The nearest published neighbours, and exactly how narrow the gap is (verified 2026-09-15)

Exact cross-hardware replication of training has been published. It is important to say that first, because an earlier draft of this section implied the ground was empty. It is not, and the closest work is strong.

Srivastava, Arora and Boneh, Optimistic Verifiable Training by Controlling Hardware Nondeterminism, NeurIPS 2024 (arXiv:2403.09603, code at github.com/meghabyte/verifiable-training) achieves “exact training replication at FP32 precision” for ResNet-50 (23 M) and GPT-2 (117 M) across three NVIDIA GPUs — A40, Titan XP, RTX 2080 Ti. Their auditor holds the trainer to a Merkle tree over the weights and binary-searches to the disputed step. They control nondeterminism by training in a higher precision than the target, rounding after intermediate computations, and sharing the rounding decisions through an adaptive threshold that selects the values sitting near a boundary. Lyu et al., TrustFL+, IEEE Transactions on Dependable and Secure Computing 2026 (doi:10.1109/TDSC.2026.3677485) carries the same technique from GPU→GPU to GPU→CPU, replaying the recorded rounding directions inside an Intel SGX enclave and reporting zero false positives across several NVIDIA architectures, with “adaptive rounding precisions” to keep non-determinism tractable in a heterogeneous federation.

So the question is not whether exact replication is possible. It is what it costs and what it demands of the run. Three differences, each measured rather than argued:

	Srivastava–Arora–Boneh / TrustFL+	this section
how the arithmetic is made to agree	logged: rounding decisions for near-boundary intermediates, shared with the auditor	specified: the kernel’s own arithmetic is reconstructed — the FMA chain, the split-K partition, the reduction tree, the MUFU table
the run itself	modified: trained at elevated precision and rounded down to the target	unmodified: stock fp32 PyTorch, the vendor’s own heuristic or a same-class forced kernel
the artefact	grows with the computation: for GPT-2 117 M, 92 MB of rounding data per step, 18–20 MB compressed (their Table 2) — ≈ 6 GB for a 300-step run	constant: 119 KB for a 300-step 124 M run, independent of the number of steps or operations
the verifier	another GPU (them), or a GPU-recorded log replayed in an SGX enclave (TrustFL+)	any CPU, no GPU, no enclave, no vendor kernel
what is checked	a sampled fraction, at a stated confidence level	every recorded step, by hash equality, with the sampled protocol of docs/04 §3.1 available on top
audit cost	≈ 1.2× the training step (GPT-2: 11 s train, 13.5 s audit)	270–480× the GPU’s step (40–50 s per 124 M step on 16 x86 threads)

Read honestly, that table says they are two orders of magnitude faster per step and they published first; we are five orders of magnitude smaller in artefact, we leave the computation alone, and we need no accelerator to check it. Both approaches instrument the training run — ours records the operation list and the kernel log, theirs records rounding decisions — but only one of them changes the bytes the run produces in order to make them checkable. A recipe can therefore be written for a run that has already happened under default settings; a rounding log cannot.

The precise claim this section supports, then: exactness obtained by specifying vendor arithmetic rather than logging it, over an unmodified stock-precision run, with a constant-size artefact, verifiable on a CPU. Not “exact replication is new”.

State exactness at a billion parameters has been demonstrated — on one machine, prospectively. Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Models (Abdullah X, arXiv:2508.12220, August 2025; single author, no venue, uncited at the time of reading) replays a training continuation with selected examples zeroed out and reproduces “a separately executed trace oracle bit-for-bit in model and optimizer state” — Pythia 2.8 B on all 2,775,208,960 model-state elements — “under pinned single-GPU environments”, with the original run instrumented in advance and an eligible checkpoint retained. It is the strongest published evidence that state exactness does not break with scale, and it is the same-hardware, same-run case: the replay runs the vendor’s kernels on the vendor’s device with the recorded controls. What it does not do, and does not claim, is what this section is about — a different party, on different hardware, with no trace of the run and no GPU, deciding the bytes. The two results are complementary: theirs says exactness survives 22× our parameter count on the trainer’s own machine; ours says the trainer’s machine is not needed.

Two more literatures bound the problem from either side.

• Determinism is not free on the training side either, and someone has now measured it. DASH (ICLR 2026 submission 23383, openreview.net/forum?id=bMi5ssfPoM, code github.com/SJTU-Liquid/deterministic-FA3) reports that FlashAttention-3’s deterministic backward pass costs up to 37.9 % throughput because gradient accumulation must be serialised, and recovers up to 1.28× of it by scheduling the backward pass as a DAG. Figures are an unrefereed submission’s own. This is the first training-side price for determinism we have found; every other number in circulation (Thinking Machines’ 26 → 42 s, CoRun’s −74 %) is inference-side. It matters here because the oracle taxes neither side: the run is stock, and the determinism is supplied by the verifier’s model of the kernel.
• Recomputation exists, along the wrong axis, and storage is converging on content addressing without it. Discarding a tensor and recomputing it is routine within a step — activation checkpointing in Megatron-LM, selective recomputation, Checkmate, and scheduling work such as Lynx (arXiv:2406.08756), which measures full recomputation at 10–30 % of training time on a 7 B model. Nobody recomputes checkpoints across the steps of a run to save storage; a search of arXiv, Semantic Scholar, OpenReview (including unpublished ICLR/NeurIPS submissions) and the indexed web returned only KV-cache and activation work. The economics of deleting and regenerating derived data, however, is a fifteen-year-old literature: Yuan, Yang, Liu and Chen (IPDPS 2010, doi:10.1109/ipdps.2010.5470453; JPDC 2011; CPE 2010) decide per intermediate dataset whether to store or regenerate by generation cost over usage rate against Amazon’s storage price. Their break-even is t_θ = genCost/(size × CostS) at $0.15 per GB-month and $0.10 per CPU-hour (IPDPS 2010, p. 6; the carbon analogue for KV caches is Bassi et al., MLArchSys 2026). Our keep-interval optimum is their rule on a sequential chain; what is new — checked against their full text, which never varies problem size — is the inversion with model scale, the interval rather than the node as the unit of decision, and the verification term (R10 §g). No optimal-checkpoint-interval model from Young (1974) and Daly (2006) to the 2024 surveys prices storage at all; they optimise time. Meanwhile checkpoint systems keep converging on the storage half — Check-N-Run, Inshrinkerator, LMC, ExCP, and independent efforts like tensorcas (2026-03), “a framework agnostic checkpoint deduplication and delta storage engine” over content-addressable storage. All of them keep the bytes. None promises a checkpoint it does not hold.

8. Reproducibility

Every cell of this programme was registered, with its predictions, in a commit preceding the instance that tested it, and every number in this paper is copied from a result file named in that registration. Registrations, result files, commit hashes and the per-card plan tables are available to reviewers on request. Companions: Measuring, Naming and Removing the Device Dependence of GPU Training Bytes (why GPU training bytes depend on the device and how to remove it) and the protocol note Proof-of-Learning at δ = 0, for which this paper is the evidence.