Determinism as a contract

The rule: model bytes are identical across runs, across thread counts (outside one documented relaxation), and across CPU architectures, and CI enforces it per commit by training a reference model on an arm64 Mac and an x86-64 Linux box and comparing file hashes.

The rule is about CPU training, and the boundary is worth stating plainly rather than leaving to inference. GPU training accumulates histogram cells with device atomics whose landing order is the scheduler's, and for its first year the same fit wrote different model bytes on every run: four repetitions of one cuda_depthwise fit at a single commit produced four different model hashes. The cells are now int64 fixed point, so the sum is the same integer in any order and a device fit is bit-identical to itself on one device and one build; the suite fits three times and compares bytes. What the device plane still does not promise is the rest of the CPU rule: identity across devices and toolkits, where the gradient kernels' transcendentals and fused multiply-adds differ, and identity with the host fit, which accumulates float cells and agrees to 1e-4. A narrower claim honestly made rather than the same claim quietly extended.

This is not an aesthetic. A bit-exact contract turns "did this change alter behavior?" from a judgment call into a file comparison, and that one property pays for itself everywhere.

What the contract caught

Compiler contraction across architectures. The cross-arch gate's first real catch: the same model trained to different bytes on arm64 and x86-64 (test pins 0.71719 vs 0.71725). The mappers were bit-identical; the divergence was clang fusing multiply-add into FMA on one architecture only. Fix: -ffp-contract=off on the host build, measured at zero cost, and the contract upgraded to a claim no reference library makes (decision 59).

A silent build-system fallback. Models differed between two supposedly identical builds; the cause was a missing OpenMP quietly downgrading to serial, which changes reduction order. A missing OpenMP is now a hard configure error (decision 60).

What the contract enables

Refactoring with proof. A recent cleanup deduplicated the numerically sensitive softmax code into one helper. Acceptance was not review confidence; it was np.array_equal against outputs captured before the change. The refactor shipped with a bit-exactness proof instead of an argument.

Features that provably cost nothing when off. New per-row weighting multiplies gradients by 1.0 when unweighted, which is exact in IEEE arithmetic, so every existing model stays byte-identical and the hash gate verifies it. The pattern generalizes: design the off-state of a feature to be arithmetically inert, then let the gate prove it.

AI-assisted development without trust. Large parts of bonsai are written by Claude (built with Claude). The reason that works is precisely this contract: an implementation campaign cannot silently change numerical behavior, because the gate would catch the changed bytes. Verification substitutes for trust at exactly the point where trust is weakest.

Transferring it

Any numerical system can adopt the ladder: fix the reduction orders (deterministic parallelism is a design choice, not a performance sacrifice; bonsai ties XGBoost with it), pin the floating-point environment (contraction, denormals), then hash a reference output in CI on more than one architecture. The gate is a few lines of workflow; the property it enforces is the difference between refactoring a numerical codebase and gambling with it.