The HPC tension

bonsai's backend seams are concepts, one type per backend. This page is where those seams meet performance: the GPU. The honest story is that a concept can only check so much. The rest is held by tests, and by rules the code refuses to break.

The concept is only a syntactic floor

The engine concepts in include/bonsai/grower.hpp carry their own contracts: HistogramEngine requires two methods, GPULevelEngine refines it with the whole device vocabulary as one concept rather than seven, because the device data plane works whole or not at all. What that page hands to this one is the part no requires-clause can hold: populate must accumulate the node's rows into the bins the mappers define, in "an order that is a pure function of configuration," with missing values in the last bin (grower.hpp). A type can satisfy every signature and bend all of it, and the comment names the consequence: it "trains silently wrong models." The compiler cannot see this, but the [cuda] parity suite can. It asserts CPU and GPU agree within 1e-4 (tests/unit/test_cuda_grower.cpp). The contract lives in the suite, and the concept is only its syntactic floor.

The planes divide state by lifetime

The device context splits its resident memory into five planes, each with its own lifetime (src/cuda/detail/device_context.cuh):

Plane	Holds	Lifetime
DeviceData	the binned matrix	once per fit
GradientPlane	per-tree gradients, interleaved to float2	once per tree
LevelPipeline	rows, histograms, staging buffers	once per level
LeafPipeline	the leafwise histogram slot pool and segment map	once per tree
ResidentPlane	labels, scores, and resident-objective state	once per fit

Dividing by lifetime is what keeps an edge honest: an upload done once per fit must never be redone per tree. Naming the planes also makes the boundary crossings countable. That is how the compute-DAG model (scripts/dag_model.py) prices a move before it is played.

The resident objective deleted a boundary instead of optimizing across it

For MSE, LogLoss, or Poisson with all-rows or Bernoulli sampling, the whole per-tree host round trip disappears. Labels and scores upload once into ResidentPlane. Each tree derives its gradients on the card, and the epilogue folds the leaf values back into the resident scores. The lesson from case E4 is the title of this section: delete a boundary, do not optimize across it. The single-GPU 16M levelwise round fell from 104 to 64 ms (decision 78). The resident model proved bit-identical to the host-objective model on a Jetson.

What stays host-side on purpose

The control plane stays on the host by design (decision 41). Split decisions cross the bus down every level, because the grow loop must observe each level's outputs before opening the next. That pins one small device-to-host sync per level, the irreducible floor of about 800 syncs per fit. On a healthy host each costs 10 to 20 microseconds (measured in scripts/dag_model.py's constants). Mapper-fit stays host-side too: its cut points come from a seeded RNG stream. Reproducing that stream on the device would risk the determinism identity for no gain.

The honest cost

The seams forbid two things a looser design would allow. There is no GPU fallback at all, per node or per tree. An engine grows a whole tree resident or refuses it in begin_root, with an error naming the limit it hit and the device="cpu" remedy. A single oversized feature therefore stops the fit rather than quietly moving it to the CPU plane, which is the same standard the benchmark protocol holds the reference libraries to: a run that asks for a device and trains somewhere else reads as a working fit and is not one. There are no cross-plane shortcuts: a placement move must not change accumulation order or precision, and the byte-identity gate catches it if it does.

What that bought:

• The parity suite: one contract, checked at 1e-4, that any engine must meet.
• Bit-identical CPU models across architectures and thread counts, checked per commit (determinism).
• The 1e-4 GPU convention: a cuda_* model is byte-identical to itself run to run (integer histogram cells) but matches the host model to tolerance, not tree for tree, because the host accumulates float cells. The tests assert host-vs-device prediction tolerance and device-vs-device byte equality (invariants).

The trade is stated plainly. The device cannot cut through the seams. In exchange, every model bonsai ships is either bit-reproducible, or provably within 1e-4 of the model that is.