Decisions¶

This log is the historical engineering record, kept for reference and for agents working in the codebase. The curated design pages live in the Design section.

Append-only log. Order = decision order. Caveman style. New entries at bottom.

1. Binning: quantile, with low-cardinality fallback¶

BinMapper::fit per feature.

Equal-frequency cuts at k/max_bin-th quantiles. max_bin = 255 default (uint8 indices).
If n_distinct < max_bin: one cut between each pair of consecutive distinct values. Bucket count = n_distinct.
Dedupe cut collisions (sentinel values like 0.0). Actual count <= max_bin, never exact.
Sampling from the start. Default sample 200K rows uniform random, fixed seed. Configurable. If column has <= sample_size rows, use full column.
Bin 0 reserved for missing. NaN + user-configured sentinel short-circuit to bin 0. Real values bins 1..n_bins-1. Quantile skips NaNs.
BinMapper serializable. Round-trip through model file. Predict on new data reuses train boundaries exact.

Rejected: equal-width (skew kills it). Quantile sketch (overkill, swap in later). xgb per-node default direction (complicates split scoring).

Knock-on: bin count varies per feature, histogram reads n_bins[fid]. Bin 0 special, split scoring skips it for real-valued cuts. BinMapper ownership vs Dataset is next decision.

Defer: min_data_in_bin knob.

2. `BinMapper` independent of `Dataset`. Two-stage API.¶

auto mappers = BinMappers::fit(train_source, cfg);
auto train   = Dataset::bin(train_source, mappers, cfg);
auto val     = Dataset::bin(val_source,   mappers, cfg);
auto test    = Dataset::bin(test_source,  mappers, cfg);

BinMappers is std::vector<BinMapper> plus minimal wrapper (count, serialize). Built once on train, immutable thereafter.
Dataset::bin is pure transform: takes source + mappers, returns binned column-major storage. No "training Dataset" vs "val Dataset" distinction.
Model file serializes BinMappers. Predict-time Dataset builds fresh from them.

Rejected: lgbm-style Dataset::from_csv(..., reference=train_ds). Couples mapper lifetime to a Dataset, awkward serialization, "training Dataset" becomes special.

Knock-on: train path is two calls instead of one. Trivial. bin is single-pass; fit does its own sampling + sort internally.

3. Trees store raw float thresholds, not bin indices¶

Tree node split = (feature_id, threshold: float). Predict reads raw float from input row, compares directly. No binning at predict time.

TreeGrower finds the best split as (fid, bin_idx) during training, then converts to threshold = cuts[bin_idx] when writing the node. Conversion is one lookup per finalized split, free.

xgb + catboost do this. lgbm stores bin indices in tree nodes (and re-bins at predict, which is why lgbm forces the reference-Dataset dance).

Knock-on: - Predict path doesn't need BinMappers. Single tree walk over raw floats. - Model file: trees serialize directly, BinMappers optional in model file (kept for diagnostics + reproducibility, not load-bearing for predict). - Training-time histogram code unchanged: still bins, still works on bin indices internally. - Float threshold means tree comparison is < on float, not <= on int. Watch for off-by-one when comparing parity vs lgbm (different convention).

4. `Dataset` storage layout¶

Column-major. Per-feature std::vector<uint16_t> (uniform width). Labels + weights owned by Dataset (weights empty if uniform). BinMappers held by value (not shared_ptr); ~30KB copy is trivial, no shared mutable state.

class Dataset {
    std::vector<std::vector<uint16_t>> features_;
    std::vector<float>                  labels_, weights_;
    BinMappers                          mappers_;
    std::vector<bool>                   is_categorical_;  // Phase 4 placeholder
    // n_rows, n_features
};

Public API: n_rows(), n_features(), labels(), weights(), mappers(), n_bins(fid), is_categorical(fid), feature_bins(fid) -> span<bin_id_t const>.

Rejected: std::variant<vector<uint8_t>, vector<uint16_t>> per feature to save ~50% on binned column memory. Saves ~45MB on YearPredictionMSD, ~308MB on Higgs. Neither pressure-tests modern hardware. Cost was variant dispatch complexity at every column scan via a visit_column wrapper. Rejected for MVP; reversible if a future dataset makes memory the bottleneck.

Group columns (ranking) deferred (non-goal).

5. (reserved)¶

Originally visit_column for variant-aware column access. Dropped when decision 4 collapsed to uniform uint16_t storage. Renumbering decisions breaks references; left as a placeholder.

6. Readers: free function per format, returning `Dataset`¶

Dataset read_csv    (const std::string& path, const DataConfig&, const BinMappers&);
Dataset read_parquet(const std::string& path, const DataConfig&, const BinMappers&);  // Phase 4+
Dataset read_libsvm (const std::string& path, const DataConfig&, const BinMappers&);  // later

BinMappers fit_from_csv(const std::string& path, const Config&);

CLI dispatches on cfg.data.format string: if "csv" call read_csv else if "parquet" .... Each reader its own translation unit. Adding a new format = new file + new branch in CLI dispatch.

No Reader concept, no abstract base, no template plumbing. File loading is once-per-program, not hot-path; concepts buy nothing here. Internal shared helper per reader (CsvReader::columns(path, cfg) -> ColumnBatch) keeps read_csv and fit_from_csv from duplicating logic.

Phase 1 ships CSV only, hand-rolled (~50 LOC). Numeric-only is enough for YearPredictionMSD. No Arrow, no parquet.

Phase 4+: Arrow optional, gated by BONSAI_PARQUET=ON CMake flag. Arrow handles CSV multi-threaded + parquet + feather + IPC. Heavier dep (transitive thrift/snappy), so kept optional. Arrow is the reader layer only. Dataset (binned storage + BinMappers + labels/weights) is bonsai's; Arrow's Table is raw column data we'd copy or borrow from.

Rejected: Reader concept + Reader auto& template params for fit and bin. Over-engineered for once-per-program file loading. Free functions are the simpler shape.

7. Determinism contract: fixed thread count, not cross-thread¶

Same seed + same data + same thread count → same model bytes. Different thread counts: predictions within numerical tolerance, but bytes may differ.

What this rules in: - Per-thread local histograms (no atomic FP adds: those are bit-unstable even at fixed thread count). - Deterministic chunking (e.g., OpenMP schedule(static)). - random_seed carries through samplers / shufflers.

What this rules out (relative to earlier framing): - Promising cross-thread bit-exactness. The earlier draft demanded fixed-order merge (tid outer, bin inner) so the per-thread reduction shape didn't depend on thread count. Dropped: costs design constraints on ParallelBackend (must expose ordered reduction primitive) and forecloses OpenMP reduction(+:...) and std::execution reduce shapes.

Field check: XGBoost and CatBoost don't promise cross-thread determinism. LightGBM offers it behind deterministic=true + force_col_wise|row_wise, and its own maintainers describe the guarantee as fragile (RFC #6731). The pragmatic, industry-standard contract is "thread count is part of the reproducibility input."

Test contract: - test_determinism_fixed_threads: two runs at n_threads=k for k ∈ {1, 4, 8} produce identical model files. Required to pass. - test_determinism_cross_threads: predictions across different thread counts agree to numerical tolerance (e.g., max abs diff < 1e-5 on YearPredictionMSD). Required to pass.

Knock-on: - ParallelBackend does not need to expose "ordered reduction" as a primitive. parallel_for + thread-local accumulators is enough. - The histogram's parallel-build description in architecture/2-histogram.md §"Parallel construction" reflects this: no fixed-tid-order requirement. - Atomic FP adds remain forbidden, but for the bit-stability-at-fixed-N reason, not the cross-N reason.

8. Two trees in Phase 1: depth-wise + oblivious¶

DepthwiseGrower → DenseTree and ObliviousGrower → ObliviousTree both ship in Phase 1. Proposal puts oblivious in Phase 4; pulled forward to force the Tree concept and TreeGrower::Tree associated type to be honest from day one.

The two tree types have structurally different on-disk shapes (flat node array vs per-level splits + leaf table) and structurally different predict kernels (walk-until-leaf vs fixed-depth branchless gather). With only depth-wise shipping, the second-tree-type machinery would be aspirational. Same rationale as logloss alongside MSE in Phase 1 (proposal §1).

Cost: one extra grower + one extra tree type of spine code, plus a second parity target (depth-wise vs xgboost/LightGBM, oblivious vs CatBoost). Both targets share YearPredictionMSD.

9. `Tree` is a concept; minimum surface = predict ×2 + diagnostics¶

template <typename T>
concept Tree = requires(T const t,
                         std::span<float const> row,
                         std::span<float const> rows, size_t n_features,
                         std::span<float> out) {
    { t.predict(row) }                       -> std::same_as<float>;
    { t.predict(rows, n_features, out) }     -> std::same_as<void>;
    { t.n_leaves() }                         -> std::convertible_to<size_t>;
    { t.depth() }                            -> std::convertible_to<size_t>;
};

Concept, not abstract base. Booster<Gr, ...>::trees_ is std::vector<typename Gr::Tree>, monomorphized. No vtable on predict.

Two predict overloads under one name (single-row returns float; batch fills out). Disambiguation by arity. Row-major batch input, matches xgb / lgbm. CatBoost's column-major fast path is motivated by predict- time rebinarization, which we don't do (decision 3: float thresholds). Oblivious can transpose-on-demand internally if profiling justifies.

Rejected: leaf_index(row) (DenseTree and ObliviousTree leaf-index spaces aren't unified; defer to Phase 4 if SHAP / leaf-output predict is wanted); walk(visitor) (no shared node shape between the two impls); serialization on the concept (lives in bonsai::io per decision 6).

10. Shrinkage is baked into leaf values at tree construction¶

Grower receives learning_rate as a constructor argument (in TreeConfig) and writes lr · -G/(H + λ_l2) into leaves. Trees are pure functions of input rows; predict has no learning-rate knowledge. Matches xgboost, LightGBM.

Rejected: per-iteration learning_rate argument to grow() (Phase 1 doesn't need decay schedules; non-breaking to add later as a grow overload).

Knock-on: Tree concept doesn't carry a set_shrinkage mutator; trees are immutable post-construction.

11. `ObliviousTree`: per-level `default_left`, not per-node¶

Every node at level d of an oblivious tree shares the same (feature_id, threshold, default_left) triple: that's the symmetric- tree contract. Relaxing default_left to per-node-at-level recovers a small accuracy edge in heavily-missing data but breaks the branchless predict kernel and isn't what CatBoost does.

LevelSplit { uint32_t feature_id; float threshold; bool default_left; } × depth. Leaf table of size 2^depth.

When a level's feature has no missing rows in training data, default_left is don't-care; the splitter records whichever orientation scored higher (arbitrary if no missing rows existed) and predict honors it without thinking.

12. Sampler is the booster's responsibility, not the grower's¶

grow(ds, grad, hess, row_indices). The grower receives sampled row indices; it doesn't know what sampler produced them. "Use all rows" is just row_indices = 0..n_rows-1.

Keeps TreeGrower concept narrow. Sampler swappable via Booster<Obj, Gr, Sa, Backend> independent of grower choice. Determinism contract testable on the grower without re-wiring the sampler.

Rejected: grower owns sampler as a member (lgbm-style). Couples grower template to sampler template; bigger cartesian product on Booster instantiations for no compositional gain.

13. Branchless NaN routing in predict¶

bool is_nan  = std::isnan(v);
bool less    = !is_nan && (v < threshold);
bool go_left = less | (is_nan & default_left);

Compiles to mask / select on x86-64 and ARM. Preserves vectorizability for ObliviousTree's batched predict; also keeps DenseTree's single-row predict tight.

Rejected: branchful if (isnan(v)) ... else ... (loses oblivious's SIMD story); pre-cleaning predict input (caller can't know per-feature default direction; rules itself out).

Knock-on: predict-time sentinels (e.g. -999) declared in BinMapperConfig are not honored at predict, only std::isnan. Caller contract: convert sentinels to NaN before predict. Matches xgb / lgbm.

14. Splitter is a template parameter on the grower; one `SplitFinder` concept¶

template <SplitFinder Sp = HistogramSplitFinder> class DepthwiseGrower;
template <SplitFinder Sp = HistogramSplitFinder> class ObliviousGrower;

Static dispatch through to the splitter; inlined into grow(). Default makes the common case ergonomic; explicit Sp makes the extension API real.

struct SplitCandidate {
    uint32_t feature_id;
    uint16_t bin_idx;        // grower converts to threshold via cuts[bin_idx]
    bool     default_left;
    double   gain;
    bool     valid;
};

template <typename T>
concept SplitFinder = requires(T const f,
                                std::span<Histogram const> hists,
                                Dataset const& ds,
                                double sum_grad,
                                double sum_hess) {
    { f.find(hists, ds, sum_grad, sum_hess) }
        -> std::same_as<SplitCandidate>;
};

Both growers consume the same vector<Histogram> shape and produce the same SplitCandidate. Depth-wise calls find once per frontier node with that node's histograms; oblivious calls find once per level with the folded level histograms. The splitter doesn't know or care whether its input is per-node or level-pooled.

Earlier draft of this entry split this into PerNodeSplitFinder / LevelSplitFinder to make mismatched grower/splitter pairs a compile error. Rejected: histogram-based scoring collapses the two signatures to the same shape, so the "compile-time rejection" was illusory. Phase 4 splitters that don't fit this shape (e.g. an exact splitter scanning raw rows) earn their own concept when written.

Rejected: type-erased unique_ptr<SplitFinder> member (the dynamic- dispatch shape we explicitly chose against in proposal §3.4).

15. Splitter returns one best candidate per call¶

find(...) returns a single SplitCandidate. valid = false if no positive-gain split exists or no candidate clears min_gain_to_split.

Rejected: returning all per-feature candidates and letting the grower pick max. No caller wants this; flexibility nobody's asking for.

16. Splitter tie-break: lowest `fid`, then lowest `bin_idx`¶

When two candidates have equal gain (within bit-exact equality, not tolerance), prefer the one with the lower feature_id; if those tie, prefer the lower bin_idx. Stable, deterministic at fixed thread count. Matches lgbm's tie-break order (xgb's is implementation- dependent in their hist updater).

Knock-on: same convention applies to the smaller-sibling choice in the subtraction-trick wiring: when n_left == n_right, left wins.

17. Partitioning: per-node row-index lists (strategy A)¶

Each live FrontierNode carries its own std::vector<uint32_t> of row indices. At root: one list with the booster-supplied row_indices. At each split: partition the parent's list into (left_rows, right_rows), replace parent in frontier with two children carrying the new lists.

Both growers use this strategy (oblivious folds per-node histograms into level histograms at scoring time; ~1.5% overhead on YearPredictionMSD-scale data).

Rejected: single row_to_node array of length n_rows rebinned on each split (xgb's hist updater shape; lgbm's voting parallel mode). Beats per-node lists on cache locality at very shallow depth, loses at typical max_depth = 6. More importantly: the subtraction trick wires naturally onto per-node histograms. A single rebinned position vector either fights subtraction (the all-live-nodes-in-one-pass kernel doesn't know to skip the larger sibling) or imports per-node branching back into the kernel.

Knock-on: oblivious grower needs per-parent gain summation across the frontier: for each candidate (feature, bin), sum score(left, λ) + score(right, λ) − score(parent, λ) over every parent (CatBoost's symmetric-tree gain; see decision 30). Same O(n_features · n_bins · |frontier|) order as a fold; the difference is in what is accumulated (gain, not histogram cells). An earlier version of this knock-on said "needs a fold step (level_hists = sum_per_feature(...)) before split scoring"; that was wrong because score(g, h) is non-additive; corrected 2026-05-22.

18. Frontier holds histograms inline (no histogram pool)¶

FrontierNode { rows, sum_grad, sum_hess, hists }. vector<Histogram> per node, sized n_features. The grower carries std::vector<FrontierNode> frontier and rotates it level-by-level.

Lists and histograms are local to grow(); the grower is stateless across calls (one boosting iteration → one allocation cycle). Matches lgbm's SerialTreeLearner.

Rejected: histogram pool with slot-id indirection (xgb's hist updater). Recycles allocations across the tree, but Phase 1 isn't allocation-bound. The pool refactor is contained if profiling later shows the allocator dominating.

19. Subtraction trick from day one, both growers¶

Build smaller child by row-scan; derive larger by larger_hist = parent_hist - smaller_hist. Halves histogram-build work across the tree (per 2-histogram.md §"Why subtraction halves it"). Implemented from day one because retrofitting means restructuring the grower's per-node memory.

Per-parent protocol: 1. Splitter scores parent.hists; commit candidate. 2. Partition parent.rows into (left_rows, right_rows). 3. Pick smaller (left wins ties). 4. Build smaller's hists by row-scan. 5. Derive larger's hists by parent_hist - smaller_hist. Histogram carries its own (total_grad, total_hess) so operator-= subtracts cells and totals together. 6. Push left, right into new frontier in left-then-right order (frontier order is structural, independent of build order). 7. Parent's hists released when parent goes out of scope.

For oblivious: same per-parent protocol, run once per parent at each level. Cross-level subtraction (deriving level d+1's level histogram from level d's) doesn't help: different levels score on different features, so per-feature histograms for level d+1 have to be built regardless.

20. Phase 1 regularization knobs¶

TreeConfig fields:

Knob	Default	Meaning
`max_depth`	6	Hard cap on tree depth.
`min_data_in_leaf`	20	Node row-count floor (and child row-count floor before splitting).
`min_sum_hessian_in_leaf`	1e-3	Effective row-count floor under non-MSE objectives.
`lambda_l2`	1.0	L2 reg on leaf weights, in gain formula and leaf value.
`min_gain_to_split`	0.0	Minimum gain to accept a candidate.

Validated in grower constructors; ConfigError with key path on bad values. Section is [tree] in TOML.

Rejected for Phase 1: max_leaves (leaf-wise concept; depth-wise's natural cap is max_depth, oblivious's leaf count is 2^depth exactly).

Leaf value: leaf_value = learning_rate · -G / (H + lambda_l2), applied at finalization (decision 10).

21. `Objective` is a concept; static methods, no instance state¶

Matches the SplitFinder shape (decision 14). Two static functions required: compute(preds, labels, grad, hess) writes per-row gradients and hessians; eval(preds, labels) returns a scalar mean loss. Dispatch is at the Booster<Gr, Obj, ...> template parameter, fixed at compile time. No vtable; no shared mutable state across calls.

Rejected: virtual base class (loses compile-time dispatch); concept-with-instance-methods (no Phase 1 objective needs instance state).

22. `Objective` does not own initial score or link inverse¶

The first-tree bias prediction (mean for MSE, log-odds for logloss) and the predict-time link inverse (sigmoid for logloss) live in the booster, not the objective. Rationale: both are score-accumulator concerns. The booster maintains the running raw-score prediction array, decides whether the bias comes from config or labels, and is the sole owner of the predict path. Pushing them into Objective would either force every objective to expose a transform and an initial_score it might not need (e.g. MSE: transform is identity), or invite a fragmenting set of optional methods on the concept.

See 5-booster.md (TBD) for where these land.

23. Phase 1 objectives: MSE + binary logloss, single-output¶

Two MVP impls satisfy the Objective concept: MSEObjective (regression) and LogLossObjective (binary classification). Both consume 1D per-row floats_view for preds and labels, write 1D floats_out for grad and hess.

Rejected for Phase 1:

Multi-class / softmax. K-output extension; touches Objective, Booster, Tree (K-output leaves). Phase 4.
Quantile, Huber, Tweedie, Cox. Out of scope; satisfy the concept when added.
Custom user objectives. No registry needed: anyone satisfying the concept can drop in as a Booster template parameter.

24. `compute` writes raw-score grad/hess; output is overwritten, not accumulated¶

Objective::compute writes grad and hess outright; callers don't zero buffers first. preds are raw scores throughout (logloss does not apply sigmoid inside compute or eval); the booster keeps an additive raw-score accumulator across iterations and applies the link only at the outermost predict call. Matches xgboost / LightGBM; keeps boosting math additive.

hess is always non-negative (MSE: 1; logloss: p·(1−p) ∈ (0, 0.25]); the splitter's min_child_hess (decision 20) catches near-zero hessians before they propagate.

25. Sample weights are applied by the booster, not the objective¶

When Dataset.weights is non-empty, the booster multiplies the grad and hess buffers by the weight vector immediately after Objective::compute returns and before handing them to the grower. Keeps every Objective impl focused on loss math; the multiplication is a 2-line buffer loop that doesn't belong duplicated across objectives.

Rejected: a WeightedObjective<T> wrapper that satisfies Objective by composing with weights. Adds a template layer with no semantic content; same effect as the booster-side multiply.

26. Dispatch: flat table over cartesian product, `IBooster` at boundary¶

Status 2026-08-26: the shipped registry is three axes, cartesian_product_t<Objectives, Growers, Samplers>; a Splitters axis never shipped and DispatchConfig carries three names (include/bonsai/registry/typelists.hpp). The Candidate A deliberation this entry cites lives at the pinned archive.

Runtime → static boundary uses Candidate A from architecture/6-dispatch.md: a constexpr std::array keyed on a name-tuple, generated by for_each_type over cartesian_product_t<Objectives, Growers, Splitters, Samplers>. Each cell is a monomorphized Booster<O,G,S,Sa> factory. Lookup at the config boundary returns unique_ptr<IBooster>.

Cost: one virtual call per update_one_iter. Acceptable. The hot path is histogram building inside update_one_iter, not the call itself; per-iteration vcall is dwarfed by the per-iteration histogram pass. Static-everywhere is preserved inside the iteration body, which is what the proposal §3.4 rule actually targets.

Rejected:

Nested registry callbacks (Candidate B). Fully static, zero vcalls anywhere, but forces continuation-passing at the boundary and drags the whole training run into the innermost lambda.
Dressed-up nested lambdas (Candidate C). No advantage over A or B once the type-level builder is in play.
Hybrid flat-table + generic callback. Doesn't compile: std::array of function pointers can't be generic over the callback type.

Invalid combinations (none in MVP) are pre-filtered at typelist construction by building sub-products over compatible sub-typelists and concatenating. No runtime check, no concept predicate, no instantiation of bad cells.

Backend placement deferred to 7-parallel.md; dispatch stays 4D for now; promotion to 5D or separate composition both stay open.

27. Booster shape, training loop, and Sampler concept¶

Status 2026-08-26: the code has outrun the ratified shape: Booster takes three template parameters, grow() returns leaf values so the loop re-walks nothing, and the identity sampler is AllRowsSampler. include/bonsai/booster.hpp is the reference; the ratified doc lives at the pinned archive.

Ratifies architecture/5-booster.md.

Class shape. IBooster is the boundary erasure type with a minimal CLI-facing virtual surface (update_one_iter, eval, predict, n_iters, accessors for save/load). Booster<Obj, Gr, Sp, Sa> is the real class, monomorphized per cell of the dispatch table (decision 26). One vcall per update_one_iter from the CLI; zero inside.

Training loop (update_one_iter). Six steps in order:

Obj::compute(scores_, labels, grad_, hess_): full row set.
Apply Dataset.weights to grad_, hess_ if present (decision 25).
Sa::sample(grad_, hess_, rng, out_indices) → row indices for the grower.
grower.grow(dataset, grad_, hess_, row_indices, split_finder, cfg) → tree.
scores_ += learning_rate * tree.predict(train_rows): re-walks the tree per training row.
trees_.push_back(std::move(tree)).

Sampling runs after grad/hess (not before) because GOSS samples on |grad|, the grad pass is small (~5-10%) compared to histogram building (60-80%, what sampling actually targets), and reordering would force a "does this sampler need grad?" flag on the concept.

learning_rate. Applied at score-update time (step 5), not pre-scaled into leaf values. Saved trees carry raw leaf values; the booster reapplies the rate at predict and at score-update. Matches xgb's "shrinkage = booster concern" model.

Score update via re-predict, not cached leaf values. Step 5 calls tree.predict(train_rows) rather than reading a precomputed (row → leaf_value) array from the grower. Cost is one extra O(n_rows × depth) walk per iter, small at MVP scale. xgb / lgbm / catboost all cache the row→leaf mapping as a byproduct of growing and reuse it for the score update; bonsai defers that optimization to Phase 2 (after benchmarking). When justified, change grower return from Tree to (Tree, std::vector<float> train_leaf_values) and replace step 5 with a flat add. Pure additive change to the grower→booster boundary; Tree stays clean.

Initial score (bias). Booster owns it (decision 22). Three sources in priority: cfg.init_score; objective-appropriate default from labels (mean for MSE, log(p/(1-p)) for logloss); value loaded from disk for a continued booster.

Predict path. Raw-score predict sums tree predictions plus init_score, applies learning_rate per tree. User-facing predict applies the objective's link inverse (identity for MSE, sigmoid for logloss) via if constexpr on Obj. Inverse link lives in the booster (decision 22), not in Objective::compute (decision 24).

Obj is purely-static. No instance member; T::compute and T::eval are static (4-objective.md / decision 21).

Booster borrows Dataset. Lifetime sits with the CLI; update_one_iter takes Dataset const&. Saved model is BinMappers + trees + init_score, no Dataset. Booster does not own BinMappers (decision 3: predict path doesn't need them).

Save / load are I/O, not booster methods. Free functions save_booster(IBooster const&, path) and load_booster(path) -> unique_ptr<IBooster>. Rationale: load needs the four component types before it has a Booster to dispatch on, so it reads names from disk and calls into the same registry path make_booster uses: that's structurally the dispatch boundary, not a member function. save is the symmetric counterpart.

Sampler concept (Phase 1). Static members, same shape as Objective and SplitFinder:

template <typename T>
concept Sampler = requires(floats_view grad, floats_view hess,
                           std::mt19937& rng,
                           std::span<size_t> out_indices) {
    { T::sample(grad, hess, rng, out_indices) } -> std::same_as<std::size_t>;
};

Returns the count of selected indices written into the head of out_indices; the buffer is owned by the booster and reused across iterations. RNG is passed in by the booster, which owns determinism (decision 7).

NoSampler is the Phase 1 identity impl (writes 0..n_rows-1, returns n_rows). GOSS and BernoulliSampler are Phase 4. The sub-product machinery from 6-dispatch.md handles any sampler-grower incompatibilities. Sampler doc folds into 5-booster.md for now; spin out into 5b-sampler.md if it grows.

Rejected:

Obj as instance member. No state needed in MVP; revisit if a future objective needs config.
Cache row→leaf mapping in MVP. Adds grower→booster API surface before benchmarking justifies it.
IBooster::save / IBooster::load as virtual methods. Load has no Booster to dispatch on; both belong in an I/O module.
Sample-before-grad ordering. GOSS dependency + small grad cost + concept-flag avoidance.

28. Spine complete; insert Phase 2.5¶

Milestone (2026-05-18). Phase 1 (Serial MVP) and Phase 2 (benchmark harness) are complete. The spine is end-to-end working on California Housing: Dataset, BinMapper, BinMappers, Histogram, depth-wise + oblivious TreeGrower, DenseTree, ObliviousTree, histogram SplitFinder, Objective concept + MSEObjective + LogLossObjective, Sampler concept + AllRowsSampler, Booster<O,G,Sp,Sa> + IBooster, registry / dispatch flat table, dispatch resolution doc (6-dispatch.md). The Python sidecar runs bonsai vs xgboost / lightgbm / catboost on the same TOML config. Eval baseline pinned at rmse=0.7175214 (regression net via tests/unit/test_eval_baseline.cpp).

Phase 2.5 inserted. Between Phase 2 and Phase 3, before turning on parallelism, the next focus is a CLI / config usability and design pass plus the small items glossed over during the Phase 1/2 sprints. No new spine, no parallel backends. Items captured as commits during the work rather than pre-listed. Phase 3 (Parallelism) follows; YearPredictionMSD becomes the perf benchmark there.

Parallel backends come after a design pass. The two remaining spine items (ParallelBackend concept + first impl) wait on architecture/7-parallel.md, which fixes the threading-model design calls before any backend code is written.

29. Model file serializes the full `Config` via NLOHMANN macros, not the existing TOML codec¶

On-disk format (2026-05-20). save_booster(IBooster const&, path, BinMappers const&, Config const&). Full Config rides along in the msgpack envelope under "config". load_booster reads it straight into LoadedBooster::cfg and calls make_booster(out.cfg): no synthesized-Config indirection (was {dispatch, learning_rate} torn apart at save and stitched back at load). Format version bumped 1 → 2; no v1 artifacts in the repo, fail-loud on stale files is correct.

Mechanism. NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE per Config sub-struct (Data, BinMapper, Tree, Booster, Dispatch, Metrics) plus one for Config itself. Same macros are used for the tree-node POD records (DenseTree::InternalNode / LeafNode / Params), replacing ~65 lines of hand-typed JSON-key mirror code. One-shot adl_serializer<std::optional<T>> added in the same TU: nlohmann v3.11 doesn't ship optional support. Every macro lives in src/io/model.cpp (no public-header nlohmann leakage).

Why not the existing TOML codec. The TOML path uses Section descriptors + field_name<MemPtr>() (source_location-based extraction) to drive serialization with zero per-struct boilerplate. The JSON path could reuse those Section tuples the same way: that was the originally-proposed approach. Rejected in favor of the macros: less code (~10 macro lines vs ~30 lines of generic fold + two new files in bonsai::config), one fewer abstraction layer to read when debugging the on-disk format, and the macro form lives next to the only consumer (model.cpp). Trade-off explicitly accepted: the macro field-list duplicates the existing Section field-list. A member added to a struct but not to one of the two enumerations silently drops from that serializer.

Why not modern C++ reflection. P2996 (static reflection, C++26) collapses every NLOHMANN macro in this file to one template per direction. Not available in our toolchain. When it lands, this decision is the natural retirement point: delete the macros, ship the reflection-based serializer, format version stays at 2. Boost.PFR is positional-only (no field names) and brings a dependency for no JSON-key benefit. Rejected.

Inspection. Model files are now jq-able as a structured tree: nlohmann::json::from_msgpack(read_file(path)).dump(2) yields {magic, version, config: {data, bin_mapper, tree_config, booster_config, dispatch, metrics}, bin_mappers, init_score, trees}. This was a load-bearing factor in the encoding choice: a TOML-string-in-JSON alternative (~2 lines using dump_toml/parse_toml) would have hidden Config behind one escaped-string blob and broken jq access to individual fields.

Knock-on. Every Config sub-struct gained bool operator==(...) const = default; so round-trip tests can assert loaded.cfg == cfg in one line. Also widened the existing [model_io][config] test to populate every Config leaf with a non-default value (covers each leaf-type's nlohmann conversion in one shot).

30. `ObliviousGrower`: fold-then-score was wrong, revert and re-spec¶

Date. 2026-05-21 (landed), 2026-05-22 (reverted).

What landed (2026-05-21)¶

ObliviousGrower<SplitFinder SplitterT> in include/bonsai/grower.hpp + src/grower.cpp. The level-scoring step folded per-feature histograms across the frontier into one summed level histogram, then called the existing single-node HistogramSplitFinder::find on that fold. Registered as impl_name = "oblivious" in the Growers typelist; the registry's cartesian product picked up {mse, logloss} × oblivious × all_rows automatically. Model I/O was generalized to be tree-type-polymorphic (try_save_as<B> / try_load_into<B> use typename B::tree_type; new tree_to_json overload + tree_from_json<TreeT> specialization per tree type). Histogram::operator+= added to drive the fold.

What went wrong¶

The gain function score(g, h) = g²/(h + λ) is non-additive:

score(Σ g_i_L, Σ h_i_L) + score(Σ g_i_R, Σ h_i_R) − score(Σ g_i, Σ h_i)
  ≠
Σ_i [ score(g_i_L, h_i_L) + score(g_i_R, h_i_R) − score(g_i, h_i) ]

Folding histograms before scoring gives the first expression. The gain induced by applying one split to every parent in the frontier is the second expression. Bonsai's fold-then-score therefore did not compute the right gain.

The bug surfaced during depth=2 testing as a "fold equals root histogram" property: because rows are partitioned (not removed) across the frontier, the fold at level k+1 reconstructs the root histogram exactly, so the splitter re-picked the same (feature, bin) at every level and produced degenerate trees where 2^depth − 2 leaves were empty. I documented this as inherent to oblivious + basic gain. It isn't; it's an artifact of the wrong gain function.

Verification against CatBoost (2026-05-22)¶

Inspected catboost/private/libs/algo/greedy_tensor_search.cpp. The symmetric-tree path calls CalcBestScore → CalcStatsAndScores, which builds per-leaf histograms across the current depth's leaves and aggregates gains via SetBestScore. Per-parent gain summation confirmed. The fold-then-score approach has no CatBoost analog.

Resolution (2026-05-22)¶

Reverted the broken implementation in one commit:

ObliviousGrower<SplitterT> declaration removed from include/bonsai/grower.hpp.
make_level_node helper + ObliviousGrower::grow impl + explicit instantiation removed from src/grower.cpp.
Histogram::operator+= removed from include/bonsai/histogram.hpp.
Growers typelist reverted to TypeList<DepthwiseGrower<...>> in include/bonsai/registry/typelists.hpp; impl_name<ObliviousGrower<...>> removed from include/bonsai/registry/names.hpp.
Two Booster<..., ObliviousGrower<...>, ...> explicit instantiations removed from src/booster.cpp.
tests/unit/test_oblivious_grower.cpp deleted; CMake entry removed; oblivious cases removed from test_make_booster.cpp and test_model_io.cpp. The ObliviousTree-specific tree_to_json overload and tree_from_json<ObliviousTree> specialization in src/io/model.cpp were also removed because -Werror=unused-function would otherwise fire.

What was kept¶

Infrastructure that is independently valid (does not assume the broken impl):

ObliviousTree::splits() / leaf_values() accessors on include/bonsai/tree.hpp. Needed by I/O once the correct grower lands.
using tree_type = typename Gr::Tree; public alias on Booster in include/bonsai/booster.hpp.
Tree-type-polymorphic try_save_as<B> / try_load_into<B> in src/io/model.cpp (uses typename B::tree_type), and the NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE macros for ObliviousTree::LevelSplit / Params.
tests/unit/test_grower_helpers.hpp shared fixtures, grower-agnostic, still used by test_grower.cpp.

Outstanding¶

Decision 8's Phase-1 commitment (depth-wise + oblivious) is not honored. The design-review drift flag at reviews/2026-05-19-design-review.md §"DenseTree / ObliviousTree" remains accurate. The correct algorithm is documented as a design target in architecture/3-tree.md §"Oblivious grow loop"; implementation pending in Phase 2.5 (user-authored).

Lesson¶

Mathematical primitives need a sanity check before being trusted across an aggregation boundary. score(g, h) looks superficially linear in g, but the g² term kills additivity. Two more checkpoints that should have caught the bug earlier:

Cross-reference against the reference library before implementing. CatBoost's source explicitly aggregates gains per-leaf. A 10-minute read of greedy_tensor_search.cpp would have surfaced the right shape before any code was written.
Design tests that distinguish the right answer from a plausibly-wrong one. The original depth=2 test was content with "4 leaves, structurally correct" and even rationalized the degenerate [2 non-empty, 2 empty] outcome. A test that demanded "all 4 leaves carry rows" or "level-1 split differs from level-0 on some non-trivial fixture" would have failed loudly.

31. `LeafwiseGrower`: best-first growth on a gain-keyed heap, `max_leaves` primary¶

Third grower, dispatch.grower_name = "leafwise" (LightGBM's default strategy). A std::vector<Candidate> maintained with std::push_heap/std::pop_heap holds every expandable leaf ({SplitInput, SplitOutput, depth}) keyed on split gain; each pop converts one leaf into two children, so live_leaves counts up and growth stops at TreeConfig::max_leaves (new field; 0 = unbounded, max_depth stays as the cap). Reuses make_root / split_node / finalize_as_leaf / HistogramNodeSplitFinder unchanged and emits a DenseTree, so registration is just the typelist + impl_name edits.

std::vector + heap algorithms over std::priority_queue: top() returns const&, which fights moving the histogram-heavy SplitInput out; pop_heap + std::move(heap.back()) doesn't.
Tie-break: equal gains resolve to the lower node id (FIFO-ish), so trees are deterministic.
Semantics: with max_leaves = 2^max_depth and a separable dataset, leafwise reproduces depthwise's tree exactly (covered by unit test).

32. Parallelism: OpenMP behind a one-function seam, determinism at any thread count¶

bonsai/parallel.hpp exposes parallel::for_each_index(n, f): an OpenMP parallel for (dynamic schedule, chunk n/(threads*4)) with a serial fallback when OpenMP is absent, plus set_n_threads fed from a new [parallel] n_threads config section (0 = all cores). Every parallel site assigns each index to exactly one thread and performs no cross-thread reductions: per-feature histogram fill, per-feature split scans (per-feature bests merged serially in feature order, preserving the tie-break), row-wise predict, objective grad/hess, score updates, CSV row parsing, binning, mapper fitting.

Consequence: models and predictions are bit-identical to a serial run at any thread count: stronger than the proposal's fixed-thread-count contract (decision 7), because the row-parallel-within-feature + per-thread-histogram-merge design that motivates the weaker contract hasn't been needed yet. If it ever is (single-feature datasets), the contract degrades to fixed-N as originally specified.

Rejected for now: the ParallelBackend concept as a 5th dispatch dimension (proposal §3.4, 7-parallel.md TBD). One free function covers every call site today; promoting it to a dispatched component adds a typelist dimension with a single implementation. The seam keeps the door open: std::execution/TBB would slot in behind the same signature.

33. Hot-path perf: ordered gradients, stable scatter, node totals out of `add`¶

Three measured wins on Year Prediction MSD (M2, 8 threads), all bit-identical outputs:

Ordered gradients (LightGBM trick): populate_from_rows gathers grad/hess into node-row order once, so each of the 90 per-feature scans reads them sequentially instead of re-walking two full arrays with scattered indices.
Stable split scatter: split_node replaces std::partition + two assigns with a two-pass exact-size stable scatter. Stability keeps every node's rows ascending (root's are iota), so bin lookups walk memory near-sequentially at every depth.
Node totals once: Histogram::add no longer maintains running totals (2 redundant double-adds per row×feature, duplicated per feature); node totals are one O(n_bins) cell sum over hists[0], hoisted per node in the split finders.

CSV load: whole-file read + line index + row-parallel from_chars parse straight into column-major storage, and the train file is parsed once (mapper fit + binning share the batch) instead of twice. Load 7.4s → 1.3s; 200-iter depthwise fit 73s → ~27s; leafwise ~12s.

34. Feature-parity round: colsample, GOSS, early stopping, L1, and the OOB score bug¶

Four reference-library features landed in one pass, each benchmarked A/B on Year Prediction MSD with the equivalent knob enabled in xgboost / lightgbm / catboost (protocol + tables in the feature-gap record):

tree.feature_fraction: per-tree feature subsample drawn from a grower-owned rng (tree.feature_seed), histograms built for selected features only; unselected slots are zero-binned placeholders the finders skip. Node totals moved to SplitInput::totals() (first populated hist).
sampler_name = "goss": LightGBM's gradient one-side sampling; the Sampler concept now takes mutable grad/hess so the sampler can amplify the small-gradient sample in place.
booster.early_stopping_rounds: incremental valid eval (IBooster::score_base / accumulate_last_tree, one single-tree predict per iteration) + truncate to the best iteration.
tree.lambda_l1: XGBoost-style soft threshold on the gradient sum in both the gain score and the leaf value.

The GOSS benchmark exposed a latent correctness bug in every subsampled path: GrowResult.values was only stamped for sampled rows, so out-of-bag rows' entries stayed 0 and the booster's score accumulator silently diverged from the real model for those rows: their gradients were computed against predictions missing whole trees. Bernoulli had been quietly paying ~2% RMSE for this (9.1873 → 8.9916 on MSD after the fix); GOSS diverged outright (RMSE 24.7, worse than predicting the mean) because it re-selects by |grad| every iteration and fed on its own staleness. Fix: growers now route unsampled rows through the finished tree in bin space (route_unsampled, split bins recorded during growth), which is exact with respect to the float-threshold predict path: bin(v) <= b ⟺ v <= cuts[b] under the right-inclusive binner, missing bin routed by default_left on both paths.

Lesson (rhymes with decision 30): the booster-side values shortcut was only ever validated with all_rows. A contract as easy to state as "every row's train value equals the tree's prediction for that row" deserved a test the day the first subsampler landed.

35. Remaining-gap round: objectives, monotone, interaction, DART¶

Second feature-parity pass (protocol and A/B tables in the feature-gap record); sparse/EFB explicitly stays out of scope until the harness has a sparse dataset to measure against.

Objectives became Config-constructed instances (like Sampler) so parameterized losses carry state: [objective] huber_delta / quantile_alpha. Statics satisfy instance-call syntax, so MSE/LogLoss kept their static methods and only gained trivial ctors. MAE / Huber / Quantile land with sign/clamped/pinball gradients and median/quantile init scores. Known limitation: no leaf-renewal pass, worth ~10% MAE vs lightgbm/xgboost's renewed leaves on YearMSD; bonsai matches lightgbm on huber, where renewal matters less.
Monotone constraints: candidate splits on a constrained feature are rejected when bounded child weights violate the direction, and children inherit midpoint-fenced leaf bounds via SplitInput::lo/hi; whole paths are provably monotone. Costs every library the same ~2% RMSE on CH.
Interaction constraints: SplitInput::allowed/path carries the permitted feature set down the tree; a feature may split only where some group covers the whole path (or alone). Same ~9% RMSE cost as xgb/lgbm on a two-group CH config.
DART: dropout of existing trees with bin-space routing to recover dropped train contributions (no per-tree caches), rescaling with xgboost's normalize_type="tree" factors: the DART paper's 1/(k+1) starves the new tree by ~1/lr under shrinkage and measurably tanked RMSE until replaced. bonsai's DART now degrades less than xgb/lgbm's at the same settings. Incompatible with early stopping by construction (throws).
Oblivious grower rejects monotone/interaction constraints at construction rather than silently ignoring them.

36. Python bindings: nanobind over the CLI's own seams, static libomp¶

_bonsai (nanobind, python/bonsai package, pip install . via scikit-build-core) wraps exactly the seams the CLI uses: config:: apply_overrides for params (dotted keys, same codec as --set), cli::train_with_progress for fit (so early stopping and valid sets come for free), io::save_booster/load_booster for model files interchangeable with the CLI. No training or prediction logic lives in the binding. The sklearn-ish BonsaiRegressor accepts first-class knobs plus a params dict of dotted config keys. Parity test: same config through the module and the CLI agrees to atol 2e-4 on California Housing predictions.

The libomp lesson. Linking the extension against Homebrew's libomp.dylib deadlocked the process the moment xgboost built a DMatrix: a sample trace showed one OpenMP call stack spanning two different libomp images (ours and xgboost's bundled copy): classic duplicate-runtime interposition. Fix, standard for wheels: BONSAI_OPENMP_STATIC=ON links libomp.a into the module and -Wl,-exported_symbol,_PyInit__bonsai strips every other export (1015 leaked kmp symbols before). Verified by interleaving bonsai / xgboost / lightgbm training in one process.

Native benchmark rows. compare.py adds in-process "(native)" rows when build/python is importable, timed like the reference libraries (no subprocess, CSV, or model-save overhead, though bonsai's train() still includes binning, which xgb's timed train does not). Re-baselined: RMSE identical to the CLI rows; predict drops 0.20s -> 0.08s on YearPredictionMSD, landing between xgboost (0.017) and lightgbm (0.105).

37. Feature importance recorded at grow time; the guide series¶

Importance. Split-count and gain importance ship behind IBooster::feature_importance(ImportanceType), bonsai importance, and the Python feature_importances_ / importance(type) surfaces. The one design decision: gain is stamped when the split is created (split_gains per node on DenseTree, level_gains on ObliviousTree, serialized, format v5) because it is not reconstructible from a stored tree. Accumulation is a 20-line walk in booster.hpp. Verified by cross-library agreement on California Housing: bonsai and lightgbm agree on both types including their disagreement with each other: gain crowns MedInc, split-count crowns Longitude/Latitude (many fine-grained, individually-small splits), the textbook argument for gain as the default and a finding the agreement test now pins.

The guide. docs/guide/ is a nine-chapter pedagogical series (concept → math → the actual implementing code → runnable experiment → war story) positioned as a deliberate differentiator: reference libraries document parameters, the guide documents mechanics against a codebase small enough to read. War stories are real ones from this log (OOB stale scores §34, DART's k+1 trap §35, the two-libomp deadlock §36, the split-vs-gain disagreement above). Stale narrative docs were refreshed in the same pass (context, report addendum, architecture 2–8, new 7-parallel.md), and milestones are now git-tagged (the MVP submission tag, v0.2.0–v0.5.0).

38. Completing the non-categorical gap: rows 10-17 in one push¶

Every remaining non-categorical row of the feature-gap record landed (tables and datasets per row in that doc):

Leaf renewal (10): GrowResult::leaf_ids + an objective renew_leaf hook; the booster regroups rows by leaf and replaces Newton steps with loss-optimal values (residual median / alpha-quantile / clamped-mean huber). Closed the recorded ~10% MAE gap outright: bonsai now ties lightgbm on mae/quantile and leads on huber.
Prediction extras (13) + warm start (14): predict_at / staged / pred_leaf / dump; --init-model continuation that rebuilds training scores by bin-space routing and reuses the loaded mappers.
Classification benchmark (11): streamed HIGGS subset, AUC in both the C++ metric registry (rank-sum) and compare.py. The logloss path's first live outing landed between xgboost and lightgbm.
TreeSHAP (15): Algorithm 2 over per-node covers (stamped at grow time, format v6); verified against a brute-force Shapley reference to 1e-9 plus the efficiency property at every level.
Multiclass (16): BoosterFor became a trait so {softmax, G, Sa} routes to a dedicated MulticlassBooster, the one objective whose K-output shape the 1-D Objective concept can't express. Covertype: bonsai depthwise leads the field on accuracy.
Sparse input (17): LIBSVM reader behind data.format, densified, with the boundary stated plainly: input parity yes, sparse compute no. a9a AUC within 0.2% of xgboost.

Recurring lesson, third occurrence (after §30, §34): the benchmark is the strongest test. lightgbm's multiclass metric rejection, catboost's regressor/classifier split, and a9a's short test-split feature space were all caught by running the harness, not by unit tests.

39. Categorical stage 1: measure before building¶

Decision. Before implementing native categorical splits (gap row 12), add a genuinely categorical dataset and measure what native handling buys. Amazon employee access (OpenML 4135): 9 integer-ID features, RESOURCE at 7.5k distinct values: the regime one-hot cannot reach and arbitrary ID orderings hurt most.

Design. The measurement isolates technique from library: lightgbm runs twice, once with the IDs as plain numerics and once with categorical_feature declared. That within-library delta (+0.0144 AUC, and faster fits) is what Fisher set splits are worth, independent of engine differences. bonsai contributes its two practical options today (raw IDs: 0.8476; K-fold target encoding: 0.8462) and the other references their native modes (xgboost 0.8498, catboost 0.8812).

Findings. (a) The stage-2 prize is real but modest: ~+0.007 AUC from bonsai's best to lightgbm-native. (b) catboost's +0.026 lead over lightgbm-native does not come from target encoding per se (plain K-fold target encoding was no better than raw IDs) but from the ordered scheme plus feature combinations. (c) bonsai beats lightgbm when both are denied categorical machinery, so the gap is the feature, not the engine. Stage 2 (set splits) proceeds with a measured ceiling instead of a hope.

40. GPU-resident growing: widen the builder seam to an optional level backend¶

Decision. Phase 3 (device-resident partitioning and split finding, design in architecture/11-gpu-resident.md) extends the builder policy with three optional hooks (find_splits_many, partition_many, finalize_rows) detected via if constexpr + requires, the same idiom as phase 2's populate_many. The host grow loop remains the single algorithm narrative and the decision-maker (leaf-vs-split, smaller-child pairing, constraint propagation); the device executes the data plane. SplitInput gains sums and row_count so it degrades to node metadata when histograms and rows stay device-resident.

Rejected. A device splitter as a fourth typelist dimension: there is one device implementation and it is coupled to the CUDA builder's state, so a registry axis buys combinations nobody can instantiate (the same restraint as decision 32 for threading). A row_to_node map (xgboost's shape): decision 17's reasons hold on the device too: per-node segments pair naturally with the subtraction trick and stable per-node row order. CUB/Thrust for the partition scan: a hand-rolled three-kernel scan keeps the backend a single self-contained TU with no new dependency.

Consequences. In resident mode SplitInput.hists/rows are empty; the inspect path for tests and debugging is an explicit download_histograms. The cuda_depthwise determinism contract is formally tolerance-equal (prediction/RMSE tolerance + split-agreement rate in parity tests, never tree equality); CPU-only builds stay bit-identical. Copy-back mode is retained as the degrade path (deep trees, oversized max_bin) and for the oblivious/leafwise growers.

41. Grower data-plane as the `LevelStep` strategy; retire the copy-back ladder¶

Decision. Reframe decision 40's "escalating optional hooks" as a single compile-time Strategy: a LevelStep<SplitterT, Engine> (primary template = host data plane; partial specialization for GPULevelEngine = device data plane) selected by engine type, so grow() reads as one control-plane narrative with the host/device fork localized to one specialization instead of smeared across six if constexpr/if (resident()) sites. resident() is removed: the per-tree mode is captured once from begin_root's bool into the LevelStep. The concept ladder collapses to two, renamed to shed the GoF-Builder connotation (the type is the pluggable compute substrate, and the CUDA one supplies the whole device data plane): HistogramEngine (host, from HistogramBuilder) and GPULevelEngine (the device-resident vocabulary, from ResidentHistogramBuilder). The populate → populate_many → resident progression was a dev-research ladder (phases 1–3, preserved in git and doc 11's measured stages); only device-resident is kept: populate_many, BatchHistogramBuilder, and the GPU copy-back histogram path are retired, and the rare decline (oversized max_bin, or level buffers that won't fit) falls back to CPU histogram building via the engine's existing cpu member. Design in architecture/12-grower-backend.md. Refines, not overturns, decision 40: host-owns-decisions, one-grow-loop, no-new-typelist-dimension, and the all-or-nothing device coupling all carry forward.

Rejected. Runtime strategy objects (a unique_ptr<LevelPlane> or std::variant handed to the loop): the dynamic-dispatch shape decisions 14/26/32 explicitly chose against: it would need a benchmark to prove zero-cost rather than guarantee it, add type erasure the grow loop avoids, and thread into every grower. Keeping the three-tier concept ladder: populate_many is a host-plane batch optimization, not a device peer, so the middle tier documented a distinction that no longer earns its name. Keeping GPU copy-back as the decline fallback: it keeps a whole kernel path alive for a rare case (default max_bin=255 always goes resident); the CPU fallback trades a slice of that rare path's throughput for a materially smaller CUDA backend.

Consequences. update_nodes (182 lines, 18 params) decomposes into plan_level + LevelStep methods + commit_children, each ≤~40 lines; the CPU path becomes branch-free (a reader never meets a GPU concept). Oblivious and Leafwise route through the shared LevelStep, so their CUDA variants become a one-line alias + registry entry (not registered here: a later pass; leafwise's sequential gain-heap limits its device value). Docs 10/11's grower-side seam narrative is superseded by doc 12; their device-kernel content stands. Zero perf cost is a release gate, measured on a Thunder 4×A100 against xgboost (scripts/bench_gpu.py, MSD, before/after) in addition to CPU bit-identity and cuda tolerance-equal parity. Landed (commits b4d223c → a4764ba): every phase held 392/392 both configs, sha256-identical CPU models for all three growers, and unchanged resident-path launch counts; the leafwise open call resolved against singleton-frontier unification (a per-heap-pop LevelPlan buys ceremony, not shared code: split_node lives in level_step.hpp as the single-node data plane) and the oblivious unsampled routing stayed its own (leaf-table indices, not DenseTree node ids).

42. GPU oblivious via a device level-find; no `cuda_leafwise`¶

Decision. Register cuda_oblivious (ObliviousGrower<CudaHistogramEngine>), completing the pass decision 41 deferred. The resident plane is reused wholesale (an oblivious level is a depthwise level where every node takes the same split), so the only new device piece is the level-find: find_level_split sums each candidate cut's child scores across the whole frontier (32-node chunks into per-feature global scratch, any frontier width) and requires per-node min_child_hess feasibility, mirroring update_best_for_feature_for_level; a fused find → reduce → child-sums launch chain pays one sync per level. GPULevelEngine gains the method; LevelStep routes LevelSplitFinder growers to it. Measured (RTX 5090, fair full-pipeline timing): 3.9–4.1 s vs CatBoost-GPU 7.3–8.2 s on MSD, RMSE matching CPU oblivious exactly.

Rejected: registering cuda_leafwise. A working LeafwiseGrower<CudaHistogramEngine> was built and benchmarked, then withdrawn: best-first growth expands one node at a time, which the level-batched resident plane cannot serve (advance swaps whole levels), so every histogram was computed by the engine's CPU member: a cuda_ registry name executing CPU work misleads bonsai info, serializes into model metadata, and mislabels benchmarks. The comparison it existed for holds without it: CPU leafwise (11.1–11.5 s) beats LightGBM's CUDA leaf-wise backend (12.0–12.9 s) on the 5090. A true device leafwise needs a non-swapping advance: phase-4.

Consequences. trains_here keys on the cuda name prefix (registry convention) instead of one hardcoded name; skip_without_cuda keys on the grower's Engine alias; the oblivious grow loop consumes per-node child sums from the level-find (children's SplitInput.sums are otherwise unknowable: device histograms are not host-scannable), and host/device leaf finalize is symmetric via LevelStep::finalize_leaves.

43. CI: GitHub Actions gates every PR; sanitizers as a build option, not a preset¶

Decision. One workflow (.github/workflows/ci.yml, ubuntu-24.04, clang-21 from apt.llvm.org so CI uses the Makefile's exact Linux toolchain): build-test (Release + ctest), sanitize (BONSAI_SANITIZE=ON, ASan+UBSan, RelWithDebInfo), format (clang-format --dry-run --Werror), tidy (make lint off a configure-only tree: clang-tidy needs just compile_commands.json), python (nanobind module + bindings tests), and cuda-compile. Sanitizers are a CMake option (BONSAI_SANITIZE) applied globally so FetchContent'd Catch2 carries the same instrumentation; a preset file would be a second configuration surface for a Makefile-driven repo. This discharges the proposal's sanitizer-in-CI commitment.

CUDA is compile-only. GitHub has no GPU runners, but the kernel TU is the code path a CPU-only CI would otherwise never touch. The job installs only cuda-nvcc-12-6 + cuda-cudart-dev-12-6 (headers, ptxas, libdevice, no driver) and builds with -DBONSAI_CUDA_ARCH=sm_80, since native requires a device to probe. Runtime GPU validation stays where it has always been: make test-cuda on a GPU host.

Consequences. LSan is off in the sanitize job (detect_leaks=0): libomp's pool allocations outlive main and drown real leaks; ASan+UBSan remain fatal (-fno-sanitize-recover=all). BONSAI_SANITIZE + BONSAI_CUDA is a configure error: sanitizing device code is not supported and a half-instrumented binary would imply coverage that isn't there. FetchContent _deps and the toy CSVs are cached per lockfile-ish keys; a cold run needs the network, warm runs don't.

The first honest lint run. make lint had been discarding run-clang-tidy's stderr, so tool failures on macOS printed "no findings"; the Linux CI job surfaced 128 findings the local gate never saw. Response: fix the real bugs (a use-after-move in the for_each_type fold, two unchecked optional accesses, an exception escaping main, plus small google-*/naming/enum-size items) and curate .clang-tidy down to checks the codebase actually satisfies: the disabled block documents each family (pointer arithmetic at C/CUDA/codec boundaries, reference members, callback F&& APIs, concept archetypes, established long functions). The Makefile now fails lint when the tool itself fails. Re-enabling a curated-out check is a deliberate cleanup PR, not a silent flip.

44. `n_threads = 0` means capped auto, not all hardware threads¶

Decision. Auto worker count is min(omp_get_max_threads(), 16). Explicit n_threads = N passes through untouched, including N above the cap or above the core count. Measured on a 60-vCPU host (issue #2, MSD 463k×90, 200 iters, depth 8): the uncapped default ran CPU depthwise 10× slower than 16 threads (167 s vs 15.9 s) and cuda_depthwise 4× slower (51 s vs 12.2 s): per-level parallel sections are short (≤ ~250 nodes), so with many workers OpenMP barrier spin-wait dominates useful work, burning CPU-hours per fit. 16 was near-optimal there and is at or above the core count of every dev machine this project targets. The cap does not change the determinism contract: no cross-thread reductions exist at any thread count (decision 32).

Rejected. Work-scaled formulas (clamp(n_rows/K, 1, hw)): the pathology is barrier latency as a function of frontier width, not row count; a formula would be false precision. Setting OMP_WAIT_POLICY programmatically: setenv only works before runtime init, and inside the Python module another extension may have initialized OpenMP first. kmp_set_blocktime: libomp-specific with no header guarantee. The honest version of both is documentation: OMP_WAIT_POLICY=passive is the operator knob for oversubscribed many-core hosts (7-parallel.md).

45. Ingest profiled, then one optimization kept: sort-based quantile cuts¶

Decision. BONSAI_INGEST_PROFILE=1 breaks the CSV-to-Dataset pipeline into read / index / parse / mapper-fit / bin / buffer (the GrowProfiler pattern; accumulates across train + valid, prints at exit). Profiling MSD (463k×90) on an M-series laptop attributed half of ingest to mapper-fit: create_cuts ran ~253 nth_element calls per feature over a shrinking 200k suffix. Replaced with one std::sort + stride reads: every prior nth_element(lo, begin+k, end) call selected the absolute k-th order statistic (its prefix already held the smaller partitions), so the sorted array read is value-identical, verified by model-hash equality on MSD. mapper-fit 0.53 s → 0.25 s; total ingest 1.06 s → 0.76 s. End-to-end on the RTX 5090 benchmark (fair full-pipeline timing): cuda_depthwise 2.98 s → 2.61 s, cuda_oblivious 3.91 s → 3.50 s (medians of 3), RMSE unchanged.

Measured and rejected. memchr field splitting: parse 0.26 s → 0.25 s; MSD fields are ~7 chars, too short for SIMD scanning to beat the char loop; reverted rather than kept as complexity without payoff. mmap read and a batched transform were not attempted: read (0.11 s), bin (0.09 s), and buffer (0.03 s) are each under the 100 ms bar the plan set for touching a stage. from_chars already dominates parse and libc++ 21's implementation is Eisel-Lemire-class; no fast_float dependency.

Deferred: binned-dataset cache. A sidecar binary (source size+mtime+bin-config hash as the invalidation key, raw column dumps for the uint16 matrix) would cut warm reloads to ~0.1 s, like LightGBM's binary datasets. It is a product feature, not a benchmark lever: cross-library comparisons stay cold-parse for fairness (decision on bench_gpu.py timing), so it waits for its own PR.

46. Scaling suite: in-memory sweeps, synthetic Friedman target, frontier-as-data¶

Decision. The scaling study (scripts/bench_scaling.py) measures fit/predict complexity in rows, cols, bins, and threads against xgboost/lightgbm/catboost from in-memory numpy float32 through each library's Python API (bonsai via the CUDA-enabled nanobind module), not CSV+CLI like bench_gpu.py. Corner CSVs would be 15–20 GB of text whose parse time is orthogonal to the question; the CLI's old justification (module was CPU-only) died with the previous PR. Fairness holds because fit is timed from raw arrays and includes each library's own ingestion (ColumnBatch+binning / QuantileDMatrix / lgb.Dataset / Pool); predict is timed from a raw test matrix; quality is scale-free R² train/test. Synthetic target is a generalized Friedman-1 (blocks of 5 informative features with decaying weights over ~20 of the columns, uniform features so bins populate evenly, noise for best-achievable R² ≈ 0.9); make_regression was rejected as purely linear: trees would neither differentiate on bins nor separate libraries. Data is deterministic in (rows, cols, seed) only, so bins/threads sweeps reuse identical matrices.

Grid. Base 1M×100×255 (depth 8, 100 iters, lr 0.1, 16 threads; 3 repeats at base for variance) with per-axis sweeps: rows ×4 to 16M, cols to 65k with rows shrinking past 4k cols (cells ≤ 2^31: a joint 2D log-fit over rows+cols cells de-confounds the tail), bins to 65535 (bonsai's bin_id_t cap; other libraries swept to their own refusals, recorded not assumed), threads {1,4,16,64} at base. A full cross-product at the user-set extremes is 2^40 cells; per-axis is the honest affordable shape.

Frontier as data. Every (cell, variant) runs in a child process with a size-scaled timeout; status ∈ ok/oom/timeout/error/unsupported/skipped with a message, and RAM/VRAM estimators pre-skip hopeless corners per host. What each GPU tier can and cannot fit is a first-class result (the reason for running 5090 + A100-80GB + 4090). The child-per-run design also delivers per-run profile capture for free: bonsai's exit-time profilers flush to the child's stderr. lgbm_cuda is declared unsupported in v1 (pip wheel lacks CUDA; source build deferred); catboost-GPU's 254-bin cap is applied and recorded as bins_effective.

47. Ingest and histogram-allocation rounds: measure on the platform that has the disease¶

Decision. Rounds 1–2 of the optimization campaign (PRs #8/#9/#10): the Python module ingests via a parallel cache-blocked transpose, then (round 1b) bins directly from the row-major numpy matrix through new BinMappers::fit/Dataset::bin overloads: peak RSS fell from 4.8× to 1.8× of raw X with byte-identical models; and histogram cell blocks recycle through a size-class pool (HistBlockPool) with the oblivious level-finder's prefix scratch hoisted to per-worker storage. The pool's win is invisible on macOS (its allocator already recycles; paired Mac A/B was deliberately flat) and decisive on Linux, where fresh >mmap-threshold blocks page-fault per populate: G1 measured CPU depthwise bins 4095 at 3.1×, 16383 at 5.2×, and the 65535 cell went from timeout to completing. Lesson recorded: allocator-behavior optimizations must be validated on the deployment platform; the Mac can prove only non-regression.

48. The "5090 anomaly" was a defective host; benchmark pods now pass a sync-latency probe¶

Decision. The scaling study's ~11–14s per-cuda-fit overhead on the 5090 host class was fully diagnosed in G1: one specific rentable machine has a GPU sync round-trip of ~300µs (healthy: 4µs measured on a second 5090), invariant to schedule flags (spin/yield/blocking), with perfect PCIe Gen5 x16 and 23.7GB/s pinned bandwidth, consistent with host-level ASPM/IRQ misconfiguration, unfixable from a container. bonsai's ~20–30k synchronizing ops per fit × 300µs reproduces the excess exactly. The allocator hypothesis was refuted by A/B on that host (async vs sync alloc identical); the stream-ordered allocator, pinned batched bins upload, and dynamic shared-memory opt-in (PR #11) stand on their own merits: the smem opt-in moved the 4095-bins cell from CPU fallback to GPU for a 17.4× win and exposed issue #12 (cuda_oblivious fallback trains garbage above the cliff, pre-existing). Benchmark protocol change: every rented pod must pass a 30-second sync-latency probe (>50µs → reject the pod), and the round-1 fleet's 5090 rows are annotated as defective-host data.

49. Row-wise histogram fill over a row-major u8 mirror; determinism relaxes to fixed thread count¶

Status 2026-08-26: Dataset::row_major_bins was renamed; the mirror is reached as Dataset::mirror() (include/bonsai/dataset.hpp). The canonical determinism statement moved to invariants.md and learn/determinism-as-a-contract.md.

Decision. The scaling study's flat 2.5–4× CPU fit gap vs LightGBM was populate-bound (50–85% of fit) and, on deep small nodes, gather-bound: the feature-parallel fill reads one binned column per feature with bins[rows[k]] scatter, so a sparse node's every access misses cache, measured at ~0.7G adds/s against ~3.3G adds/s for large dense nodes (M2, adds counted by the profiler). CpuHistogramEngine::populate_many now fills u8 (max_bin ≤ 255) data row-wise over a lazily built row-major mirror (Dataset::row_major_bins, +n_rows×n_features bytes, never built by CUDA or predict-only paths): a level's nodes become row-block work units; each reads its rows' bins as contiguous 1×n_features strips regardless of node sparsity, streams grad/hess once total instead of once per feature, and accumulates into private per-block partial histograms merged in fixed block order. Nodes below a work threshold (fill ≥ ~16× the partial zero+merge cost) run as one block writing the node's cells directly (byte-identical to the old order), and the whole level shares one parallel section, so 128 deep nodes are 128 units instead of 128 serial parallel-section spawns. Base cell (1M×100×255, M2, 8 threads): fit 26.7→16.9s, populate 20.1→9.6s; 4M rows 43.0→25.3s; 1M×512 35.9→21.8s; identical R². LightGBM's same-cell fit on the M2 is 18.0s: first cell where bonsai CPU leads it.

The contract spend (user-approved). Multi-block nodes' sums depend on the block count, a pure function of node size, selection width, total selected bins, and the configured thread count, so models are bit-identical at a fixed n_threads (decision 7's original contract) but no longer across thread counts. Single-block nodes, the u16 feature-parallel fallback, split scans, and predictions still match serial order exactly. docs/architecture/7-parallel.md §"The determinism contract" is the canonical statement.

Measured and rejected. Two-accumulator parity splitting of the feature-parallel fill (microbenchmark promised 1.6×): exactly flat in the real loop at 1 and 8 threads: the microbench's arrays were cache-resident, isolating FP-chain latency the real streaming loop hides; kept the microbench-lies lesson, dropped the code. Block-count oversubscription beyond 4× threads: flat on M2. The row-major mirror as eager dual storage in Dataset::bin: rejected, CUDA runs would pay RSS for a mirror they never read.

50. Float histogram cells; double reductions; the empty-split demotion guard¶

Decision. HistCell is {float, float} (was double). Per-cell sums are bounded by node size and gradients/hessians arrive as float, so cell storage carries only per-cell rounding; every reduction that crosses cells (Histogram::totals(), fill_prefix, the node finder's running left sums (split_sums_at now takes doubles)) accumulates in double and converts once at the store. Measured (paired, M2 8 threads): base cell fit 15.3→13.2s, 8M×100 53.3→45.7s (populate 1.23–1.25×), R² identical to six decimals at both scales; California-Housing eval baseline moved 5.4e-6 (re-pinned). Histogram pool, partial slabs, and prefix scratch all halve. The CUDA engine had already proven the shape (float per ≤32k-row chunk, double merge, RMSE parity): this brings the CPU fill to the same discipline. User-approved trade: models change (one-time re-pin), quality doesn't.

The bug it flushed out. Sibling subtraction in f32 leaves noise ~O(cell_sum × eps) in the derived child's cells, far above the gain > 0 gate, so a degenerate cut (every row one side) can score a tiny positive gain. In f64 the same mechanism existed at 1e-16 and never fired; in f32 it produced empty-child splits whose cover-0 nodes made SHAP contributions NaN (caught by test 60). Fix is structural, not precision-tuned: after partitioning, demote_empty_splits (depthwise plan) and the leafwise equivalent convert any split with an empty child back to a leaf: the partition's row counts are ground truth at any precision. Pre-allocated child nodes remain as unreachable placeholders; predict and SHAP walk from the root.

Rejected. A hist_precision runtime knob (f32 partials only, f64 default): measurement showed no quality cost, so the config surface buys nothing, and a compile-time macro would fork the ABI of a public-header type across builds and double the CI/wheel matrix. Keeping f64 cells with f32 only in partial slabs: leaves the direct-fill nodes, subtraction, and finder scans at 2× traffic for no accuracy benefit that the double-reduction discipline doesn't already provide.

51. quantile_step ceiling stride: cuts never exceed the budget¶

Decision. quantile_step uses the ceiling stride ceil(n/(budget+1)) (was floor(n/budget), floored at 1), guaranteeing at most cut_budget cuts for any subsample size. The floored stride overshot whenever the subsample wasn't comfortably above the budget (400 distinct values at the default max_bin=255 produced 401 bins, 1000 produced ~335), giving small datasets more granularity than requested and silently disqualifying them from u8 storage and the row-wise fill (issue #17, found while writing decision-49 tests). Every model's cut positions shift (this is the second model-changing round after decision 50, landed back-to-back deliberately): synthetic 1M×100 R² identical at 0.8991; the California Housing eval baseline improves 0.7175214 → 0.7157657 (−0.24%), consistent with budget-respecting quantiles being no worse while small datasets regain the fast path.

Rejected. Keeping the floor and clamping cut count post-hoc (drop every k-th surplus cut): non-uniform quantile spacing for no benefit over the ceiling stride. Treating it as a documentation caveat: the silent u8 disqualification interacts with two performance features that key off max_bin<=255, which is too much surprise for a one-word fix.

52. Device residency: REFUTED by experiment; the lever is per-level staging latency¶

Decision (revised twice, then measured). The phase-A experiment (PR #28, per the review directive on the design PR: prove it before redesigning APIs) refutes device-resident gradients: skipping every per-tree grad/hess upload saved 1.6s of a 42.5s fit (3.7%) at 16M×100 on an L40S, with upload_s unchanged at 7.5s. Both of this decision's prior premises misattributed that upload line: first to the one-time bins upload (~70ms), then to bulk gradient bytes (~1.6s at pinned rates). The line is actually hundreds of small per-level Staged<> syncs (node sums, bounds, row offsets: latency-bound pageable copies × ~800 tree-levels per fit), which residency does not touch. The next honest lever for the 42.5 vs 27.9s gap to xgboost-GPU: batch/pin the per-level staging and cut per-level host↔device round-trips. The CPU/GPU API-consistency redesign the experiment gated should target that: a level-transaction interface would make the batching natural.

What the experiment paid for itself with. A live mainline landmine: demote_empty_splits (decision 50) orphans device-plane row stamps when it fires after the device has scattered a demoted split's rows (one mega-leaf per tree at rightmost-spine node ids 2^k−2, feeding a score runaway), dormant under double-precision device sums, extracted as its own fix. Plus the diagnostic pattern that found it: per-tree device-state dumps diffed against the model's own predictions.

Rejected. Continuing to phase B (device binning) or the API redesign on the unmeasured premise; debugging the experiment's residual 16M-only quality gap (r² 0.79 vs 0.89) once the timing verdict had already killed the approach.

53. The level-transaction engine narrative (adopted)¶

Status 2026-08-26: the narrative doc this entry ratified was dissolved by decision 114; its still-true residue, the plane composition, lives as comments in src/cuda/detail/device_context.cuh, and the concept sketch it drafted shipped under different names (see include/bonsai/grower.hpp).

Status. Executed. Step 1 (transaction vocabulary on both planes, byte-identical) landed in PR #33; steps 2–3 landed as one change set: the identity row list cached on device and restored D2D per tree instead of re-uploaded (the avoidable share of the root line), and end_tree handing the node value table to the engine, which maps rows to leaf values on device and returns values/leaf ids in two bulk copies (replacing the finalize line's per-tree host stamping loop over every row). The Impl decomposition shipped in the same set. One deliberate deviation from the sketch below: gh_ordered lives in LevelPipeline, not GradientPlane; it is the level-row-ordered gather and ping-pongs with gh_b.

Decision (proposed). The CPU/GPU engine APIs unify around a level-transaction narrative (begin_tree / open_level(LevelInputs) → LevelOutputs / apply_level / end_tree() → TreeEpilogue) with the backend an implementation detail, per the review commission on the device-residency design. The shape is derived from the measured 16M ledger (PR #31), not from a lever bet: the root becomes an ordinary one-node level (dissolving begin_root's bespoke per-tree staging, the ledger's largest line at ~14s), open_level's single-struct input batches the per-level staging by construction, and end_tree owning the per-row epilogue is where finalize residency (8.4s of per-tree D2H + host stamping) can land with its real motivation. CudaHistogramEngine::Impl (41 buffers) decomposes into DeviceData / GradientPlane / LevelPipeline planes with ledger-exposed lifetimes. Migration in three bit-identical-gated PRs: host plane first, device plane onto the transactions, then the device epilogue. Full design: docs/architecture/14-engine-narrative.md.

Rejected. Redesigning around the refuted residency/staging levers; a single templated engine (hides the narrative); fusing the level phases into one transaction (the control plane must observe results between them: doc 12's host-control contract stands).

54. Device binning at ingest (adopted)¶

Status. Implemented. cuda_ingest (both arms), the IngestPlane receipt on Dataset with lazy host materialization, plane adoption in ensure_dataset, pipeline wiring by grower-name prefix, and the doc-16 instrumentation (bins_upload, fin_wait/fin_d2h, dbin, fit-profile in the bench harness) shipped as one change set. One correction from implementation: the module path bins from the borrowed row-major numpy view (features_view), CSV from the feature-major ColumnBatch; the doc-15 draft had the arms swapped.

Decision (proposed). Ingest joins the transaction narrative as the zeroth verb (ingest(raw, mappers) → IngestPlane) with the backend an implementation detail (doc 16 frames the narrative as a compute DAG; this is the last node outside the vocabulary). The host backend's ingest is today's fill_binned, untouched. The CUDA backend's ingest streams raw columns through double-buffered pinned staging and a lower_bound-exact bin kernel; its product is an opaque IngestPlane handle carried by Dataset as the transaction's receipt (defined in the CUDA TU, null in stub builds, the row_major_ lazy-mirror precedent). The train pipelines select the ingest backend the way growers dispatch (cuda name prefix + cuda_available(), training dataset only); ensure_dataset adopts its own plane instead of copying and uploading host bins. Host binned columns are not materialized in device mode: the fallback-decline arm (oversized max_bin, decidable at ingest) keeps the host path outright, and route_unsampled's bin_at triggers a one-time cached D2H materialization only when row sampling is on. Cuts stay host-fitted (identical transform semantics ⇒ bit-identical bins ⇒ before/after models must be equal, the gate). Motivation is the measured line, not a bet: host bin ~4.6s of 1.6G binary searches + an unlapped 1.6GB upload, replaced by ~0.5s of overlapped transfer+kernel at 16M×100 (same-pod L40S; priced by the measured 19GB/s gh edge, scripts/dag_model.py). Ships with the missing lap counters (bins_upload, finalize fin_wait/fin_d2h, ingest dbin): the PR #35 refutation was designed against an undecomposed finalize line, the fourth such miss. Full design: docs/architecture/15-device-binning.md.

Rejected. Engine-side rebinning (the host transform is the cost); retaining raw floats on Dataset (+6.4GB RSS at 16M); device mapper-fit this round (reservoir-RNG identity: model-changing, deferred to its own decision); a config knob for device binning (it is an implementation detail of the cuda growers, not a user choice); the first draft's DeviceBins side channel on Dataset (identical mechanics, but the API grows by exception instead of by the narrative's vocabulary).

55. The cut-quality gap is not sample size (study)¶

Study (2026-07-12). The residual test-r² gap to xgboost-GPU at the 16M×100 cell (bonsai 0.8791 vs xgb 0.8800) was hypothesized to come from fitting cuts on a 200k-per-feature subsample. Sweeping bin_mapper.n_samples over {200k, 1M, 4M, 16M (full column)} on one pod, same cell, same seed: r²_test moves 0.879083 → 0.879048 → 0.879194 → 0.879261 (+0.0002 end to end, ~2% of the gap), while fit cost is flat (the mapper's reservoir scan, not the sort, dominates and even that is noise at this scale). Verdict: sample size is a dead lever; the 200k default stands. Follow-up (same day, issue #42) dissolved the gap entirely: bonsai trained on xgboost's exact cut set (extracted via DMatrix.get_quantile_cut()) scores the same as on its own cuts (0.879034 vs 0.879083), and xgboost's own r² moves by 0.001 (the full size of the "gap") between max_bin 255 and 256 (0.879953 vs 0.878966, same pod, same seed). There is no sketch deficit; at this cell every library sits in a ±0.001 band governed by threshold-placement chance, and no research round is warranted.

Rejected. Raising the default n_samples (cost without benefit); treating the gap as a defect to chase this round (it is a documented trade at +0.0009 r², with bonsai holding a 3× host-memory and ~3× predict-speed advantage at that cell).

56. Quality-campaign fixes: the oblivious veto, exact duplicate cuts, the true softmax hessian (adopted)¶

Decision. Three model-changing fixes from the 2026-07 quality campaign (benchmarks/quality-campaign-2026-07.md), each probe-confirmed on real datasets before implementation: (1) the oblivious level scan no longer vetoes a candidate when one frontier node's children fall under min_child_hess: the infeasible node contributes zero gain and the broadcast split still applies (empty children are first-class since #57); this was worth 3–26% rmse on real data and moves bonsai-oblivious ahead of catboost. (2) create_cuts emits one right-inclusive cut per distinct value when the subsample's distinct count fits the budget (the stride+dedup previously collapsed house_sales' 13-value bedrooms column to 7 cuts); measured net-neutral on the campaign but objectively correct: the California pin moves 0.7157605 → 0.71625 (+0.07%). (3) the multiclass hessian is the true diagonal p(1−p); the factor-2 variant halved every Newton step and cost exactly 2× the iterations to match lightgbm (letter 0.9515 → 0.9613 at the same budget). Post-fix, bonsai-depthwise is the best-scoring library on 8 of 9 campaign datasets.

Rejected. Keeping the veto with a lower default min_child_hess (the veto is wrong at any threshold: one node should never censor the level); weighting duplicate cuts by count above the budget (deferred to #63, where continuous placement is the live question); retaining the factor-2 hessian for "xgboost compatibility" (xgboost loses to both at matched budgets precisely because of it).

57. Count-weighted cuts for heavy-value columns only (adopted)¶

Decision. The #63 follow-up to decision 56's cut work: above the distinct-value budget, a column where some value's count reaches a mean-sized bin gets lightgbm-shaped greedy allocation: heavy values take a bin to themselves, the rest fill toward a running mean, cuts at midpoints between adjacent distinct values (greedy_weighted_cuts in src/bin_mapper.cpp). Columns with no heavy value keep the decision-51 quantile stride bit-identically: with every count below a mean bin, equal frequency already is the count-weighted allocation, so the greedy walk could only reshuffle thresholds inside the chance band. Measured on the campaign suite: house_sales 131,841 → 128,959 rmse (27% of the gap to xgboost), every other standing unchanged, bonsai-depthwise still best on 8 of 9; the California pin moves 0.71625 → 0.71719 (+0.13%, chance-band). Both cut placements were probed and are metric-equivalent (midpoint vs value cuts produce identical training bins when the sample covers the column); midpoint kept as the principled choice for unseen test values.

Rejected. Greedy allocation for every over-budget column (measured: house_sales gains more, 126,528, but magic_telescope drops below lightgbm and phoneme's depthwise drops below the best ref: a chance-band tax on every continuous dataset to overfit one; the per-column rule takes the heavy-value win without the tax); closing #63 (bonsai at 511 bins matches xgboost at 256 on house_sales: the refs extract more from an equal budget on dense-continuous columns, so per-column budget allocation stays an open research question, kept in tension with decision 55's synthetic verdict).

58. Categoricals resolved by measurement: an encoder, not an engine feature (adopted)¶

Decision. Categorical support ships as OrderedTargetEncoder in the Python package (causal/ordered target statistics, seeded and deterministic, keep_codes giving trees both the response-rate and identity views) plus guide chapter 13: the C++ core stays numeric. The call was bought with a probe, not taste (scripts/probe_categorical.py, evidence in benchmarks/categorical-tradeoff-2026-07.md): each reference library's categorical machinery toggled on/off at matched knobs on amazon/adult/kick measured native Fisher set splits (the doc-17 stage-2a design, which would have grown the ~1,400-line split/tree/SHAP/model-format core by roughly a third) at +0.029 / +0.000 / −0.018 AUC by lightgbm's own toggle: a coin flip whose complexity every user carries. Meanwhile ~100 lines of preprocessing hit 0.8590 on amazon, beating lightgbm-native (0.8572) and closing 48% of the gap to catboost-native (0.8894, whose ordinal baseline collapses to 0.779, so much of its celebrated categorical gain is recovering its own weak numeric path). On the repo's own amazon split the encoder is worth +0.049 AUC (0.811 → 0.860), pinned by test_encoding.py. Doc 17's design stays on the shelf, priced and declined; the probe method is codified as the feature-admission skill.

Follow-up (same day). Crossed pairs close the rest of the gap, still from preprocessing: cross=2 on the encoder packs each pair of code columns into an int64 key and applies the same ordered TS: amazon 0.8604 → 0.8877 vs catboost-native 0.8897 (chance-band at this test size), with catboost's own toggle as the control: max_ctr_complexity=1 (crosses off) drops it to 0.8587, below our singles line: the crosses were its entire remaining edge, and our single-column ordered TS already beats theirs. Triples measured 0.8859 (overfit, rejected as a default). Probe: scripts/probe_crossed_ts.py; pinned in test_encoding.py.

Rejected. Stage-2a native set splits (measured median gain ≈ 0, negative on kick, cost concentrated in the most-read files in the repo); plain or K-fold target encoding (stage 1 measured the leak: 0.8462 vs 0.8590 ordered: causality is load-bearing, guide 13 derives why); xgboost-style native handling (its own toggle loses on 2 of 3 datasets); engine-side stage 2b for now (per-tree permutations + crossed-category statistics are catboost's remaining amazon edge, +0.030, and crossed-TS preprocessing is the next cheap probe before any engine work is reconsidered).

59. Cross-architecture bit determinism: no fp contraction on the host plane (adopted)¶

Decision. -ffp-contract=off for all host C++ (CMake directory option; the CUDA kernel TU opts back to fast, the device plane's f32-chunk/f64-merge scheme owns its precision story and no cross-platform hash contract covers it). Found when decision 57's midpoint cuts split the California pin by platform (0.71719 arm64 vs 0.71725 x86-64, each internally deterministic): a temporary CI test dumping every cut bit-exactly proved the bin mappers identical across platforms, which cornered the divergence in training arithmetic: clang contracts a*b+c into single-rounded fma on targets that have the instruction (arm64) and cannot on baseline x86-64, and the new thresholds put one split decision inside that one-ulp window. Flipping -ffp-contract=off on the Mac reproduced the Linux value exactly, confirming the mechanism. Measured cost: nil (interleaved 5-rep California 2000-iter fits: 1.736 vs 1.738 s; the hot paths are adds and divides, not fused-multiply-add shapes). This upgrades the determinism contract from thread-count invariance (decision 32) to the same model bits on any host architecture: no reference library makes that claim. The eval pin and the model_hash.py baseline are now platform-independent by construction (baselines refresh with this change; the campaign quality tables were measured pre-flag on arm64 and move at rounding level only).

Rejected. Widening the pin margin to a platform band (surrenders the pin's one-ulp sensitivity and the reproducibility story); per-platform pin values (documents the symptom, keeps the disease); -ffp-contract=off on the CUDA TU too (fma is the native device op; nothing gates device bits across platforms, and the device/host boundary is already tolerance-checked where it must be).

60. Issue #72 resolved: the "cross-arch divergence" was OpenMP build variance (adopted)¶

Decision. Parallel training is bit-identical across host architectures, proven, not just claimed: with matched builds, arm64 and x86-64 produce byte-equal models at every thread count tried (t1 f35340da495343c1, t2 9e2c081bd5e3fcfe, t4 fb5692c2aebea4fe, t8 afc31f746baaafb7, sampled default 8c2c375a331a1bb7; per-block root-histogram cell bits diffed to zero on a 2-thread minimal artifact). Issue #72's "architecture divergence" was a phantom with a real cause: find_package(OpenMP) fails on a stock Mac (homebrew libomp is keg-only, the llvm kegs ship no OpenMP), the failure was a silent STATUS fallback to a serial build, and serial builds train different (valid, but not build-reproducible) bits than parallel ones because the fill plan's block counts scale with parallel::n_threads() (the documented fixed-N contract). Worse, the Makefile's python configure sent its output to /dev/null, so which build you got was invisible. Three fixes: (1) CMake hints homebrew's keg-only libomp explicitly on macOS; (2) OpenMP-not-found is now a hard configure error: a build variant that changes model bits must never be silent; serial stays available as an explicit -DBONSAI_OPENMP=OFF; (3) the python configure's OpenMP/error lines are no longer swallowed. The cross-arch CI gate now asserts both serial and parallel hash equality across the runner pair. Model bits are henceforth a pure function of (input, config, configured thread count), on any machine.

Rejected. Keeping the silent serial fallback (it manufactured issue #72 and cost a night of ulp-level forensics (a fill-plan bit dump, thread/iteration ladders, and an on-pod x86 bisection) to exonerate arithmetic that was never guilty); making the fill plan thread-count-independent (a fixed global block count either starves big hosts or taxes small ones; the fixed-N contract is the documented, measured trade); chasing the third hash family observed on one destroyed local build and one opaque runner configure (the loudness fix eliminates the class, silent variants can no longer exist).

61. The populate round: software prefetch closes the 16M CPU gap (adopted)¶

Decision. One change, priced from the ledger before it was written: the fill row loop prefetches the mirror strip and grad/hess pair sixteen rows ahead (run_fill, src/grower.cpp). Below the root a node's rows are an ascending subset, so successive row-major strips sit at irregular strides the hardware prefetcher cannot follow; the 16M×100 ledger showed the loop DRAM-latency-bound: populate 82.1s of a 107.4s fit (row loop 78.3s), everything else ≤ 12s. After: row loop 45.5s (−42%), fit 75.8s, a dead tie with xgboost-hist's 75.7s, same pod, same session (was 1.42× behind). Reads only, models byte-identical (hash gate f35340…/8c2c37… unchanged, r² to four decimals), 4M improves 30.2→24.3s. This closes the largest visible CPU loss on any chart.

Rejected. The same prefetch in the partition passes (measured: 75.85s vs 75.76: noise; refutation logged and the change reverted, per the doc-16 rule that unmeasured complexity does not ship); constant-hessian cell elision (the hess add lands on the already-loaded cache line: priced as ~free before any code); prefetch-distance tuning beyond the first guess (the first guess hit the ledger's floor; further squeezing trades maintenance for noise-level gains).

62. The GPU find-kernel round, abandoned by instrumentation; the 16M GPU frontier belongs to catboost (adopted)¶

Decision. No GPU kernel-optimization round ships, and the docs stop implying a large-scale GPU speed crown over catboost. Both conclusions are measurements, not opinions.

The round was scoped to speed up the device find kernel (all-double warp scan, suspected FP64-bound on an L40S). Instrumenting first (a profile-gated sync splitting the find lap into kernel-compute vs device→host transfer (find_kern_s/find_d2h_s in histogram_engine.cu, profile path only)) showed the find kernel costs 0.17s at 16M. The "8.4s find" in the grow profile is the profiler's opening cudaDeviceSynchronize in find_splits_many catching the previous level's asynchronous histogram kernels; the find scan itself is negligible. The genuine ~8s GPU cost is the histogram accumulation, which already sums in float shared memory (double only for the bounded cross-chunk merge), so the obvious precision lever is spent. Every banked hypothesis was refuted by one measurement: the instrument-first pass is the deliverable, turning a speculative multi-hour rewrite into a measured no-go.

That reframed the question from kernel speed to the whole accuracy-vs-time frontier at 16M (benchmarks/gpu-pareto-16M-2026-07.md, all one pod). The honest result: bonsai strictly dominates xgboost-GPU (reaches xgboost's 100-iteration accuracy 0.8776 in 30.16s vs 36.69s, and beats it at every matched-accuracy point), but catboost owns the frontier at every accuracy above ~0.875: catboost@150 (28.35s, 0.8892) is faster and more accurate than bonsai cuda_depthwise@100 (30.16s, 0.8776). Decomposed against the structural match (bonsai's symmetric cuda_oblivious), catboost's lead is two independent gaps: a per-round speed gap (~15–20%, its tuned symmetric-tree kernel) and a per-round convergence gap (+0.011 test r² at 100 iters, same tree shape, its ordered boosting corrects the prediction-shift bias in ordinary gradients). The earlier "bonsai is more accurate than catboost" reading was a fixed-100-iteration artifact that the frontier dissolves. Closing this needs both a histogram-kernel rewrite and an ordered-boosting-style change to the core booster; winning either alone leaves the other gap standing.

Consequence. bonsai's crown over catboost rests on the other axes: bit-identical cross-architecture determinism (§59–60, no competitor has it), a third of the host memory, an ~1,800-line engine, chance-band categorical parity via preprocessing (§58), and best-of-field CPU quality on 9 of 10 real datasets, not on large-scale GPU throughput, where it is strictly ahead of xgboost and roughly 20% behind catboost.

Rejected. A histogram-kernel memory/atomic-contention rewrite under the promo deadline (real project, not a knob); quoting the fixed-100-iteration cell as evidence bonsai out-accuracies catboost (the frontier refutes it); any framing of "strictly superior to all three libraries on performance" (unsupported: catboost wins this cell on speed).

Corrected by §63. The "convergence gap" attributed to catboost's ordered boosting above was wrong. The feature-admission ladder refuted ordered boosting (a wash vs boosting_type=Plain, which catboost uses at scale anyway) and bin quality, then isolated the +0.011 to a bonsai bug: the GPU oblivious level-find still vetoed level candidates on infeasible nodes, a defect the CPU had fixed (issue #60). Patched, GPU cuda_oblivious matches its CPU twin and catboost's accuracy exactly. Only the ~19% per-round kernel-speed gap is real: a bounded optimization target, not an algorithmic disadvantage. Study: benchmarks/catboost-scale-edge-2026-07.md.

63. GPU oblivious lost accuracy at scale to a missing issue-#60 port, not to catboost's algorithm (adopted)¶

Decision. Port the CPU level-find's issue-#60 behavior to the device level_find_kernel: an infeasible frontier node (child hess < min_child_hess) contributes its parent score (zero gain) instead of vetoing the whole level split. The kernel previously computed warp_all(feasible) across the frontier and dropped any candidate with one infeasible node: the pre-issue-#60 pathology (decision 56) that at depth ≥ 5, where some node is always near-empty, rejected every good deep cut. The veto, its device scratch (level_feas), and the now-unused warp_all helper are removed.

Why it was found. cuda_oblivious scored 0.8638 test r² at 16M vs its own CPU grower's 0.8749 (and catboost's 0.8751), while cuda_depthwise matched CPU (0.8776 vs 0.8782). Precision was ruled out (device histogram cells are double, more precise than the CPU's float); ordered boosting and bin-sample quality were ruled out by the feature-admission ladder. That localized the loss to the oblivious device level-find, and the code diff to the CPU was the veto. The bug hid because small-data GPU-vs-CPU oblivious tests never produce a near-empty node; the new test forces one with a high min_child_hess at depth 7 (fails pre-fix, GPU −0.019 vs CPU 0.324; passes post-fix).

Consequence. Same pod, 16M: cuda_oblivious 0.8638 → 0.8749, matching CPU to the fourth decimal and catboost's 0.8751: the accuracy gap that read as a convergence disadvantage (§62) was ours all along, and every cuda_oblivious user at depth ≥ 5 was silently getting worse models at scale. The honest residual against catboost is now a single ~19% per-round kernel-speed gap (0.238 vs 0.292 s/iter), which is efficiency, not algorithm. Full [cuda] suite green (125,864 assertions); no CPU model bits touched, so the cross-arch hash gate is unaffected (the device plane is tolerance-equal, not bit-equal, by design).

Rejected. Closing the residual kernel-speed gap in the same change (a separate, bounded optimization: measure the oblivious kernel's occupancy and launch overhead first, per decision 62's instrument-first rule); leaving the veto with a comment (it silently degraded every deep oblivious GPU fit: a correctness bug, not a tuning choice).

64. One shared row sample for binning, not one reservoir pass per feature (adopted)¶

Decision. BinMappers::fit draws a single seeded row sample for the whole matrix and gathers each feature's values at those rows, instead of each feature independently reservoir-sampling its own column. The old create_subsample ran std::ranges::sample over a NaN-filtering view (an O(n) pass) once per feature; at 16M×100 that was 100 passes over 16M rows, and the BONSAI_INGEST_PROFILE lap put mapper-fit at 8.45s on the Mac / ~5.7s on an L40S, pure serial host time before any training (and time catboost, which bins on device, does not pay). The shared sample runs the O(n) selection once, then each feature does an O(n_samples) gather: mapper-fit 8.45s → 0.35s at 16M (24×), r² unchanged to four decimals (0.8782 both): the bin-sample study (§ the scale-edge note) already showed sample count is quality-neutral past ~200k rows, so a different-but-equivalent sample moves nothing.

Determinism and scope. Bit-identical for any dataset that fits the sample: n_rows ≤ n_samples takes the whole-column path unchanged, so every small-data test and the serial model hash (model_hash.py runs 500k rows with n_samples=500000) are untouched. Only fits above n_samples get new cuts: the default (sampled, 8-thread) hash changes, superseding decision 60's 8c2c375a331a1bb7; the serial f35340da495343c1 stands. std::ranges::sample over an iota_view uses selection sampling with mt19937 (deterministic and architecture-independent) so the cross-arch gate (which compares arm==x86 dynamically, no hard-coded baseline) still holds, and the fixed-thread-count contract (decisions 59–60) is preserved. Two tests pin it: a >n_samples fit is reproducible cut-for-cut, and the ColumnBatch/features_view overloads agree.

Consequence. Measured same-pod at 16M (L40S, 100 iters): cuda_oblivious 19.65s vs catboost 18.85s at matched accuracy (0.8749 vs 0.8751), a ~4% residual, down from the ~19% §63 flagged, with mapper-fit falling from ~5.7s to 0.77s of the fit. Combined with §63's accuracy fix, bonsai's GPU oblivious went from slower and less accurate to even on accuracy, within 4% on speed. Every large CPU fit gets the same ~8s ingest saving. It is the standard lightgbm/xgboost approach (sample rows once); bonsai had simply not adopted it.

Rejected. Preserving the exact per-feature reservoir bits (would keep the 100× O(n) cost for no quality gain, since the sample is quality-neutral); a bit-preserving fast path only for NaN-free columns (fragile: depends on std::ranges::sample's implementation-defined selection matching a hand-rolled reservoir, and still O(n) per feature to detect NaN).

65. Reusable pre-binned Dataset with a sealed bin config (adopted)¶

Decision. PR #91 adds a Python-level bonsai.Dataset(X, y, weight=None, max_bin=255, n_samples=200000, seed=0, min_data_in_bin=1) that runs BinMappers::fit + Dataset::bin once at construction, then feeds the same bonsai::Dataset to every train(params, dataset) call. A hyperparameter sweep or CV loop skips the per-fit bin pass entirely. Verified bit-identical to fitting from (X, y) directly.

Consequence. Because the object address is now stable across fits, the CUDA resident-matrix upload-skip cache (decision 54's ensure_dataset) actually fires for the first time: previously every train() built a fresh Dataset, so the cache never hit. Lifetime is simple: the wrapper pins the numpy X buffer, and FeatureBuffer borrows it row-major with no float materialization.

Sealing the bin config. bin_mapper.* overrides via params are rejected by key-prefix at the Python boundary. The harder case is a config file: value comparison can't distinguish an absent section from one explicitly restating defaults, so a file setting max_bin = 255 against a Dataset built with max_bin=63 must still error rather than silently discard the Dataset's binning. Two follow-up commits on PR #91 closed this: f899e52 added a structural check, config::toml_has_section(), that tests for section presence rather than value equality, and a further pass made the guard general and exposed min_data_in_bin.

Rejected / deferred. Disk persistence, row-subsetting for CV, and fit(dataset) on the sklearn estimators are deliberate non-goals of this MVP. The eval_set/early-stopping path in train(dataset) was a known gap in the MVP, tracked and closed separately (see branch feat/dataset-eval-set).

66. Prebuilt wheels from native runners, not manylinux containers (adopted)¶

Decision. PR #94 + #98, shipped in v1.2.0: 15 wheels + sdist attach to the GitHub Release automatically on release: published. Matrix is {ubuntu-22.04, ubuntu-22.04-arm, macos-14} × py{3.9–3.13}, CPU-only.

Why not cibuildwheel/manylinux. bonsai requires LLVM ≥ 20 with libc++ (C++23 std::print/std::mdspan). The official LLVM release binaries need a newer glibc than the manylinux_2_28 container ships, so wheels build on plain runners with apt.llvm.org LLVM 21 (Linux) / brew llvm@21 (macOS) instead. auditwheel/delocate vendor libc++ (+ libc++abi/libunwind) into each wheel; OpenMP is statically linked via the pre-existing BONSAI_OPENMP_STATIC, so zero C++ changes were needed to make this work. auditwheel tagged the result manylinux_2_34 (dual-tagged 2_35): actual glibc floor 2.34, so RHEL 9/Alma 9 are covered, a better floor than the Ubuntu 22.04+ design target implied.

Python floor. Lowered to 3.9 in PR #98 (concrete user use case, reversing an earlier decline). Four touchpoints, not one: requires-python, the CMake find_package(Python) floor (a pyproject-only audit missed this; the local 3.9 build caught it), the wheels matrix (now 15 legs), and ruff target-version=py39 so 3.10+-only syntax can't regress in.

Verification. Every wheel installs into a clean venv and runs a smoke script: regressor fit, classifier predict_proba, Dataset train, and (added after PR #95 found from_file broke on Python 3.10 because tomllib is 3.11+ stdlib and the original smoke test never called it) a save/from_file round-trip across the whole version matrix.

Consequence. PyPI publish is deliberately deferred until trusted publishing is registered (bonsai-gbt name + pypi.org trusted publisher, an owner action); wheels ship via GitHub Releases in the meantime. CUDA wheels are out of scope here and tracked as issue #99: the design is a fat-arch linux x86_64 leg plus a rented-GPU validation gate before assets attach, since GitHub-hosted runners can build CUDA but never execute it.

Rejected. A manylinux_2_28 container image to reach older distros (a documented follow-up, not blocking; no user has asked); PyPI publish in this PR (needs the owner-side trusted-publisher registration first).

67. Automatic per-feature bin budgets: declined by measurement; explicit user edges: open design question (partial)¶

Decision. No engine API for per-feature bin budgets or user-supplied edges (issue #63's residual; lightgbm ships max_bin_by_feature, catboost ships per_float_feature_quantization). Priced at zero core cost by scripts/probe_binning.py on five datasets at campaign knobs: budgets emulated exactly through the issue-#61 one-cut-per-distinct rule for bonsai, and priced natively via lightgbm's own toggle. Best observed gain anywhere is +0.0011 (synthetic with known signal structure, unlimited extra budget), at the decision-55 chance band; the importance-guided policy loses outright on adult (−0.0011 AUC) and kick (−0.0096), and lightgbm's own toggle is ≤ +0.001 at best and negative on kick. The inverse-allocation control costs up to −0.084 r², so budgets are causally live but asymmetric: uniform 255 sits at saturation and reallocation can only break things. The headroom policy on MSD (−0.0003) also kills the hypothesis that decision 55's +0.001 cut-quality residual is a resolution-allocation problem. Full table in benchmarks/binning-tradeoff-2026-07.md.

Consequence. What is declined is the ACCURACY-motivated feature: automatic importance-driven budget allocation buys nothing at the 255-bin default and misallocates on real data. What stays open is the CAPABILITY: user-supplied explicit bin edges are a deployment-artifact question the probe never measured. The pre-discretization emulation is bit-exact at train time but leaves the binning OUTSIDE the model artifact: every serving path must re-apply the transform forever, and predict on raw features silently produces garbage. A native Dataset(bin_edges=...) would bake edges into the per-feature BinMappers, which already serialize into the model, so predict round-trips on raw values; the design is specified in architecture doc 18 and admission-gated on a concrete workload. The default also gained evidence: uniform max_bin = 255 is robust in a way per-feature schemes are not.

Rejected. An engine API "because lightgbm and catboost have one" (their own measurements above justify declining it); shipping ManualBinner as a package class (unlike the categorical encoder, the preprocessing won nothing, so it stays a documented recipe, not an API surface).

Reopen the accuracy branch if: a memory-constrained regime forces the global budget down (max_bin ≤ 64, where saturation no longer holds and allocation could matter; never measured). Admit the capability branch when a concrete workload needs domain-mandated bins in the model artifact (regulatory bands, clinical thresholds, reproducing an existing scheme); doc 18 prices it.

68. The Grinsztajn benchmark becomes the standings suite; bonsai has the best mean rank (adopted)¶

Decision. Adopt the 55-task Grinsztajn et al. (2022) benchmark (OpenML suites 297/298/299/304) as the external standings suite, replacing the hand-picked internal ten as the citable table (scripts/run_tabular_suite.py, benchmarks/grinsztajn-2026-07.md; the internal campaign stays as the fast smoke tier). At the paper's medium protocol and campaign-matched knobs across 990 fits: bonsai has the best library mean rank, 2.04 vs xgboost 2.11, lightgbm 2.53, catboost 3.33, with the most consistent profile in the field: second place or better on 44 of 55 datasets and last exactly once. xgboost keeps the most outright wins (26 vs bonsai's 10): it is the peak library at small-data knobs, bonsai the consistency library. bonsai leads categorical regression outright (mean rank 1.77).

Why external. A self-picked suite invites a selection-bias objection that no honesty of execution can answer; a published benchmark selected by third parties removes it. The claim format also improves: distributional (mean rank, head-to-head, rank distribution) instead of a win count over ten.

Caveats recorded with the numbers. The 10k-row cap is decision 55's regime (xgboost's cut-quality edge at its strongest); ordinal codes strip catboost's categorical machinery (uniform convention, undersells catboost on categorical tracks); depthwise and leafwise coincide at these knobs so library-level best-variant ranking is used.

Rejected. OpenML-CC18 (image-flattened datasets dilute the tabular story); TabZilla-176 (volume over curation); running references on GPU for wall-clock (GPU paths change reference numerics and reproducibility; accuracy standings are hardware-independent). TabArena submission (the living leaderboard with external protocol) is the post-promo follow-up, not rejected.

Correction (same day). The first run hardcoded xgboost's min_child_weight = 1 instead of the campaign mapping (= min_data_in_leaf = 20, scripts/reference_params.py), giving xgboost ~20x smaller leaves than the other libraries were allowed; under that skew xgboost ranked 2.11 with 26 outright wins. Re-run at the campaign mapping: bonsai mean rank 1.73 with 27 outright wins, lightgbm 2.35, xgboost 2.73, catboost 3.20; bonsai >= xgboost on 42 of 55. Because min_child_weight is hessian-weighted (20 implies ~80+ rows per leaf on classification, harsher than the others' 20 rows), the two conventions bracket xgboost and the claim kept is the one that holds at both ends: bonsai has the best mean rank under either. The remaining named losses to xgboost, year (+0.0066 r²) and yprop_4_1 (+0.0053), are the decision-55 residual with real datasets attached. Sensitivity rows preserved in results/grinsztajn-2026-07-xgb-mcw1.jsonl.

69. The benchmark charter: bonsai.bench in the wheel, two divisions, one row schema (adopted)¶

Decision. The benchmark harness moves inside the Python package (bonsai.bench: params, metrics, synth, runlog, datasets, and the grinsztajn/scaling suite runners), shipped in the wheel behind a [bench] extra so pip install bonsai-gbt[bench] reproduces the published tables; import bonsai stays numpy-only via lazy imports. Every result row is one of two closed divisions: quality (metric primary, timing never citable) or perf (timing_mode mandatory: in_memory vs pipeline), self-described by row schema v1 (division, suite, knobs + hash, metric/value, timing_mode, git sha, full host capture). Knob sets (CAMPAIGN, SCALING) and the two declared lightgbm leaf conventions live only in bonsai.bench.params: hand-re-deriving reference mappings caused decision 68's published correction and is now structurally prevented and test-pinned. The normative rules are docs/method/benchmark-protocol.md; migration verified by a byte-identical model-hash gate, byte-stable Friedman goldens, and a zero-mismatch replay of committed grinsztajn rows.

Rulings. Completed probes are provenance and stay as-run (annotated, never refactored onto the new library); bench_categorical's deviating knobs stay annotated rather than rerun; benchmark data stays in tests/data/ (moving it would touch configs, Makefile sentinels, and CI caches for zero mechanical gain) with the registry as machine truth and the test-pin fetchers kept build-independent for CI; superseded results files are deleted, git history being the archive (rebaseline.jsonl/.md removed, superseded by the dated re-baseline).

Rejected. A repo-local scripts/benchlib (users could not reproduce tables from an install); moving data out of tests/data; regenerating any committed row under the new schema (old and new rows coexist; readers tolerate both).

70. CUDA wheels: fold-in, static cudart, and a rented-GPU release gate (adopted)¶

Decision. The linux x86_64 release wheel ships the CUDA backend (issue #99): SASS for sm_70;sm_75;sm_80;sm_86;sm_89;sm_90;sm_120 plus a compute_90 PTX forward-JIT floor (sm_100 JITs; BONSAI_CUDA_ARCH became a list, BONSAI_CUDA_PTX_ARCH pins the PTX), built with CUDA 12.8 (12.6's ptxas cannot emit sm_120; clang cannot target 13) and CUDA::cudart_static (BONSAI_CUDA_STATIC_RUNTIME), so the extension carries zero CUDA DT_NEEDEDs, auditwheel grades manylinux_2_34, and GPU-less machines import a de-facto CPU wheel. Measured on the leg-0 probe before commitment: the whole backend costs 2.33MB of wheel (control 1.27MB; xgboost's GPU wheel ~300MB) and ~5s of build for six extra arches; no CUDA runtime symbols exported. Because GitHub runners can build CUDA but never execute it, releases gate through one rented L40S session that boots the candidate runtime image (docker/runtime.Dockerfile, wheel baked in, published as ghcr.io/daniel-m-campos/bonsai:{<tag>-cuda,cuda} after the gate): on-pod wheel_smoke_cuda.py (both cuda growers, pre-registered 1e-3/0.005 parity bands, byte-stable save/load round-trip) plus model_hash.py, whose output must equal CI's, extending the decisions 59/60 byte-identity contract to shipped artifacts on real GPU hardware; the wheels workflow's hash-compare job asserts the same triple equality (macos-arm, linux-arm, linux-x86-with-CUDA) on every wheel build. Teardown is unconditional with a loud leftover-pod sweep; a red gate withholds only the CUDA artifacts and the CPU release stands. Driver floor documented as R525 (CUDA 12 minor-version compatibility); the gate exercises 12.8-static-cudart on RunPod's 12.4-driver hosts every release by construction.

Rejected. Vendoring dynamic libcudart via auditwheel (a second mangled runtime in-process beside torch/xgboost); an nvidia-cuda-runtime-cu12 pip dependency (mandatory NVIDIA download for CPU-only users); a separate bonsai-gbt-cuda distribution and +cu12 local tags (pre-registered fallback if the wheel had exceeded 50MB, it measured 2.33; local tags are PyPI-illegal); building wheels inside the bonsai-ci image (its 12.4 pin is the RunPod container-start driver ceiling, its ptxas cannot emit sm_120, and decision 66 already rejected container wheel builds); toolkit 12.6 without sm_120 SASS (registered as the fallback if 12.8-on-R550 minor-version compatibility fails at the gate; it did not).

71. The post-fix 16M GPU frontier: split by budget, ceiling recovered (adopted)¶

Decision. The 16M accuracy-vs-time frontier was re-measured on current main (2026-07-14, same-pod L40S, scripts/gpu_pareto.py with ladders extended into the deep end), superseding the pre-fix 2026-07-12 run whose bonsai points carried both the decision-63 oblivious accuracy defect and pre-populate-round fit times. The post-fix frontier decomposes cleanly into fixed cost plus marginal cost per round: bonsai cuda_oblivious ~4.6s + 155ms/round, catboost ~11.8s + 77ms/round, xgboost ~22.3s + 58ms/round. Consequences, recorded as the standing verdict: bonsai owns the fast end (lowest fixed cost; fastest to every accuracy up to roughly r² 0.88), the curves cross at the ~100-round operating point (20.1s/0.8749 vs 19.5s/0.8751, the scaling tables' measured tie), catboost owns time-to-accuracy in the deep end (0.8973 in 35.1s vs bonsai's 0.8974 in 51.9s) through its 2x cheaper marginal round, and the accuracy ceiling now belongs to bonsai by a rounding digit (0.8981 vs 0.8980 at 450 iters), a region the pre-fix defect made unreachable. Validation of the supersession itself: the depthwise ladder reproduces the old run's r² to four decimals (no fix touched it; determinism across pods), the oblivious ladder shows the defect's removal point by point (0.8638 to 0.8749 at 100 iters), and bonsai times dropped ~28% while reference libraries moved only with fleet variance. Evidence: benchmarks/gpu-pareto-16M-2026-07.md + jsonl (replaced in place, git history is the archive per decision 69); the results-ledger chart now shows this run. The named next perf target if the deep end becomes load-bearing: bonsai's 155ms marginal round vs catboost's 77ms.

Rejected. Keeping the pre-fix frontier beside the new one (decision 69: superseded results are deleted, not attic'd); extending the README's fixed-iteration "fastest GPU slot" claim to the whole frontier (false above r² 0.885 and the claims table stays exact); chasing the marginal-round gap now (the 100-round operating point is the standing benchmark regime and is a tie; the deep end has no user yet, so the gate applies).

72. The marginal-round campaign: 155 to 104 ms, the frontier taken whole (adopted)¶

Decision. Decision 71's named target (bonsai's 155ms oblivious marginal round vs catboost's 77) was attacked instrument-first per doc 16 and closed in two stages (PR #148) plus this close-out. Stage 0 built the price list before any lever: a profile-only sync peel replayed the decision-62 misattribution (6.1s filed under lfind_stage was the previous level's async histogram kernels draining at the next sync), cudaEvent pairs priced the async build directly, and conservation flushed two residues nothing else explained (make_root's 64MB/tree host identity copy at 33ms/round; a final-level histogram build for children that are leaves at 22ms/round). Stage 1 landed the three levers the table priced above ~10ms: the identity contract (full-data fits pass empty rows + row_count; the engine iota-builds and caches identity on device), deterministic device root sums (fixed-grid two-pass reduce replacing a 16M-row host loop), and the final-level skip (advance_layout_only() keeps only the segment flip stamping needs). Three levers were killed by their pre-registered criteria at a combined price under a millisecond (epilogue sync scope 0.1ms, per-level memset 0.5ms, pinned gh staging break-even). Same-pod: round 181→125ms, fit 19.43→13.88s, r² four-decimal identical, CPU model hash byte-identical.

Frontier verdict (2026-07-15 re-run, same-pod L40S US-NC-1, superseding in place per decision 69). bonsai cuda_oblivious ~3.4s + 104ms/round, cuda_depthwise ~3.1s + 121ms (the levers were engine-level, so depthwise fell from 187ms too), catboost ~12.4s + 76ms, xgboost ~21.6s + 65ms. The bonsai-catboost crossover moved from ~100 rounds to ~320, which sits inside both libraries' accuracy plateaus: bonsai is now first to every measured accuracy up to r² ~0.895 (0.8749 in 13.9s vs 19.7s; 0.8948 in 24.5s vs 27.9s), the 300-iter points are a statistical tie (0.8974/35.3s vs 0.8973/35.1s), and the ceiling stays bonsai's (0.8981 vs 0.8980). Accuracy reproduced to four decimals at every shared point, the campaign's behavior-preservation contract measured from the outside. The ship bar (≤110ms) was met; the crown bar (≤77ms, kernel parity) was not, and did not need to be: its premise was that the crossover mattered inside the useful range, and the fixed-cost advantage plus the 33% marginal cut pushed the crossover past it.

Stage 2 (histogram-kernel engineering): not spent. The plan's own gate asked for honest histogram compute above ~80ms/round with parity otherwise unreachable; the event-timed build is ~72ms and parity turned out unnecessary for the frontier. The floor is recorded instead: the 104ms round is ~72ms histogram kernel + ~32ms partition, bus, and per-level residue. Reopen if a workload lives at 450+ rounds of this cell, where catboost still reaches its (lower) plateau about 3s sooner.

Consequence. scripts/dag_model.py and doc 16 were refreshed from this pod's profiled cells: the old find 7.62s node is retired (the find kernel is 0.13s; the weight was always the histogram build, now its own event-timed node at 7.9s/fit), conservation closes at fit level (est 14.9s vs measured 15.70s), and the all-device floor is ~13.9s with the histogram build at 70% of device compute. Guide 11 gained the campaign's rows and the one-round summary of the method. The results ledger regenerates from the superseded-in-place artifacts.

Rejected. Stage 2 now (gate not met, crown achieved without it); keeping the pre-campaign frontier beside the new one (decision 69); quoting the 104ms as a cross-pod absolute (fleet spread is ~25%; the number that transfers is the decomposition shape and the same-pod deltas).

73. Explicit bin edges ship as a Dataset capability (adopted)¶

Decision. Doc 18's design is built: bonsai.Dataset(X, y, bin_edges={col: edges}) bins listed columns at user-supplied interior cut points; unlisted columns fit as usual. The plumbing is exactly the constructor-not-mechanism the doc predicted (BinMapper::from_edges beside the loader-trusted from_cuts; a BinEdges override on both BinMappers::fit overloads with edge mappers seeded before the parallel region so validation errors never cross it; one bin_edges argument on the Python Dataset), no model-format bump, no hot-path branches, no CUDA change. The admission gate is recorded as met by roadmap signal: decision 67 asked for a workload that needs domain-mandated bins inside the artifact, and the owner ranked the capability onto the roadmap; accuracy remains a non-claim (decision 67 measured saturation).

The one thing the design missed, found by the acceptance test. The first implementation appended only the +inf sentinel, and the acceptance test's cross-band assertion failed: bands above and below the LAST edge were inseparable. Root cause is an engine convention, not a bug: the split scan (histogram.hpp cut_cells) never offers the last real bin as a candidate because for fitted columns that cut is degenerate (the observed maximum defines it). For user edges the band above the last edge is a domain statement, so from_edges appends a FLT_MAX cut to close it as a real bin plus the +inf sentinel to keep the missing bin NaN-only: k edges give k+1 splittable bands, NaN routes to its own bin, and raw +inf inputs land in the missing bin exactly as they do for fitted columns. Edges must be finite and below FLT_MAX (reserved), strictly increasing, non-empty; bad column indices and duplicates throw ConfigError.

Verification. The acceptance test is the one the decision-67 emulation structurally cannot pass: fit with domain bands, predict on RAW values (within-band invariance, cross-band separation, right-inclusive edge membership), save/load round-trip predicting byte-identically, all with no external transform. Byte-identity gates: model_hash.py unchanged (serial f35340da495343c1, sampled 8c2c375a331a1bb7), full suite 38,296 assertions green, overridden-absent fits bit-identical by construction (the override path is never entered).

Rejected. Validating inside from_cuts (the loader's trusted path stays branch-free; user input is validated at its own named entry); a config-file/TOML surface for edges (arrays of floats per column do not fit the flat dotted-key override grammar; the Python Dataset is the artifact-construction door and the CLI can gain one later if a workload asks); exposing the emulation as a package class (decision 67's ruling stands, superseded by the native capability).

74. The FLT_MAX top-band closer: the missing bin is NaN-only on every path (adopted)¶

Completed by decision 117: +inf still binned as missing after this change, so the NaN-only claim held for finite values only until then.

Decision. create_cuts now appends a FLT_MAX cut before the +inf sentinel on every fitting path (issue #155), so every finite value above the last placed cut bins into a real, splittable top band instead of the NaN sentinel. This closes three leaks the fitted paths carried: the stride path's top tail (~one mean-bin of rows), the greedy path's final group (an entire heavy run when the maximum is a capped value), and rows above the 200k-sample maximum on any path. It also removes a train/predict routing skew: leaked rows trained down the learned default_left branch but predicted by raw threshold comparison (right of everything), so the same row routed differently in training and deployment; the valid/early-stopping path scores raw and was already on the deployment side. The budget slot was already reserved (from_sample's max_bin - 2), so finite cut placement is untouched and no bin count exceeds max_bin. The invariant now matches decision 73's edge columns: no finite value ever bins as missing, anywhere, and the mechanism (the closer cut) is the same one from_edges uses.

Evidence (scripts/probe_missing_bin.py, campaign knobs, both growers). The mechanism synthetics are decisive: capped-max heavy value r² 0.483→0.988 (the capped 10% carried signal and trained as missing), rare top-tail signal +0.16, signal above the sampled max at 1M rows +0.008. The real suite is chance-band flat: higgs ±0.0001, airline ±0.0006, california ±0.001 with opposite signs per grower, and amazon's −0.002 sits inside that dataset's own 0.0034 spread across max_bin 253-256 with no closer at all (the null-perturbation yardstick; threshold-placement churn on ID codes). Full table in benchmarks/missing-bin-closer-2026-07.md.

Why adopt (contrast with decision 67). The probe suite happens to contain no capped columns; the wild does (clipped sensors, capped amounts, 999-coded maxima), and there the failure mode is works-vs-doesn't, not a tuning delta. The fix costs nothing at runtime, spends a slot that was already reserved, removes a correctness asymmetry rather than chasing accuracy, and unifies the sentinel semantics across fitted and user-edge columns.

Standings re-validation (the surprise). The citable Grinsztajn table was re-run bonsai-arms-only against frozen reference rows (same protocol, seeds, and knobs; runner identity proven by the decision-69 replay). Expectation was chance-band; the result was not: mean rank 1.73 to 1.44, outright wins 27 to 36 of 55, second-or-better 44 to 50, last-place finishes 1 to 0, head-to-head >= lightgbm 37 to 46, >= xgboost 42 to 48. The honest decomposition: per-dataset values mostly moved inside 0.001 (43 of 55; 8 improved beyond it, 4 regressed, worst house_16H −0.0040), but the 10k-row cap is the regime where the greedy path is nearly universal (mean bin ~39 rows, so 461 of 495 fits changed), the suite was full of photo-finish second places, and the closer's small nudges broke them systematically one way. Rank tables amplify near-ties; the durable movements are the head-to-head counts and never-last. The decision-55 residual also narrowed: year recovered +0.0036 of its +0.0066 gap to xgboost. The grinsztajn jsonl's bonsai rows are superseded in place; README and the ledger carry the new table.

Consequence. Models change wherever a leak existed: canonical hashes move to serial 09dbf47353033362 / sampled ca7174cb1560221e (data digests unchanged; the cross-arch CI gate is dynamic equality and unaffected), and the California pin improves 0.71725→0.7153 (−0.27%, inside the band; lineage in the test comment). New invariant test pins all three paths: observed max and out-of-sample larger values bin below the sentinel, NaN bins into it. An all-NaN column now emits {FLT_MAX, inf} (a real bin exists even with nothing observed). A tree may now split real values away from missing at the last cut, a candidate that previously did not exist; small-fixture tests were unaffected because the new candidate is infeasible or zero-gain there.

Rejected. Keeping the leak as a documented budget trade (the trade was never priced until now, and the price on capped columns is catastrophic); a config knob for the closer (invariants are not options); closing the leak by clamping transform to the last real bin instead (mimics lightgbm's convention but leaves the top band unsplittable and the skew intact for default_left trees).

75. Per-node/per-level feature subsampling: declined by measurement (adopted)¶

Decision. No colsample_bylevel / colsample_bynode equivalent in the grower (issue #45; xgboost ships both, lightgbm ships feature_fraction_bynode). bonsai already samples features once per tree (tree.feature_fraction), and the feature only earns its plumbing (the grower's selected-features path plus a CUDA find-staging change that today assumes a per-tree selection) if per-node/per-level beats per-tree. Priced at zero core cost by scripts/probe_feature_subsample.py on four datasets (8 to 200 features) at campaign knobs, toggling the reference libraries' own knobs. In isolation, per-node/per-level never leads: on california/higgs/year (<= 90 features) every arm sits in the decision-55 chance band or per-tree wins, and on the 200-feature synthetic where feature bagging matters most, per-tree at 0.5 (−4.73 rmse) dominates the best per-node arm (−1.18) by ~4x. Stacked on per-tree (their intended use) per-node adds a genuine −0.6 rmse on the 200-feature case, its one real signal, but a tuned per-tree fraction ALONE beats the stack: bytree0.4 (89.34) is better than bytree0.5+bynode0.5 (90.26). Full tables in benchmarks/feature-subsample-tradeoff-2026-07.md.

Consequence. The regularization per-node offers is reachable, and exceeded, by tuning the single knob bonsai already ships; the feature is a strictly dominated lever, so it changes no standings and unlocks no workload feature_fraction cannot already serve. The max_bin = 255-style outcome repeats: the existing simpler control is validated as sufficient. No core lines, no new config surface, no CUDA find-staging rework.

Rejected. Building it "because xgboost and lightgbm have it" (their own toggles above justify declining it, the categorical/bin-budget pattern of decisions 58/67). Reopener: a real many-feature workload where per-node/per-level beats a tuned feature_fraction beyond the chance band, none found across 8 to 200 features; bonsai's wide-data strength is the regime to watch, but even synthetic 200-feature data favours the existing knob.

76. Data-parallel multi-GPU: built, measured to parity, parked as an experiment (adopted)¶

Decision. The data-parallel multi-GPU engine (architecture doc 19: MultiCudaHistogramEngine, the cuda_multi_* growers, parallel.device_ids) is withdrawn from main and parked on the experiment/multi-gpu branch. The track was built through its full plan and validated end to end (correctness held everywhere: identical r2 across single, 2, and 4 GPUs at 16M and 64M rows, both peer and host-staged reduction regimes, on 2x A40 and 4x A100 NVLink hardware), then priced by five optimization levers across four same-pod ladders: parallel per-device fan-out, sliced per-shard finalize copies (which alone took the 64M fit from 184s to 63s), per-shard gradient upload slices, pinned staging, and a double-buffered reduction pipeline. The end state is END-TO-END PARITY with the single-GPU engine (16M: ~16.5s vs ~15.8s; 64M: 58.9s vs 57.5s), against a pre-registered bar of 1.3x at 2 GPUs and 1.8x at 4. The floor is architectural, not a tuning residue: gradients are host-computed per tree, so the gh stream must cross host memory every tree regardless of staging strategy (pinned equaled pageable), and the level reduction's cost is its correctness syncs, not transport (forced host-staged equaled peer on an NVLink mesh). What main keeps: parallel.device_id (single-device selection), the CudaDeviceContext extraction and header/implementation split (structural improvements to the single-GPU engine), and the tightened GPULevelEngine concept.

Consequence. The supported multi-GPU story is fit-parallelism: N GPUs run N independent fits (sweeps, CV folds, ensembles) via parallel.device_id, which scales linearly by construction and compounds bonsai's existing advantages (a fit's small memory footprint allows more concurrent fits per card; bonsai.Dataset bins once for the whole sweep). Capacity is likewise a weaker motivation for bonsai than for any competitor: u8-binned storage puts roughly 500M x 100 rows on one 80GB card, so vertical scaling covers the realistic range. The experiment cost ~$17 of pod time and every conclusion above is a measurement.

Rejected. Keeping the engine in main at parity (a checkbox feature that grows the core and the registry for no measured win is exactly what the feature-admission discipline exists to refuse; this track bypassed that gate as a personal-use experiment, and the outcome re-validates the gate); further lever rounds (the two remaining costs were each refuted by a targeted fix, which is the signature of an architectural floor). Reopener: a device-resident objective, where each GPU computes its shard's gradients from resident scores, eliminating the host gh stream. That redesign is also the single-GPU engine's own next frontier (the decision-72 marginal-round floor is the same host/device boundary), so it should be pursued for single-GPU wins first, with the parked branch inheriting the result.

77. The device-resident objective: the per-tree host round-trip deleted (adopted)¶

Decision. Decision 76's reopener was pursued single-GPU first, as its own campaign (issue #171), and shipped for the MSE objective. When a fit is eligible (an objective with a device gradient, no DART, no sample weights, a sampler that never reads gradient values), the booster arms a resident mode: labels and initial scores upload once per fit, each tree's packed (grad, hess) array is derived on the device from the resident scores by a two-line kernel, and the tree epilogue routes every row through the finished tree in bin space and fuses the leaf value into the resident scores. Per tree, nothing crosses the bus in either direction: the gradient upload, the interleave pass, the values and leaf-id downloads, the host objective loop, and the host score update all cease to exist. Ineligible fits take the host path untouched, BONSAI_HOST_OBJECTIVE=1 forces it, and the CPU plane is untouched by construction (model hashes byte-identical; the eligibility seam is a compile-time no-op for CPU growers).

Evidence (stage 0 priced, stage 1 measured; benchmarks/resident-objective-2026-07.md). The instrument-first price list on existing counters put the reachable pool at 35.9ms/round at 16M rows (22% of the fit) and 128ms/round at 64M (20%), clearing the pre-registered kill bar (12ms and 10%) threefold. The same-pod interleaved A/B on the shipped branch then measured cuts LARGER than the pool: oblivious 16M 136.1 to 102.6ms/round (24.6%), depthwise 16M 20.1%, oblivious 64M 16.4%, depthwise 64M 14.9%, because the routing epilogue is also cheaper than the stamp-and-copy epilogue it replaced. r2 identical to every reported digit in all pairs; the full-data resident model is bit-identical to the host-objective GPU model on Jetson parity tests. The ship bar (15ms/round at 16M) was met at 2.2x; the same-share projection puts the decision-72 frontier round near the 77ms stretch bar, to be confirmed by a frontier re-run.

Consequence. The decision-72 marginal-round floor moves: the round is now histogram build plus partition plus find, with the objective boundary gone. The complexity ledger, measured per the panel this campaign introduced: 616 non-test lines (about 380 in the CUDA plane, 240 in generic headers), zero new config knobs, zero registry growth, and the fit loop keeps its host shape (the eligibility seam is one named call, try_resident_round; the capacity predicate is one shared function both begin_root and resident_begin apply, so the decline conditions cannot drift apart; the resident state is armed per Dataset and disarms with a sync when the Dataset or a runtime gate changes). One depth-scaling note: the routing epilogue is O(depth) dependent loads per row against the O(1) gather it replaced, measured cheaper at depth 8; if deep-tree fits ever regress there, the reopener is a hybrid epilogue (leaf gather for partitioned rows, routing only for out-of-bag rows). The decision-76 multi-GPU reopener is now live on its parked branch (per-shard gradients from resident shard scores are exactly this mechanism), and remains parked pending a workload. Stage 2 (LogLoss, Poisson, resident sample weights) is justified by these numbers and follows as its own step; Softmax stays host-side (per-class tree shape, its own campaign).

Rejected. Extending eligibility to GOSS (it reads and reweights host gradients; a device GOSS is its own design); renewal objectives (MAE, Huber, Quantile renew leaves from host residuals; resident mode would have to download what it just avoided uploading); persisting resident state in the artifact (scores are training state, not model).

78. The frontier re-run confirms decision 77: unconditional at 16M (adopted)¶

Decision. The 16M accuracy-vs-time frontier was re-measured on current main (2026-07-18, one L40S pod, scripts/gpu_pareto.py, 22 same-pod points), superseding the decision-72 run in place. bonsai cuda_oblivious decomposes to ~3.8s fixed + 64ms/round (was ~3.4s + 104), cuda_depthwise to ~2.8s + 88 (was ~3.1s + 121), while the same-pod controls moved only with fleet variance (catboost 76 to 78, xgboost 65 to 59), attributing the whole marginal cut to the device-resident objective. The stretch bar decision 77 projected is met on the measurement that matters: bonsai's marginal round is now cheaper than catboost's on the same pod, the decision-72 crossover no longer exists at any measured horizon, and the one residue that run named honestly (catboost reaching its plateau sooner at 450+ rounds) is gone: 0.8979 in 31.9s vs 0.8980 in 46.4s, a fourth-decimal tie at 45% more wall clock.

Consequence. The frontier verdict is unconditional for the eligible regime (MSE, no DART, no sample weights, uniform or no row sampling); ineligible fits keep the decision-72 frontier, whose residue was already plateau-depth only. The ledger chart and benchmarks/gpu-pareto-16M-2026-07.md carry the superseded-in-place evidence. The remaining round is histogram build plus partition plus find; kernel engineering stays unspent per decision 72's gate, now with a higher bar since no competitor pressure remains at this cell.

Rejected. Claiming the fourth-decimal ceiling (fleet noise, either direction); extending the claim beyond the eligible regime before stage 2 lands LogLoss and Poisson.

79. Resident LogLoss, Poisson, and sample weights: the eligible regime widens to the common case (adopted)¶

Decision. Stage 2 of the resident-objective campaign (issue #171) extends the device-resident objective from MSE to LogLoss and Poisson and admits per-row sample weights, closing the campaign. The gradient kernel is templated on the objective kind and a weighted flag (compile-time dispatch, no per-row branches), formulas mirror the host implementations exactly in float with IEEE expf, and the Poisson raw-score clamp moved to one shared header constant so host and device cannot disagree. Weights upload once per fit beside the labels under the same dataset identity key. Eligibility now reads: MSE, LogLoss, or Poisson; weights or not; no DART; uniform or Bernoulli row sampling. Everything else keeps the host path untouched, and BONSAI_HOST_OBJECTIVE=1 still forces it.

Evidence. Parity better than the tolerance budgeted for transcendentals: resident predictions are bit-identical to the host-objective GPU path on every tested case (LogLoss, Poisson including a clamp-stress fit that is exact by construction, weighted MSE, weighted LogLoss under Bernoulli). Same-pod interleaved A/B at 16M rows, 100 iterations, oblivious: LogLoss 15.4s to 11.5s (25.7% cut), Poisson 14.3s to 10.6s (25.8%), weighted MSE 22.2s to 10.4s (53%), metrics identical to six decimals in every pair.

The surprise the A/B surfaced. The weighted host path was paying a serial 16M-element weight multiply every tree, roughly 80ms/round of single-threaded work. The loop is elementwise with no reduction, so it is now parallel (bitwise-identical models by construction); the 53% weighted cut above was measured against the pre-fix host path, and the honest post-fix comparison will be nearer the unweighted cuts. Weighted CPU fits and ineligible weighted GPU fits inherit the same fix.

Rejected. Fast-math exp intrinsics (parity is worth more than nanoseconds in a membw-bound kernel); softmax residency (per-class trees, its own campaign if ever); GOSS residency (it reads and reweights host gradients by design).

80. The categorical reopener predicate is established; the build decision waits for launch strategy (adopted)¶

Decision. Decision 58's escape hatch named two conditions for reopening native categorical splits: crossed-TS preprocessing failing to close the catboost gap, and a workload where that gap is load-bearing. The first is now measured (scripts/probe_tabarena_cat.py, benchmarks/tabarena-cat-probe-2026-07.md): on the cat-heavy TabArena subset, catboost's own reference toggle (native categoricals vs the same model ordinal-encoded) prices its categorical machinery at 68% of its remaining lead over bonsai with the ordered target encoder (mean share -0.0099 of a -0.0147 remaining gap, 12 datasets), while the pure-numeric control is bit-identical in both arms. The distribution is honest: beyond the chance band on 8 of 12, inside it on 3, and the machinery hurts on 1; the leave-one-out ratio spans 47 to 81%. Where the machinery's price is largest (amazon, kddcup09, splice), ablated catboost loses to bonsai_ts outright: the encoder already matches everything except the native per-split ordered statistics and feature combinations. The machinery also costs catboost 4.2x train time on these datasets.

Consequence. The reopener predicate stands, but no build starts: the second condition (a load-bearing workload) is a launch-strategy judgment tied to whether TabArena standing is a product goal (issue #157). If it fires, the design starting point is architecture doc 17 (priced and shelved), now with a measured target: the per-split share on cat-heavy data, roughly one point of AUC-scale metric. The other two thirds of the catboost story (its pure-numeric small-data lead) belongs to a future ordered-boosting campaign and no categorical work will close it.

Rejected. Treating the TabArena Elo gap alone as the load-bearing workload (a leaderboard is evidence of capability, not of a user's workload); building the doc-17 engine speculatively ahead of the launch call (the admission discipline held once and the price list only got sharper).

81. Ordered boosting: declined at stage 0, the mechanism is not the small-data edge (adopted)¶

Decision. The ordered-boosting campaign died at its own stage 0, per the pre-registered kill. CatBoost's own boosting_type toggle, run at both its defaults and matched knobs on a 12-dataset pure-numeric small-data pool (scripts/probe_ordered_boosting_rung0.py, benchmarks/ordered-boosting-probe-2026-07.md), prices the mechanism at zero or below: Ordered beats Plain beyond the chance band on 0 of 12 at matched knobs (1 of 12 at defaults), the mean share is negative in both task families, and on the two datasets where the toggle moves most Ordered is distinctly worse while bonsai beats CatBoost outright. Ordered also costs about 3.9x Plain's train time, which is the sanity check that the toggle really engaged. The reachability prototype produced its own refutation with a named cause: a faithful 2-fold honest-gradient booster does not converge, because a fold's gradient never sees its own accumulator and therefore never vanishes; early stopping truncates it far short of usefulness, and the simpler two-booster form is bagging, not the mechanism.

Consequence. The long-standing attribution "CatBoost's small-data lead is ordered boosting" is withdrawn from the record. The residual decomposes elsewhere: the probe's single-split harness does not reproduce the bagged gauge's CatBoost lead at all (bonsai is competitive to better on 10 of 12 there), so what remains of the aggregate-leaderboard edge points at the bagged-ensemble protocol interaction and tuning defaults, plus the categorical machinery decision 80 already priced on cat-heavy data. No bonsai booster-math campaign follows from any of those. The what-to-use-when row softens its attribution accordingly.

Rejected. Building any ordered-gradient scheme into the core (the mechanism fails its own vendor's ablation on the target regime); extending the prototype to high fold counts (the non-convergence is structural at small k, and the compute grows k-fold against a mechanism already priced at zero). Reopener: a measured bagged-protocol interaction showing Ordered contributing under bagging where it does not single-split, or a user workload where CatBoost's small-data lead is load-bearing and survives matched-knobs Plain.

82. Static permutation averaging: declined, the categorical substance is per-split (adopted)¶

Decision. The cheapest route to CatBoost's measured categorical share, K-permutation-averaged ordered target statistics as plain preprocessing, was priced on the decision-80 cat-heavy pool at matched protocol (scripts/probe_static_k_encoder.py, benchmarks/static-k-encoder-probe-2026-07.md) and declined by its pre-registered WEAK criterion: K=8 recovers a negative share of the gap to native CatBoost (pool mean -0.026, TS-active -0.039), the K curve is non-monotone (small bump at 4, negative at 8, with an 0.018 swing on a 1000-row dataset between adjacent K values), and the two datasets where decision 80 priced the machinery largest are hurt or flat at both K. The mechanism is named: the K-average converges toward leave-one-out target statistics, the leaky mode decision 58 already measured, while the single ordering's noise acts as implicit regularization that averaging destroys. The reproduction gate held bit-for-bit (max delta 8.3e-17 against the cached gauge), which also corrected the record: the gauge wrapper runs the encoder at cross defaults, not cross=2.

Consequence. The categorical question now has both bounds measured: the encoder-side static ceiling is zero, and the native per-split share is 68 percent of the remaining cat-heavy lead (decision 80). If the doc-17 build ever proceeds, it must contain the dynamic machinery (per-split ordered statistics and path combinations), and it remains gated on the launch-strategy call. One cheap deployment lever is now visible and separate from any build: the TabArena wrapper never enabled the shipped cross=2 encodings that decision 58 measured well on exactly the highest-cardinality pool member; enabling it is integration configuration for issue #157, not research.

Rejected. Productizing K-averaging in the encoder (negative at the only K that matters); higher K (the trend is the wrong direction and the mechanism explains why); treating the K=4 bump as signal (inside small-data variance, non-monotone).

83. An automatic learning-rate default: declined, even the oracle buys nothing (adopted)¶

Decision. The hypothesis that a per-dataset learning-rate rule (CatBoost's automatic default being the exemplar) is a cheap slice of the small-data defaults residual was priced and declined (scripts/probe_lr_rule.py, benchmarks/lr-rule-probe-2026-07.md; the ordered-probe pool and protocol, reproduction bit-exact). The ceiling itself is empty: a validation-selected per-dataset oracle over eight learning rates gains nothing on the pool mean, and its signature is overfitting, winning the validation split on 10 of 12 datasets but the test split on only 6. On the two datasets where CatBoost-default leads beyond the band, the oracle closes 15 percent of one gap and worsens the other. CatBoost's own automatic values, transplanted by reading the resolved rate from its params, sit in a tight 0.024 to 0.075 band around bonsai's shipped 0.05 and are a no-op; its heuristic trends upward with rows while the noisy oracle trends downward, mildly anti-correlated. A flat 0.1 control is worse than 0.05. Caveats recorded honestly: low rates ride early stopping longer (correlation -0.79 with trees kept) and the 1000-tree cap binds at the grid's low end on most datasets.

Consequence. bonsai's fixed 0.05 default stands; no auto-default feature follows. The small-data defaults residual loses its last named single-knob suspect: what remains of the CatBoost aggregate story is the bagged-ensemble protocol and its non-rate defaults (per-split randomization), which interact with bagging and belong to decision 81's bagging-interaction reopener as one combined future probe. A prior worth recording as corrected: the automatic rate had been assessed as the likeliest cheap lever on the leaderboard; the measurement says the lever does not exist at this pool's sizes.

Rejected. Fitting a size rule anyway (the leave-one-out fit swings sign across folds, pure variance); a finer sweep (the ceiling is the problem, not the grid); raising the tree cap to un-truncate low rates (the reference protocol is the comparison and the mid-grid is cap-free on most datasets already).

84. A code-metrics division: the readable-core claim made falsifiable (adopted)¶

Decision. The results site gains a self-only code division (scripts/measure_complexity.py, the results-ledger section, the benchmark-protocol subsection). It measures the bonsai tree at one git SHA with lizard (pinned uvx lizard@1.23.0): per plane (core_headers, engine_impl, cuda_plane, bindings_cli, bench_tooling, tests) it records file count, LOC, NLOC, function count, and mean and max cyclomatic complexity, plus the five highest-CCN core functions by name and the surface counts (45 parameters, 105 dispatch combinations, 9 public Python names, dependency counts). The measurement is drift-gated in CI beside the other generated pages and superseded in place on re-measurement (decision 69). The baseline: core_headers 4,926 LOC at mean CCN 2.04 with a per-function ceiling of 15, and the worst function across the core planes is the CSV parser at CCN 29 (in the IO layer, not the split, tree, or booster headers).

Why. The code-quality pillar was the only one of the four project goals with no results representation; speed had the frontier, accuracy the standings, pedagogy the Learn tracks, and the readable core carried only unfalsifiable prose of the kind the style guide forbids for performance claims. A claim about code you can read must ship with counts you can check. The five worst functions are published by name deliberately; an offender list its author curates away is marketing, not measurement.

Non-claims, recorded. LOC alone is not quality and a small number is not an argument; the division makes the claim checkable, not proven. No comparative claim against any other library is made or implied: comparative measurement (competitors at pinned tags, with a published core-selection rule and a paired capability column) is deferred until this self-only methodology has lived in public. Pre-registered CI budgets on named core functions (a maximum CCN per function, the gate that would have caught the resident-objective seam before review did) are a possible later step, held until the baseline has history to budget against.

Rejected. Comparative from day one (the methodology earns trust on our own tree first, and an unfair competitor selection would poison the division's credibility); LOC-only reporting (complexity is the load-bearing metric, LOC the least informative); hiding the offender list (naming our worst functions is the credibility move).

85. The bagged-protocol interaction: declined, CatBoost's small-data lead is not a randomization interaction (adopted)¶

Decision. Decision 81's reopener named a measured bagged-protocol interaction, CatBoost's randomization defaults decorrelating its ensemble members under bagging where they do not single-split, as the one thing that could resurrect the small-data story. It was priced and refuted (scripts/probe_bagging_interaction.py, benchmarks/bagging-interaction-probe-2026-07.md; the ordered-boosting probe's pool, splits, and loader imported read-only, bonsai_single bit-identical to that probe's bonsai arm on all 12). The headline interaction, (cat_single minus cat_bag8) minus (bonsai_single minus bonsai_bag8), is negative in both pool means (regression -0.032, binary -0.0002) and strictly inside the chance band on 7 of 12; the pre-registered REFUTED wording asked for 10-plus-of-12 in band, so that wording is recorded as not literally met, but the sign analysis makes the refutation firmer, because 4 of the 5 out-of-band cases are NEGATIVE (bonsai gains more from bagging than CatBoost, the opposite of the hypothesis) and the lone CatBoost-favoring beyond-band case is one 460-row coin-flip. The named mechanism is null: randomization_share (neutralizing CatBoost's Bayesian bootstrap and random_strength under bagging) averages -0.0004 on regression and +0.0013 on binary, inside or at the band on 11 of 12, and on the single dataset where the interaction favors CatBoost it covers the whole of it. The plain cause sits in the bagging-gain columns: 8-fold data-bagging already gives bonsai the decorrelation (bagging_gain_bonsai mean +0.104 on regression against CatBoost's +0.072), so there is no residual headroom for CatBoost's per-tree randomization to exploit, and bonsai's shipped sampler knobs (arm 3, bernoulli plus feature subsampling) reach any randomization benefit where one exists.

Consequence. Decision 81's reopener closes on its first clause. The CatBoost small-data lead that does survive under bagging, reproduced directionally on 4 of the 5 gauge datasets by the library-default arm including both largest cached leads, is a LEVEL difference from CatBoost's defaults operating under bagging (arm 7 versus arm 2), not a bagging-specific interaction (arms 4 and 5) and not the randomization the hypothesis named (arm 6). No bonsai core change follows, and no wrapper randomization lever is recommended, because the zero-core-cost response already ships and buys nothing the deterministic bag lacks on this pool. What remains of the whole CatBoost aggregate story is now fully partitioned: the categorical per-split machinery priced on cat-heavy data (decision 80), and defaults-level tuning under bagging, with the booster-math suspects (ordered boosting decision 81, static K encoding 82, an auto learning rate 83, this bagging interaction) all measured to zero.

Rejected. Building or wiring any ensemble-randomization scheme for the bagged regime (the mechanism is null and bonsai's existing knobs already reach it); treating the one positive out-of-band dataset as signal (a single 460-row coin-flip fully absorbed by randomization_share, inside the noise the ordered-boosting and static-K probes both charted at these sizes); reading the gauge-reproduction magnitudes as a bit-match (the BAG8 here is an 8-fold inner bag on a single fold-0 test split, directionally faithful, not the AutoGluon protocol). Reopener: a bit-faithful AutoGluon-bagged reproduction showing a positive CatBoost interaction beyond the band on a pool majority (not one coin-flip), or a user workload where CatBoost's bagged small-data lead is load-bearing and survives matched-knobs Plain single-model, the same load-bearing-workload bar decision 80 set for the categorical build.

86. Honest shadow-feature selection: declined as an accuracy lever on all three growers, the gain ranking already has it (adopted)¶

Decision. A refit-based honest feature selector (a shadow-feature / Boruta prototype, append a permuted copy of every column and keep only real features that beat the 95th percentile of the shadow importances, over 5 seeds with a 3-of-5 vote) was prototyped at zero core cost and priced against CatBoost's in-library select_features and against plain top-k-by-gain truncation (scripts/probe_feature_selection.py, benchmarks/feature-selection-probe-2026-07.md; two regimes over 9 datasets from the ordered-boosting pool, REAL-WIDE up to 1024 features and NOISE-INJECTED with shuffled-copy noise equal to each set's feature count, the bonsai arms run under all three growers). It was declined on its pre-registered second clause, and the decline holds per grower: every beyond-band accuracy win the shadow arm produced is matched by plain top-k truncation at the same k. Under depthwise the shadow arm moved 3 datasets beyond the band (concrete_compressive_strength +0.736 rmse after 8 injected noise columns had cost bonsai_all 0.74, breast_cancer +0.00126, MagicTelescope +0.00172), but shadow_vs_topk is inside the chance band on all 9 datasets, and on concrete and MagicTelescope the two arms select a bit-identical kept set (shadow_vs_topk exactly 0.00000), because bonsai's own gain importance already ranks real features above the permuted ones, the same ranking the shadow threshold reads. Leafwise is bit-identical to depthwise on all 9 datasets (at the campaign shape the 63-leaf budget never binds, so the gain-ordered and level-ordered growers split the same node set; the dispatch was verified engaged with a binding budget) and inherits the verdict line for line. Oblivious, the grower whose coarser one-feature-per-level importance spectrum was the live threat to the top-k control, refutes that threat in the opposite direction: it adds a fourth beyond-band win (wind), every win is again a top-k win at the same k, and the only beyond-band shadow_vs_topk in the whole 27-row grid (pima_diabetes, -0.0072) has the shadow cutoff LOSING to plain truncation. The shadow vote contributes a cutoff rule at 6x the compute of one fit, not a better ranking, under any grower. The ADOPT-SIGNAL branch failed both ways on all three: the machinery also LOSES beyond the band on the same 2 low-dimensional informative sets everywhere (pima_diabetes -0.011, spambase -0.0012), so the no-loss condition is unmet, and it does not match cat_select pool-wide (differs beyond the band on 5 of 9) at half its wall time (depthwise 0.77, leafwise 0.74, oblivious 1.20 of cat_select's 126.9 s pool total, none half or less).

Consequence. Feature selection is not an accuracy lever on this pool. Where it recovers accuracy it is recovering from injected junk that a one-line top-k-by-gain removes just as well, and where the features are all real it costs accuracy. A bonsai.select module, if it ever ships, is a wall-time-and-interpretability tool (smaller models, a noise-detection report), explicitly not an accuracy feature, and no bonsai-core change follows: the whole prototype is Python around the shipped importance("gain") call, and the measurement says even that Python wrapper does not beat a sort of the same importances. The noise-recovery precision/recall table is retained as the deliverable exhibit for the guide chapter that follows this probe: it teaches that both selectors are perfect only when the noise is truly independent (concrete, both drop all 8 noise and keep all 8 real) and imperfect the moment the real features are correlated or the pool is high-dimensional (breast_cancer, the shadow arm drops 16 of 30 real features and cat_select drops 21 while keeping 8 noise).

Rejected. Shipping the shadow selector as an accuracy feature (its wins are top-k's wins on every grower and it loses where features are real); productizing it over plain top-k-by-gain (shadow_vs_topk is inside the band on 26 of the 27 grower-dataset cells and favors truncation in the 27th); reading the concrete recovery as a selection win for the machinery rather than for the ranking (top-k recovers the same accuracy from a bit-identical kept set); treating the oblivious grower as a separate selection story (its coarser spectrum changes the numbers, not the verdict, and its one beyond-band separation favors top-k). Reopener: a workload where shadow_vs_topk clears the band in the shadow's favor on 2 or more datasets under any grower (the shadow cutoff genuinely beating truncation at equal k), or a high-noise-fraction regime where the data-driven cutoff beats every fixed k a user would guess; the interpretability and model-size case is not refuted here and is a separate, non-accuracy decision.

87. The XGBoost 3.3 recheck: every published standing survives, one competitor improvement recorded (adopted)¶

Decision. XGBoost 3.3 (2026-07-21) claimed lower GPU quantile-sketching memory and CPU histogram tiling for wide datasets, both aimed at cells bonsai competes in, so the perf claims were rechecked on one pod (L40S, US-NC-1) with three same-pod arms: bonsai at main, XGBoost 3.2.0, and XGBoost 3.3.0 in separate venvs (benchmarks/results/xgb33-recheck-2026-07.jsonl, 41 rows; the recheck subsection of the results ledger). On GPU, 3.3 matches 3.2 within noise at every measured cell (rows 1M/4M/16M at 100 cols, cols 256 and 1024 at 1M), host RSS does not move (22.1GB at 16M against bonsai's 6.9GB, reproducing the README's 3x memory ratio on a second host), and the order never changes: bonsai_cuda_depthwise fits 3.0-3.3x faster than xgb_cuda at every cell, with zero cpu-fallback nodes in every bonsai profile. On CPU at 16M bonsai is 6% behind xgboost-hist, inside the published "within ~8%, host-dependent" band. The one real improvement in the release: 3.3 halves wide-CPU hist time at 1M x 4096 (783.7s to 387.1s; no change at 1024, the tiling engages by width), which narrows bonsai's CPU lead at that cell from 2.4x to 1.19x (323.9s vs 387.1s) but flips nothing: the ledger's wide-cols standings are GPU, where CatBoost leads and 3.3 changes nothing.

Consequence. README and ledger claims stand unedited; rebaseline-2026-07.jsonl remains the authoritative standings table. The recheck rows carry explicit per-row xgboost version tags, giving the perf claims a recorded version boundary (valid against 3.2.0 and 3.3.0) for the first time. One watch item is recorded rather than acted on: the two widest CPU cells (131k x 16384, 32k x 65536) were not re-measured, and the 4096-col tiling gain suggests 3.3 could pass bonsai's CPU growers there; if wide-CPU standings ever become reader-facing, re-measure those two cells first.

Rejected. Superseding the re-baseline table from this pod (only three of its six variants ran here, and cross-pod absolutes are invalid under the ~25% fleet spread); a README edit (no number cited there moved); treating the 1024-col null as the tiling failing (4096 is simply the first measured cell wide enough for it to engage); re-measuring the CatBoost and LightGBM arms (neither shipped a release since the table was measured).

88. The wide-CPU fill routing: feature-parallel past a 24MB footprint (adopted)¶

Decision. The first production field report (issue #217: 131k x 16384 on a Xeon 6521P, LightGBM-CPU ahead of every bonsai grower 2-3x, matching the committed multi-host scaling rows) traced to the row-wise u8 histogram fill: its per-row scatter targets the whole selected histogram footprint (total_cells x 8B, 33.6MB at 16k features x 255 bins), so past the last-level cache every add misses and the partial slabs add a zero+merge pass per block (populate was 84-88s of a ~100s fit at the stage-0 profile). u8 levels whose footprint exceeds 24MB now route through the existing feature-parallel fill, previously reserved for u16 data: one thread per feature, no partials, no merge, bit-identical at any thread count. Same-pod before/after at 131k x 16384 (t=16, SCALING knobs): depthwise 1019s to 379s (2.7x, LightGBM parity at 367s), leafwise 2591s to 445s (5.8x, the 7x deficit against LightGBM collapses to 1.2x), identical r2, peak RSS 18.8GB against LightGBM's 50.1GB. The threshold was set by an interleaved same-pod A/B at 1M x 4096 (alternating builds, twice each): the row path WINS mid-width on a big-L3 host (320/337s vs 547/547s all-feature-parallel), because the EPYC's L3 absorbs an 8.4MB scatter target while the feature-parallel fill pays scattered per-feature column reads on sparse child nodes. 24MB flips only shapes that lost on every measured host.

Consequence. bonsai's wide-CPU standings move from 2-3x behind LightGBM to parity (depthwise) at 2.7x less memory; the fastest-slot claims (<=256 cols) are untouched and the narrow path is code-identical (fixed-input model hash byte-identical to before; 526 C++ and 67 Python tests green). Above the threshold, model bytes change at identical accuracy, and gain thread-count invariance the row path never had. Evidence: benchmarks/wide-cpu-hist-2026-07.md, raw rows in results/wide-cpu-hist-2026-07.jsonl.

Rejected. A per-node rows bound (<=256k rows feature-parallel, built and withdrawn: the A/B showed sparse-node feature-parallel is the mid-width problem, not node size, and the bound kept the regression at 573s); shipping the M2-calibrated 6MB threshold (it flips 4096-col shapes that a big-L3 host wins on the row path); a config knob for the threshold (a measured constant until a real workload disputes it).

Reopener / recorded follow-ups (issue #217). A cache-size-aware threshold to capture the mid-width win on small-cache hosts (M2: 101.7s to 44.9s at 131k x 4096 goes untaken by the 24MB constant; XGBoost 3.3's aarch64 cache detection is the same lever); skipping the row-major mirror for always-wide fits (2.1GB at the 16k cell); the CUDA planes' own wide wall (~5x behind xgb_cuda at the 16k cell) is untouched by this change.

89. The tiled mirror: one fill retires the wide-CPU strategy pair (adopted)¶

Decision. Decision 88's per-width strategy pair (row-wise below a 24MB footprint, feature-parallel above) lasted one day: the recorded follow-up, XGBoost 3.3's column-tiling lever, was probed immediately and dominated. The u8 row-major mirror moves to a column-block-tiled layout (2048-feature blocks, each row-major on its own; one block at narrow widths reproduces the classic layout byte for byte) and the fill runs tiles outer, rows inner, so the live scatter target is one block's histograms (at most ~4MB) at any selection width while reads stay sequential inside each block. Per-feature accumulation order is unchanged from the untiled row path, so models are BIT-IDENTICAL at every width (verified: identical model sha256 from the main and tiled builds at a multi-block width), which retires the 24MB threshold, the u8 feature-parallel route, and the cache-size-aware-threshold follow-up in one move. Interleaved same-pod A/B (two reps each, single worker): tiled beats the row path at its best cell (326/321s vs 369/377s at 1M x 4096), beats feature-parallel at its best cell (442/448s vs 514/532s at 131k x 16384), and is a wash at 16M x 100 (114/116 vs 119/112); the M2 leg takes the mid-width win the fixed threshold had left on the table (44.8s vs 101.7 at 131k x 4096) and scales linearly to 8192 with no cliff. Identical r2 everywhere.

Consequence. One fill covers every u8 width with no host model, no threshold, and no strategy dichotomy; the feature-parallel fill remains only for u16 data, its original job. Same-machine reproducibility strengthens: the u8 fill's sums no longer depend on selection width routing, and wide models regain byte-stability across versions from here on. Peak RSS at the 16k cell is 21.0GB (against 18.8 for the strategy pair and 50.1 for LightGBM), the cost of per-slice partial stripes. Evidence: the superseding section of benchmarks/wide-cpu-hist-2026-07.md; raw rows tagged run=tiled-ab in the same jsonl.

Rejected. Keeping the strategy pair alongside the tiled fill (dead code with a threshold nobody can calibrate); runtime cache detection (mooted: the tile constant is cache-conservative everywhere measured, M2 through EPYC); a fit-time empirical trial (would trade same-machine byte reproducibility for a decision the tiling makes unnecessary). Reopener: a host or shape where a 2048-feature block's ~4MB histogram footprint is not cache-resident (the tile width becomes the knob to revisit, not the strategy); the CUDA planes' wide wall stays open on issue #217.

90. The CUDA wide recheck: the wall was already gone (adopted)¶

Decision. The campaign opened to close the recorded wide-GPU gap (bonsai_cuda ~355s at 131k x 16384 against xgb_cuda ~72s in the multi-host scaling study) and closed at stage 0, because the gap had already been closed by three weeks of unrelated work: the recorded rows date to 2026-07-07/08 code (git bd783e6/972652c), before the level-transaction and device-resident campaigns landed. On current main, one L40S pod, SCALING knobs (benchmarks/results/cuda-wide-recheck-2026-07.jsonl): bonsai_cuda_depthwise 54.9s at 131k x 16384 against catboost_gpu 71.2s and xgb_cuda 76.7s, and 37.7s at 1M x 4096 against catboost_gpu 50.2s and xgb_cuda 103.7s, at 3-4x less peak host RSS (8.8-16.4GB against 25-60GB). bonsai_cuda_oblivious (61.2s at the 16k cell) also beats both references. The wide-data standings prose ("CatBoost keeps the lead, bonsai second"), stale in README and the results door, is corrected to cite this recheck.

Consequence. No CUDA wide campaign runs; a refutation-by-progress is the deliverable, and the lesson is operational: perf standings quoted from a study must carry their git sha forward, because three weeks of engine work can invert them silently (this is the second stale-row correction this week, after the CPU field report). The stage-0 price list is banked for the future: even in the winning fit, the find stage is ~80% of grow (find_kern 17.3s plus gpu_wait 24.5s of a 54.9s fit at the 16k cell) and the host-side mapper fit costs 11.5s at 16k features, so a further ~2x of internal headroom is measured and recorded on issue #217 without being spent. The full six-variant cols-axis re-baseline (superseding the July 8 study's wide cells properly) is the recorded follow-up before any wide-standings chart ships.

Rejected. Running the campaign anyway against the internal headroom (the standings motivation is gone and the crown items rank higher); superseding the July 8 scaling study wholesale from this two-cell recheck (cross-pod absolutes and the narrow cells remain valid history); leaving the stale prose standing while the jsonl told a different story (the reader-facing claim is the product).

91. The iso-volume shape frontier: measured device memory becomes an output (adopted)¶

Decision. The first campaign on the redesigned bench tooling holds rows x cols constant and sweeps aspect ratio, with measured peak device memory (dev_mem, NVML-sampled per worker child) recorded in every row and the a-priori memory gates disabled by spec, because a measured failure at a shape is the experiment's output rather than a condition to pre-empt. Two committed ladders (benchmarks/specs/): 2^31 cells from 16M x 128 to 32k x 65536, six arms; 2^33 GPU-only to 262k x 32768. One pod, RTX PRO 6000 Blackwell Workstation Edition 96GB (sm_120 via CUDA 12.8 toolkit side-installed over the cuda12.4 image, clang-21 offload), main a907895 (benchmarks/results/iso-volume-2026-08.jsonl). bonsai's CUDA growers are fastest at every cell of both ladders: near-flat 6.8-9.3s across the tall half of the 2^31 line where both references vary 1.5-2x, 4.1x over XGBoost-GPU at 67M x 128 on the 2^33 ladder (27.9 vs 113.8s at 11.7 vs 73.6GB device memory). The instrument's first catches: XGBoost-GPU fails at 32k x 65536 having allocated only 33.4GB of the 96GB card (an internal limit, not exhaustion); CatBoost-GPU allocates 90.2GB at every cell including 1M x 100 (reserves the card, never sizes to the problem); bonsai's footprint tracks the problem, 0.8 to 58.5GB along the ladder. At extreme aspect (p about 2x n) the oblivious grower holds r2 .873 where every depthwise-family arm falls to .815-.817.

Consequence. Feasibility claims now cite measured rows, not estimator output: GPU_MAX_COLS stays as the default policy for unattended sweeps on consumer cards, but it is a policy knob a spec disables, and the 96GB class measurably runs cells the old skip assumed impossible. The ad-hoc campaign-driver era ends: this campaign produced zero hand-written schemas (every row schema v1 with sha, host, and run label stamped by the harness), and the committed spec plus scripts/pod_bench_driver.sh reproduce it in two commands. The work-rig replication path (make bench-iso, host-tagged) stands open for a second same-silicon point. Watch item: CUDA 13 stays blocked until clang can target it; the campaign runs on the 12.8 toolkit ceiling.

Rejected. Fixing a VRAM budget instead of a cell budget (device memory is library-dependent, so the budget would encode one library's allocator; constant logical volume isolates shape and lets memory be the measurement); nvcc for the sm_120 build to reach CUDA 13 (the campaign must measure the clang-built binary path the wheel ships); baking a cuda12.8 image (the side-install costs 2 minutes a session and the 12.4 image is the only one that boots fleet-wide).

92. The results lifecycle: standings supersede in place, evidence freezes, staleness hard-fails (adopted)¶

Decision. Results files conflated two roles and the conflation produced drift three times in one month (the scaling history's stale wide cells stood until decision 90; the adversarial sweep found four more stale reader-facing standings claims). The split: evidence files are the dated record behind a decision, frozen forever, append-only, corrected by banner plus log entry; standings files are the current claim on one published axis, listed in benchmarks/standings.json with the single sha their rows were measured at, superseded in place by re-measurement (decision 69's code-division rule generalized). The ledger stamps every standings caption with the measured sha computed from the rows. Two gates hard-fail via scripts/check_standings.py: a decisions entry claiming a perf change carries a Standings: <axis> line and docs-check fails while any tagged entry outruns the axis's registered state (this entry, 92, is the parse baseline); the wheels publish job fails unless every axis was refreshed for exactly the version being released. Reader-facing prose never restates standings digits. Deleted under the policy (git history archives): the multi-host scaling history and its exponent analysis, the retired year-MSD track, and the Catch2 microbenches, which measured kernels in isolation and were structurally blind to the memory-system effects that actually moved standings.

Consequence. The refresh becomes one rented pod session, automated as the standings-refresh workflow: a same-pod A/B of the previous release wheel against HEAD on anchor cells is the perf-change detector (the only rigorous one, since cross-pod and CI-runner timing comparisons are invalid at fleet spread), then the standings specs re-measure and the supersession lands as one reviewed PR whose body reports the A/B verdict; a moved verdict demands a tagged decision, which the claim gate then enforces. Releases inherit a fixed cost of roughly one pod session and gain standings never older than one release. The perf division sheds its history-file class entirely; repeatability questions are answered by reps within a standings run, not by cross-vintage archaeology.

Rejected. Recipe-only storage for perf (a recipe reproduces the procedure, not the numbers; the fleet-spread rule exists because hardware is not a controlled instrument, so the measured single-sha jsonl is the irreducible minimal artifact and it already carries the recipe in every row); CI microbench tripwires (shared-runner noise floors and kernel-scope blindness; instruction counting is stable but measures the wrong thing); banner annotations for stale standings (banners rot, deletion cannot); a calendar cron for refreshes (manual plus pre-release bounds staleness without surprise spend).

93. First automated standings refresh: stale-vintage supersession, airline speed flip (adopted)¶

Decision. The standings-refresh workflow's first successful run (2026-07-31, one L40S, sha d3ffcd0) supersedes the rows, width, frontier, and airline standings in place. The same-pod A/B against the 1.5.4 wheel read no code movement: all four anchor cells within ±0.5%, so HEAD and the shipped wheel are performance-identical.

Standings: rows, width, frontier, airline

What moved and why. The published numbers moved anyway, because the superseded files predated the late-July engine work that shipped in 1.5.4: the rows ladder was measured at 434a382 (2026-07-13), before the tiled CPU fill and the radix mapper sort. The refreshed 16M x 100 headline is 10.3s (CUDA oblivious) against XGBoost-GPU's 19.6s, where the stale file read 18.4s vs 19.9s. On the airline shape the speed standings flip: bonsai CUDA depthwise is now the fastest fit at 1M and 10M rows under ordinal encoding (1.9s vs XGBoost-GPU's 2.6s at 10M), retiring the long-standing "XGBoost-GPU owns raw speed on the narrow shape" reading; XGBoost keeps only the 100k cell. The frontier holds its shape: bonsai first to every measured accuracy, terminal accuracies tied within the noise band, marginal round below CatBoost's. Width holds: bonsai CUDA fastest at every width, with the CPU arms trading the widest cell inside a rep's noise.

Consequence. This is the policy working as designed: the drift was not unreleased speed but unrefreshed files, and the refresh caught it in one pod session. Narrative captions that restated standings digits (airline provenance, frontier provenance, the cols re-baseline prose) are rewritten digit-free; the tables and generator-computed stamps carry the numbers. Chart ticks on the frontier page now derive from the data after the hardcoded ones stranded outside the refreshed range.

94. LightGBM-CUDA joins the reference arms; August supersession (adopted)¶

Decision. The bonsai-ci image builds lightgbm from source with USE_CUDA=ON (PR #257), and lgbm_cuda joins the rows, width, and frontier suites. The 2026-08-01 refresh (one L40S, sha 0b077ad) supersedes all four fast axes; the A/B against the 1.5.4 wheel read no code movement (four anchors within ±0.8%).

Standings: rows, width, frontier, airline

What the new arm shows. LightGBM-CUDA is real GPU performance at scale, not the small-data loser the decision-42-era snapshot suggested: at 16M x 100 it fits in 31.7s, 6.6x its own CPU (210.2s) and ahead of XGBoost-GPU on this pod, though 3x behind bonsai (10.5s). Its test r2 runs consistently higher at matched knobs (0.884-0.886 vs the field's 0.876-0.880), an implementation difference worth its own probe. At width it hits the histogram wall hardest of any arm: 563.7s at 131k x 16384, 11x bonsai's 50.3s, validating the decision-42 regime argument where it was made. bonsai keeps the fastest slot at every cell of every ladder.

Fleet-variance caveat, recorded. XGBoost-GPU's 16M anchor swung 19.6s to 36.9s between two same-model pods (identical CPU class) while bonsai's held within 2%; this exceeds the documented ~25% spread and flatters bonsai's published margin over XGBoost by pod luck. The same-pod ladder remains the standings per protocol, but the A/B detector anchors only bonsai arms today; adding one reference anchor to the A/B would catch reference-side pod pathology. Follow-up, not blocking.

Also fixed. The refresh workflow's month-rollover bug: a supersession across months deletes the old dated files, and the committed-files render gate (git ls-files) fired before the deletions were staged. Deletions now stage before rendering.

95. The leafwise recheck: decision 42's claim inverts at scale (adopted)¶

Decision. The decision-42-era reading "CPU leafwise beats LightGBM's CUDA leaf-wise" is superseded by measurement: on one pod (L40S, US-MO-1, 2026-08-01, leafwise-recheck-2026-08.jsonl) LightGBM-CUDA wins at every ladder scale, monotonically from 1.2x at 250k to 5.3x at 16M (24.4s vs 128.4s). The original claim was measured at 464k rows and was a small-n artifact. Doc 11's leafwise row carries the correction; the recheck table lives on the scale page.

What stands, what changes. The engineering conclusions of decision 42 stand: no cuda_leafwise registration that computes CPU histograms under a GPU name, and bonsai's existing CUDA growers dominate lgbm_cuda outright (same-pod anchor 17.2s vs 24.4s at 16M; frontier time-to-accuracy 2.2-3.1x, decision 94's data). What changes is the deferral's premise: LightGBM demonstrates leaf-wise-on-GPU is viable at scale, so the device-leafwise deferral converts from "structurally unpromising" to a tracked engine gap with a measured competitive target - issue #268, kill criterion pre-registered (beat lgbm_cuda at 16M x 100 same-pod or do not register).

Also recorded. The r2 pattern across all leafwise-recheck cells matches decision 94's depth-cap finding: lgbm_cuda scores .879-.886 while every depth-8-honoring arm (bonsai leafwise, lgbm_cpu included) sits at .872-.879, and lgbm_cpu with max_depth=-1 reproduces the CUDA scores exactly - LightGBM's CUDA learner does not enforce max_depth, so its quality column in the ladders is not at protocol knobs. Cross-pod CPU variance also logged: lgbm_cpu's 16M fit measured 104.7s here vs 210.2s on the August ladder pod, a 2x swing on a CPU arm while the GPU anchor moved 42%.

96. The standings-refresh CI workflow retires for a local driver (adopted)¶

Decision. The standings-refresh workflow's tail (supersede files, render, open the bot PR) failed on all four of its dispatches, each on a distinct one-shot bug found only by paying for the roughly three-hour pod run that reaches it: pip missing in the bonsai-ci image's uv venv, a rows-only schema ZeroDivisionError in the A/B verdict table, and the month-rollover staging bug decision 94 already logged. Every fix landed after the fact, on the next paid dispatch. The measurement half never failed: all four runs rented the pod, ran the standings specs, and produced correct jsonl, with the artifact-upload step (added after the first tail crash) recovering the data every time the tail crashed downstream. The structural problem is that the tail has no test harness short of a real pod rental, so the workflow's own CI never caught what it broke. scripts/standings_refresh.py replaces it: a measure phase (pod create, detached on-pod run, incremental scp pulls, teardown plus stray-pod sweep) and a supersede phase (registry update, staged git add -A benchmarks/ before render, A/B verdict, branch, commit, gh pr create, with a --no-pr escape hatch) that runs on a developer machine and can be rerun without touching a pod. The supersede phase was dry-run validated against the 2026-08 measurement artifact before this decision.

Consequence. The decision-92 refresh ritual is unchanged: same two phases in the same order, same release ordering (bump PR merges first, then refresh with prev_version set, then tag), same Standings:-tagged-decision gate on a moved verdict, same freshness check at publish time. Only the vehicle moved, from a GitHub Actions runner to a local script invoked by hand per the runbook. The tail is now iterable at the cost of a supersede-phase rerun against an already-measured results directory, not a fresh pod rental, so the next tail bug (there will be one) costs minutes instead of hours. Still unexercised: the supersede phase's gh pr create step itself, since the dry run stopped at the commit; the first live run is that step's real test.

Rejected. Iterating on the CI tail in place (the four dispatches already show this costs one paid pod session per bug, with no way to shorten the loop); splitting responsibilities, CI for measurement and a local script for supersede (two systems for one ritual, and the CI half brings back the pip-in-uv-venv class of failure for no reliability gain now that the local driver also measures); a scheduled cron trigger (decision 92 already rejected this for the same workflow, and a flaky tail makes an unattended cadence worse, not better). Reopener: if the supersede phase accumulates its own run-scarce failure history, revisit whether a cheap local test harness (fixture results directory, no pod) can be built before deciding whether any part belongs back on a runner.

97. `cuda_leafwise` admitted by measurement (adopted)¶

Decision. The device leafwise plane of 20-cuda-leafwise.md keeps its registration. Issue #268's kill criterion was pre-registered and is met: on one pod (L40S, US-NC-1, 2026-08-01, leafwise-ladder-2026-08.jsonl, sha 2a23f33) cuda_leafwise fits 16M x 100 in 30.8s against lgbm_cuda's 32.4s. The secondary bar is met with room: it beats same-pod CPU leafwise 4.6x to 7.1x at every cell. Four arms, best of two reps, interleaved, SCALING knobs at 100 iters and 256 leaves.

rows	cuda leafwise	lgbm cuda	leafwise (cpu)	cuda dw (anchor)
250k	2.9s	6.7s	13.5s	0.7s
1M	4.4s	7.7s	29.3s	1.6s
4M	10.2s	12.2s	66.9s	6.0s
16M	30.8s	32.4s	219.8s	22.6s

The uncapped-depth reading. At the protocol knobs a 256-leaf budget under a depth-8 cap is the full tree, so every arm returns the same tree and the capped ladder cannot see what leaf-wise is for; it is also the regime where lgbm_cuda's r2 column reads high for the reason decision 95 identified, its CUDA learner ignoring max_depth. The extra 16M arm lifts the cap (explicit 256 leaves, no depth limit) and settles both. Uncapped, cuda_leafwise scores .8859 and CPU leafwise .8859 against lgbm_cuda's .8858: the quality gap the capped rows showed was the depth cap, not the engine, and LightGBM's fixed .8858 across both arms confirms the cap was never binding on its side. The time reading goes the other way and is recorded as-is: uncapped, cuda_leafwise takes 38.6s against lgbm_cuda's 31.6s, because best-first without a cap must find a split for every leaf it creates, doubling the round count from 25,500 to 51,100 where the capped tree skips the find at the cap. bonsai is faster at matched knobs and slower at matched accuracy, and the honest statement of the admission is that both hold.

Where the time goes. The profiled 16M rep prices the frontier serialization at 350 us per round: 19.70s of grow against the depthwise plane's 10.78s for identical histogram volume, over 25,500 rounds. That is 3.5x the design budget and 11x what the stage 0 skeleton probe measured, and the counters say why the skeleton was wrong rather than the design. 51,085 launches, 9.80s of find-stage host time against 3.46s of measured kernel time, gpu_wait at zero: the plane is launch-bound with the device idle, the exact inverse of the depthwise anchor's 7.96s gpu_wait. Trivial kernels in a cadence harness model neither the real kernels' occupancy on a frontier of one nor the staging around them. The admission stands on the ladder, not on the budget, and doc 20 now carries the decomposition and the ranked levers (launch batching, the 32 ms/tree setup residue, adaptive accumulator width).

Consequence. Every bonsai grower now has device support, closing the gap issue #268 opened; a leaf-wise user gets 30.8s at 16M where before the choice was 219.8s on bonsai's CPU arm or 32.4s by switching libraries. bonsai_cuda_leafwise joins the benchmark variant registry but deliberately not the SCALING suite tuple: the standings specs do not carry the leafwise arms, so no standings verdict moves and none of the four standings files is touched. Benchmark cells may now name an explicit num_leaves, which the uncapped arm needs and which nothing else uses.

Rejected. Withholding the registration until F reaches the 100 us budget (the kill criterion was pre-registered as a fit-time comparison against a named competitor, and moving the bar after seeing the data is the failure mode admission gates exist to prevent); reporting only the capped ladder (it is the flattering half, and the uncapped arm is where the strategy differs and where LightGBM wins); adding the arm to the standings sweep (it would expand every future refresh by a slow CPU arm and a device arm for a comparison the standings do not make). Reopener: if the launch-count levers land and the uncapped arm reaches parity with lgbm_cuda on time, the uncapped regime becomes publishable as a standings shape rather than a doc-20 footnote.

98. Device leafwise stage 3: two levers land, one is refuted (adopted)¶

Decision. Stage 3 of the device leafwise campaign closes here: two levers landed, a third was built, measured, and reverted, and the closing ladder is the record. On one pod (L40S, US-NC-1, 2026-08-02, leafwise-stage3-2026-08.jsonl, sha 703a78e) cuda_leafwise fits 16M x 100 in 24.4s against lgbm_cuda's 31.9s, where decision 97 measured 30.8s against 32.4s. Three device arms, best of two reps, interleaved, unprofiled; the CPU leafwise arm is unchanged since decision 97 and is cited from it rather than re-measured.

rows	cuda leafwise	lgbm cuda	cuda dw (anchor)	decision 97 leafwise
250k	2.1s	6.6s	0.7s	2.9s
1M	3.2s	7.6s	1.6s	4.4s
4M	7.6s	12.3s	5.9s	10.2s
16M	24.4s	31.9s	23.0s	30.8s

The two ladders are different rentals of the same GPU model, so the reference arms carry the cross-ladder reading: lgbm_cuda and the depthwise anchor reproduce their decision 97 times within 2.4% at every cell, which licenses reading the leafwise column's 21% to 27% cut as the levers rather than the rental. Against decision 97's CPU arm the plane is now 6.4x to 9.2x faster, the margin over lgbm_cuda at 16M widens from 5% to 24%, and the fit gap to resident depthwise at 16M falls from 8.2s to 1.3s. r2_test is identical to decision 97 at every cell.

Lever 1, the device-resident objective (landed). The seam doc 20 put out of scope for stage 1 now arms for leafwise growth: eligible fits keep labels and scores on the device for the whole fit, so the per-tree gradient upload, the values and leaf-id downloads, and the host objective and score loops all go away. Same-pod A/B at 16M x 100, two reps each: median fit 30.23s to 24.60s, median grow 20.02s to 13.13s, r2_test and r2_train identical on every run rather than merely within tolerance. Arming is decided once per fit against the conservative bound (every feature selected, the widest feature sizing the stride) because a tree that declined to the host plane mid-fit would have no host gradients to fall back on; the whole decline predicate lives in one leaf_budget_ok.

Lever 2, pinned and packed round staging (landed, over a gate it misses at one cell). The fit-constant monotone vector uploads once per tree instead of once per split find, the histogram kernels' (offset, count, slot) triple becomes one packed upload behind three pointers into it, and what remains moves to pinned host memory and asynchronous copies: four per round against eight pageable ones. The saving is a fixed 0.37s of grow per fit, 14.5 us of the round against 25,500 rounds, and it is the same absolute number at every scale because it is per round and not per row. Read at the 16M gate cell that is 1.8% of fit against a 2% bar; it clears on grow (2.6%, ten interleaved blocks out of ten) and by 5x to 8x on fit at the other two cells of the same ladder (250k -17.1%, 1M -11.0%), and it ships on that basis. The deviation is recorded rather than hidden, and the rule it argues for is that a fixed-cost lever is gated across the ladder, not at the one cell where the most compute dilutes it.

Lever 3, the partition chain (built, measured, reverted). A parallel tiled segment scan replaced the single-threaded one, and the range copy-back moved to a non-blocking stream under the round's histogram, which read the smaller child from the scratch side the scatter had just written. Against its own parent it moves the paired median fit +0.03% at 16M, -1.2% at 1M, and +0.3% at 250k, with grow -0.8% at 16M, against the same 2% gate. Both halves are reverted rather than carried as a stream and an event nobody paid for, and the refutation is the deliverable: the 0.5s to 0.9s the stage 2 diagnosis attributed to the copy-back and the scan is real device time, but it overlaps work already in flight, so removing it buys back only the fraction that was on the critical path.

The stage 2 diagnosis was profiler misattribution, and the correction is the reason stage 3 worked. Decision 97 read the plane as launch-bound with the device idle at 350 us per round. Re-instrumented with per-kernel events, find_stage's 9.80s turns out to be 84% device compute in disguise: leaf_find's first pageable staging copy stream-syncs, so it absorbed the in-flight histogram and subtract kernels (8.2s) that the sibling planes peel into gpu_wait. The histogram runs at depthwise parity (8.99s against 8.77s of device time for the same volume), so there is no leafwise histogram penalty at this leaf budget and level batching would buy nothing. What decision 97's comparison actually measured was an uneven one: depthwise ran resident and leafwise could not, a seam a kill-switch A/B prices at 5.90s. The true leaf-plane penalty was 3.6s, and the ~200 us per round of host residue read off the profiled run costs 14.5 us unprofiled. Profiling stays the right instrument for attributing device time between kernels and the wrong one for pricing host residue on rounds this short; any future leaf-plane lever is gated on unprofiled wall clock at more than one scale.

The uncapped arm, honestly. Decision 97 recorded uncapped 16M at 38.6s against lgbm_cuda's 31.6s at equal accuracy and called it the admission's open number. The levers cut it 16% to 32.3s against 31.7s, with r2_test .8862 against .8858, so LightGBM's lead at matched accuracy falls from 22% to 2% and does not disappear. The reading stands as decision 97 wrote it, one step smaller: bonsai is faster at matched knobs and marginally slower at matched accuracy, and the remaining 0.6s is the same structural fact, that uncapped best-first must find a split for every leaf it creates, doubling the round count from 25,500 to 51,100.

Consequence. A leafwise GPU user gets 24.4s at 16M x 100 where decision 97 shipped 30.8s and the pre-registration bar was LightGBM's 32.4s. The registration, the dispatch grid, and the standings are untouched: no standings spec carries a leafwise arm, so nothing published moves and no axis needs a refresh. Three things stay open and are named rather than scheduled: the last 1.3s of fit to resident depthwise parity at 16M, which is now small enough that the next lever must be priced before it is built; small-node occupancy, which is real only when the mean node falls below roughly 150k rows and is therefore gated on that regime rather than on this ladder; and issue #278, the leafwise CPU-vs-GPU parity bound flaking at 1.06e-4 against its 1e-4 contract, which predates stage 3 and is a calibration decision on a contract surface (decision 40), not a drive-by widening.

Rejected. Carrying lever 3 anyway because its grow number was negative in every block (a lever that does not move fit is not a lever, and a stream plus an event is permanent complexity paid for a measurement inside the noise); holding lever 2 back on its missed 2% at the gate cell (the gate is a percentage and the saving is a fixed cost, so the gate cell is structurally the worst place to read it, and the same lever is 5x to 8x over the bar at the cells where the round is the fit); re-measuring CPU leafwise on this pod for a complete four-arm table (it is unchanged code, it costs more pod minutes than every device arm combined, and decision 97's numbers are the honest citation); widening the parity bound while the flake was in front of us (issue #278 exists so that the contract moves deliberately). Reopener: if a priced lever closes the last 1.3s to depthwise, or if a sub-150k-row regime makes small-node occupancy the top term, stage 4 opens with the same discipline: unprofiled wall clock, more than one scale, and a pre-registered gate read across the ladder.

99. `Dataset` takes a device hint, so the two-step form keeps device binning (adopted)¶

Decision. bonsai.Dataset(X, y, ..., device="cuda", device_id=0) bins on the device, and the two-step workflow reaches the same ingest path the fused train(pairs, X, y) call has used since decision 54. Until now it could not: device binning is chosen inside make_labeled from the grower name, and a standalone Dataset construction has no grower, so it always binned on the host. The cost was measured on one L40S at 4M x 100: fit 2.92s against 4.30s and peak host RSS 1.86GB against 2.51GB, identical r2, with the ingest profile reading dbin=0.09s bin=0.00s fused and dbin=0.00s bin=1.14s two-step. The benchmark harness hit exactly this and published it into a standings refresh before it was caught (issue #290 records the protocol response). Parity is the admission bar and it is met: 4M reads 3.207s fused against 3.209s hinted, 16M 12.056s against 12.005s, both inside interleaved-repeat noise on time and memory.

Consequence. A hyperparameter sweep binds its data once. The engine is rebuilt per train call, so a sweep over a host array pays mapper-fit plus device binning on every fit; over a device-hinted Dataset it pays once, measured at 3.63s against 0.74s over five fits at 16M, which is 5% of that sweep's wall clock and grows as the per-fit iteration count falls. Mismatch is resolved by materialization rather than refusal: a device-binned Dataset handed to a CPU grower fills its host bins on first host consumer, through the call_once path Dataset already had, and its model is byte-identical to a host Dataset's. A parallel.device_id that disagrees with the Dataset raises before any device work; device="cuda" without a build or a device raises, because it is an explicit request rather than an engine inference, while the oversized-max_bin ingest decline stays silent and reports device == "cpu" truthfully. Device-binned Datasets do not pickle. The constructor also gained n_threads, which it never honored before, so its binning ran outside parallel.n_threads.

Rejected. Binning both host and device eagerly (pays memory for a fallback most callers never take); refusing the CPU-grower combination outright (the materialization path already existed and refusing would make a reusable Dataset less reusable, which is the point of the feature); inferring the device from the first train call instead of taking a hint (binning would then happen at a moment the caller cannot see, and a sweep's first fit would pay a cost its siblings do not). Reopener: device-resident input (issue #289), which changes where the bytes come from rather than where they are binned, and shares this seam.

100. The device-leafwise campaign was measured on a host-binning harness (adopted)¶

Decision. Every bonsai CUDA number in decisions 95, 97, and 98 was measured through a benchmark harness that could not reach device binning, so the campaign's published absolutes overstate bonsai's fit time and understate the result it reports. The mechanism is decision 99's: at 4e035f0 the bench runner split its fit into Dataset(X, y, max_bin) then train(pairs, ds), and a Dataset built before a grower is named always binned on the host, so every cuda_ arm carried a host binned matrix, and about 2.5GB of extra host memory, for the whole fit (16M x 100 peak RSS reads 9.47GB in the affected ladders against 6.93GB on the fixed path). All three evidence files carry that commit and none carries the fix, 77633a6: leafwise-recheck-2026-08.jsonl at f5e7740, leafwise-ladder-2026-08.jsonl at 2a23f33, leafwise-stage3-2026-08.jsonl at 703a78e. The bias ran one way, against bonsai: no reference arm touched the affected path. Decisions 95, 97, and 98 are not rewritten: this entry is the correction of record, in the pattern decision 95 used on decision 42. Evidence: leafwise-correction-2026-08.jsonl (one L40S, US-NC-1, 2026-08-03, the campaign's three device arms at the campaign's knobs, best of two reps, interleaved, unprofiled, on the fixed path, plus the uncapped arm and two instrument controls).

rows	cuda leafwise	lgbm cuda	cuda dw (anchor)	decision 98 leafwise
250k	3.3s	8.6s	0.8s	2.1s
1M	4.2s	9.7s	1.5s	3.2s
4M	7.2s	14.8s	4.4s	7.6s
16M	18.7s	37.0s	15.8s	24.4s

The rental, stated before the readings that depend on it. This is the slowest of the three L40S rentals the campaign has used: lgbm_cuda, which never touched the affected path, reads 16% to 29% above the times decision 98 published for it, largest at the small cells where the fit is host-side fixed cost. That is why the 250k and 1M leafwise cells read above their published numbers rather than below, and it is why no absolute here should be quoted against a campaign absolute without its reference arm. What survives the rental is the ratio, and a second reading of the load-bearing cell survives with it: the 2026-08-02 standings measured 16M x 100 on the fixed path on a rental that reproduces the campaign's reference times, at cuda_leafwise 13.8s against lgbm_cuda 31.8s. The distance between 18.7s and 13.8s is taken apart rather than left as pod luck: the bench driver's profile counters price at 0% to 2%, the --data-cache memmap every campaign ladder used costs another 12% at 16M (the standings run without it, and dropping it here reads 16.5s leafwise and 13.7s depthwise), and what remains is 19% of rental on identical protocol, in line with the reference arm's 16%.

What moves. The kill criterion pre-registered in issue #268 was met far more decisively than recorded: cuda_leafwise is 2.0x LightGBM's CUDA leaf-wise at 16M on this pod and 2.3x on the standings pod, where decision 97 published 5% of margin and decision 98 published 24%. The fit gap to the resident depthwise anchor at 16M is 2.9s here and 1.7s on the standings pod, not the 1.3s decision 98 recorded: the host binning pass sat on both planes and flattered their distance, so the leaf plane is not as close to depthwise parity as the record claims. Against decision 97's CPU leafwise column the plane is 4.1x to 11.8x rather than 6.4x to 9.2x, spread wider at both ends by the same rental effect. Decision 98's 21% to 27% cut on the stage 2 column stands as measured and is a floor: both ladders carried the same handicap, so removing it from both sides raises the percentage rather than lowering it.

What does not move. Every lever delta in the campaign was a same-path A/B with the handicap on both arms, so decision 98's levers 1 to 3 and doc 20's lever 4 stand exactly as measured. CPU leafwise binned on the host by design, in every run, so decision 95's headline (LightGBM's CUDA leaf-wise beats bonsai's CPU leafwise 5.3x at 16M) is untouched; only the cuda_depthwise anchor row of that recheck carried the handicap. No standings axis moves: the standings were re-measured on the fixed harness for 1.6.0 and already read the corrected numbers, which is how the drift was caught.

The campaign's one negative finding inverts. Decision 97 recorded the uncapped 16M arm at 38.6s against lgbm_cuda's 31.6s and decision 98 at 32.3s against 31.7s, calling LightGBM's 2% lead at matched accuracy the honest half of the admission. On the fixed harness, same pod, same knobs, best of two reps: cuda_leafwise 27.5s at r2_test .8862 against lgbm_cuda 36.8s at .8858. bonsai leads the uncapped cell by 34% at matched accuracy. The structural fact decision 97 named is unchanged, that uncapped best-first must find a split for every leaf it creates and so pays 51,100 rounds against the capped tree's 25,500; what it does not do is cost the arm the comparison.

Also recorded. The bench driver forces BONSAI_GROW_PROFILE, BONSAI_INGEST_PROFILE, BONSAI_CUDA_PROFILE, and BONSAI_FIT_PROFILE on for every bonsai child, so both campaign ladders ran with the counters on despite decision 98 and the ledger describing the closing ladder as unprofiled; the ledger line is corrected here. Priced on this pod at every cell, the counters cost 0% to 2% of leafwise fit (16M: 18.8s against 18.7s), so nothing in the campaign turns on them, and BONSAI_BENCH_NO_PROFILE=1 now exists so the question is measurable rather than assumed. The --data-cache memmap costs more than the instrument it was chosen over: 12% of leafwise fit at 16M, on a flag every campaign ladder passed and the standings do not, which is worth knowing before the next cross-ladder comparison is drawn. One further observation, not chased here: two of ten device-leafwise reps on the fixed path scored r2_test .8798 and .8781 where every other rep of the same cell scored .8793 and .8783, while the affected ladders and the standings reproduced their scores exactly across reps. That is run-to-run variation inside the fit, it is adjacent to issue #278's parity-bound flake, and it wants its own measurement rather than a sentence here.

Rejected. Deleting or rewriting the three superseded evidence files (they are dated records of what was measured, decisions 95, 97, and 98 cite them, and the archive moves by correction rather than by edit; the ledger sections that render them now say in prose that they carry the handicap and point here); re-measuring CPU leafwise (it binned on the host in every run, so nothing about it moved, and it costs more pod minutes than every device arm combined); publishing the correction from the standings alone (they carry one cell, not a ladder, and not the uncapped arm that inverts); renting a second pod for absolutes that match the campaign's rentals (three rentals already agree on the ratio, which is the claim, and decision 94's fleet-variance caveat is the standing answer for the rest). Reopener: issue #290 pins the ingest contract so that a device arm which silently bins on the host fails loudly instead of waiting for a reader to notice an odd table; if that lands and any published table still disagrees with it, this correction reopens rather than a fourth ladder being rented.

101. Perf rows report ingest and train for every arm (adopted)¶

Status 2026-08-04: the committed standings behind this entry were measured before the two-step runner merged and carried no bonsai split; the 2026-08-04 refresh supersedes them with the split measured for every arm.

Decision. Every perf row now carries ingest_s and train_s alongside fit_s, bonsai included. The bench runner fits through Dataset(X, y, max_bin=..., device=...) then train(pairs, dataset); fit_s remains the outer wall clock over both and is never redefined as their sum. This was impossible until decision 99: a prebuilt Dataset without a device hint bins on the host whatever grower follows it, so its ingest number would describe a pipeline no GPU arm runs. That is not a hypothetical failure, it reached a published refresh and was withdrawn (decision 100).

Consequence. The split says something the total hides, and it says it against us. At 16M x 100 on one L40S, XGBoost trains in 7.0s against bonsai's 11.0s, 36% faster, while bonsai's ingest is 1.2s against XGBoost's 30.0s. The 3x margin in the total column is ingest, not boosting: bonsai bins on the device where the references sketch on the host and ship the result across the bus. Against LightGBM's CUDA leaf-wise and CatBoost's GPU oblivious, bonsai leads both halves (12.8s against 15.4s of training, 9.4s against 19.6s), so the deficit is specific to XGBoost's depth-wise kernels and is now a named target rather than an invisible one. The two halves also scale differently with hardware: on a second host with a faster card, bonsai's training halved while its ingest held flat and XGBoost's ingest got worse, which a single column cannot express.

The standing evidence is a parity arm, not an assertion. Every refresh fits the anchor cell through both the fused and two-step forms, interleaved on the same pod, banded at 5% on fit_s and on peak RSS, and a failure stops the supersession before it touches a file rather than annotating the result. This run read 12.01s fused against 11.99s two-step, 0.2% apart, with the split at ingest 1.05s and train 10.94s. Without that arm a regression in the device hint would silently reintroduce host binning and post a plausible ingest number, which is exactly how the withdrawn refresh passed unnoticed.

Rejected. Reporting the split for the reference libraries only, which is what the harness did and what made bonsai's column read as absent rather than fast; deriving bonsai's ingest from the profile counters, which perturb short rounds and are the wrong instrument (decision 98); and reusing one Dataset across repeats or variants, which amortizes a cost each row must be charged in full. The amortization is real and worth measuring, but as its own study: one device Dataset served three growers in 1.1s of ingest plus 25.3s of fits rather than three separate ingests.

102. Device-resident input bins in place (adopted)¶

Decision. Any CUDA array supporting DLPack is accepted as training data and binned where it already lives, so a caller whose data is on the GPU reaches a trained model without a host round trip. This closes the one regime where bonsai's ingest advantage inverted: for host input, device binning means the host never materializes a second full-size copy (decision 54, and the headroom column added under issue #291 prices it at roughly 8% of the input array against 2.5x to 3.5x for the reference libraries), but a caller already holding a device array had to copy down to host memory first, which XGBoost's QuantileDMatrix avoids by sketching in place. Measured on one L40S at 4M x 100 with the data already resident, 20 iterations: 1.30s and 1.28s to download and fit, against 1.04s and 0.96s binning in place, maximum prediction difference 9.5e-7.

Consequence. The bin mapper still cuts on a host sample, and deliberately so: sampling on the device would change the sampled set and therefore the model, which doc 15's phase-2 rule forbids. The device arm instead gathers exactly the rows bin_sample_rows names into a compact block, downloads that block, and runs the ordinary BinMappers::fit, which re-samples it and finds it already at sample size. Cuts are bit-identical to the host path by construction, and the download is the sample rather than the matrix: 80MB against 6.4GB at 16M x 100. Ownership is borrow-for-one-call: ingest copies into a plane owning its own device memory, so nothing bonsai holds points into the caller's allocation afterward. Stream ordering is the producer's under DLPack, which synchronizes at export, and bonsai reads on the default stream, so there is no handle to plumb and nothing to wait on. Labels and weights are accepted device-resident and downloaded once, because every consumer of them (the host objective, the eval loop, the resident uploader) reads them host-side; when that stops being true they can stay. Mismatch follows decision 99: a device-id disagreement raises before any device work, a device request without a device or a build raises, and device="cpu" with device input raises rather than silently copying back. The cuda_ingest bin-count decline inverts here, since there is no host copy to fall back to: the device arm always mints a plane, and a declining grower takes the lazy host materialization instead.

Parity, stated in two parts because it is two claims. For CPU growers the model artifact is byte-identical to the host path, which is the bin-identity claim: the grower reads the device's bins materialized on the host. For cuda_* growers the contract is tolerance-equal at 1e-4 on predictions and nothing stronger, because device float atomics are not reproducible run to run; three repeats of one cuda_depthwise fit on one unchanged host array produced three different model hashes on the same pod. That is the same phenomenon behind the r2 spread recorded in decision 100 and the parity-bound flake of issue #278, and it is a property of the device plane rather than of this feature.

Rejected. Sampling the bin mapper's cuts on the device (changes the model, and the compact-gather download costs 80MB to avoid that); retaining a borrowed device pointer past the call (an aliasing contract callers cannot see); accepting device input into the standings ladders (every arm is handed the same host array, and a device-input study is a separate measurement where every arm that supports such input is measured on it); parsing __cuda_array_interface__ by hand (built first and replaced before ship: nanobind's DLPack import runs the same validation with no owned code, and the one producer class it does not reach, numba's device array, arrives through cupy.asarray). Reopener: keeping labels device-resident once a consumer can read them there.

103. The device bin plane is tile-blocked (adopted)¶

Status 2026-09-05: the quantised sub-histogram reopener is taken by decision 124 (int64 fixed point); the tile's shared cost doubled rather than halved, so width 16 stays priced, not opened.

Standings: rows, width, frontier, airline

Decision. The CUDA binned matrix is written and read in feature tiles of 8: feature f lives in tile f / 8 at strip position f % 8, tile t starts at cell n_rows * t * 8, and one row's strip inside it is the tail-aware min(8, n_feats - t * 8) cells wide. This is the layout decision 89 adopted on the host (Dataset::row_major_bins), at the width device shared memory allows. It is the plane, not a copy of it: both ingest arms write it, the host upload stages it, materialize un-tiles on the way home because the host store is per-feature columns, and every device reader moved with it, the histogram build by a new kernel and the small-node build, the partition count and the resident epilogue by one index expression each. The layout arithmetic lives in exactly one function and the width is one constant. The histogram build owns one tile per block, loads a row's strip in a single aligned vector load, reads the row id and its gradient pair once per tile rather than once per feature, and keeps one sub-histogram per lane with no warp-parity duplication, so eight lanes at 255 bins ask 16 KiB and the whole build stays inside the static shared budget with no opt-in. A tree that subsamples features rides the same tiles through a per-feature slot map, and bin counts too wide for a tile's sub-histograms fall back to one feature per block on the same plane, which costs what the feature-major plane cost, so the wide-bin envelope is unchanged.

Consequence. Measured on one L40S (EU-NL-1, interleaved A/B against main, three reps, 16M x 100), where tile 8 is what ships and tile 16 is the width the depthwise build alone would have picked:

cell	main	tile 16	tile 8
depthwise depth 8, train	11.02s	6.85s	7.23s
depthwise depth 8, `adv_hist`	7.95s	2.49s	3.07s
depthwise depth 4, train	3.58s	3.70s	3.61s
leafwise 16M, train	12.64s	14.65s	11.31s

The histogram kernel the whole gap lived in falls 61%, and the depth-8 fit falls 34%. The crown reading is the one that matters: at this cell bonsai's depthwise arm was 59% behind XGBoost's trainer and is now inside 4%, which is a statistical tie at this pod's spread. Leafwise gains 10.5% for free, having never been the target, depth 4 is neutral at 0.8%, and r2_test spans {0.8793, 0.8798} on both branches equally, the known flutter rather than a difference. The width is 8 and not 16 because one plane serves both growers and the choice is therefore joint: 16 is better for depthwise by 5% of fit and worse for leafwise by 15.9%, which fails the pre-registered 5% leaf-plane bar, and it fails it for the reason doc 20's lever 4 predicted, since a leafwise round histograms one node and a tile-narrowed grid stops filling the device. 8 passes every pre-registered bar and improves both planes, so it ships, and the depthwise width question is not closed but priced: reopener, an adaptive accumulator width (16-bit or quantised sub-histograms, named and never opened in doc 20) would halve the tile's shared cost and put 16 or wider back on the table for both planes at once.

Rejected. Feature grouping at fixed layout, refuted by measurement in PR #332: it varied the block's feature width while the plane stayed feature-major, so it paid a G-fold shared footprint and collected none of the gather benefit, which exists only when the bytes a block needs are adjacent, and its G=8 arm additionally kept the warp-parity duplication and so ran at half the occupancy before reading a byte differently. Row reordering into node order, which sounds free because the partition already computes the ordering and is not: the partition permutes the row list, not the matrix, so a contiguous gather would mean physically permuting the whole matrix once per level, priced at 25.6 GB per tree against the 107 GB it removes, and it is the only option here that changes what a row id means. Full row-major (ELLPACK shape), dominated at this feature count: a block cannot own 100 features of shared histogram, so it reads a partial unaligned strip out of each row and reintroduces exactly the partial-sector waste tiling removes, while route_add loses its coalescing and both the host upload and materialize need a transpose anyway. Per-plane widths, one for depthwise and one for leafwise, which is not an option without a second copy of the matrix and therefore the 1.6 GB the probe form was retired to avoid. The probe record, including the pre-registered criteria this measurement was read against, is PR #335; the shipping measurement is this PR's.

104. The standings are six scenarios on two gated planes (adopted)¶

Standings: gpu-tall, gpu-wide, gpu-extreme, cpu-tall, cpu-wide, gpu-early-stop

Decision. The perf standings are redesigned from four grid axes to six single-cell scenarios: an iso-volume tall/wide pair per plane (2^31 cells on GPU, 2^28 on CPU), one VRAM-ceiling extreme on GPU where an OOM is a published result, and early stopping as a standing behavior axis. Every scenario publishes the same dimensions, the fixed/variable split (ingest_s, train_s) beside peak host RSS and per-process device memory, with one fused wall clock at gpu-tall from the parity arm. Arms pair by growth strategy on one page, grower by grower, hardware never mixed in a table. The standings card moves to the RTX PRO 6000 Blackwell (96GB), the card the extreme scenario is sized to. Axes carry a plane, and the release gate skips any axis whose plane's sources are byte-unchanged since its refresh, so a one-plane change re-measures one plane.

Consequence. A full refresh is 48 jobs where the retired grids ran hundreds, and a routine one is smaller still under the plane gate; the three-hour refresh that motivated this (issue #318) becomes tens of minutes. The framing complaint is answered structurally: fixed and variable costs are separate columns rather than a footnote under a total, memory is two honest numbers rather than one host figure, and the tall/wide contrast replaces ladders whose interior rungs backed no claim. The grower rename rides along (decision at issue #305, shipped in the same train): the growth policy is levelwise; the tree shape remains oblivious where CatBoost's vocabulary is meant.

Retired. The rows, width, shape, frontier, and airline axes and their files; the accuracy-time frontier's unique content (the capacity sweep) and the airline suite's real-data story leave the standings entirely, by ruling, and twenty-three closed-campaign evidence files leave the tree with them. Git history is the archive (decision 92), and the generated archive page maps every retired record to its decisions and the ref where its data lives. The renderer fell by a fifth of the repo's bench tooling in the same stroke.

Rejected. Per-grower pages (one page with panels compares better); keeping the airline axis under the canonical Pafka protocol (ruled out rather than filled); a per-plane width or per-scenario knob surface (SCALING stays the one knob family); resolving archive refs at render time (shallow CI checkouts cannot see deleted-file history, so refs are pinned at generation). Reopener: doc 21's component axis joins this registry with plane: gpu when its design is built; the extreme scenario re-sizes if the standings card changes.

105. The sparse fill is one composition: buffer and reduce (adopted)¶

Decision. The CPU sparse-node histogram fill becomes a single decomposition, admitted as the default and as the only path (issue #360). A level's sparse nodes are cut into fixed 1024-row blocks; the node-major block list is split into n_threads contiguous ranges; the lowest range touching a node accumulates straight into that node's arena; every higher range owns one partial, zeroed on first touch and skipped entirely when never touched; a second pass over (node, feature) pairs sums the partials in ascending range order. The per-node block plan it replaces is deleted with it: block counts derived from node size, selection width and bin footprint and capped at four blocks per thread, one partial per block, a merge per multi-block node. So is BONSAI_HIST_REDUCE, the env toggle that selected between the two during the A/B. Dense-node routing to the column fill is untouched, and the grain is a constant because a block is streamed once and never re-walked, so its size gates no cache reuse and trades only balance against per-block setup.

Measured. The gain is small and one-sided, which is what carried it. Train falls 3% on the 12-thread EPYC, reads 0 to -2% on the 16-thread Xeon, and shows no separable change on the M2 against a rep-to-rep drift near 1s at 2M x 128, depth 8, 100 iterations, 8 threads. Across the interleaved paired reps on every host the new arm never lost one. Peak RSS falls, and that part is structural rather than tuned: because the block list is node-major and thread ranges are contiguous, the ranges touching a node form a run, the runs telescope, and a level needs at most n_threads - 1 partial buffers however many nodes it holds. The old plan bounded blocks per node and paid a partial for each, so its slab grew as O(n_threads x n_nodes) where this one is O(n_threads). At depth 8 that is orders of magnitude.

Consequence. Models shift. A node's summation order now depends on the level-wide partition, so at a fixed thread count and a fixed dataset the depthwise and levelwise CPU model bytes differ from every version before this one, and the next standings refresh re-baselines rather than compares. Determinism itself is unchanged and stated in architecture/7-parallel.md: buffers are keyed by the partition index rather than the OpenMP worker, so scheduling cannot reorder a sum, and a fixed n_threads reproduces. The contract's dependency list is wider than it was, because the block list is cut level-wide: one node's row count moves another node's cut points. The load-balance trade is written down in the same place, because the obvious future fix silently undoes the memory result. Emitting exactly n_threads work units is what buys the partial bound, and it degrades for_each_index to chunk 1 with nothing left to steal, which is the static partition that doc's scheduling rationale warns about on asymmetric cores. The bound and the balance are one dial and this fill picks the bound.

Rejected. Shipping the toggle, which the design review priced as the one outcome worse than both alternatives on every axis it could price: two schedules to keep correct, two determinism stories under a doc that describes one, roughly 175 lines dead on arrival, a knob whose value 1 did not mean 1, and a fill path no test could reach because the selector read the environment. Default or decline, and the measurement decided which. Also rejected: grafting one ingredient of the reference decomposition at a time, refuted twice before this composition was tried whole; and a work-proportional block count, which the old plan needed because a block cost a full slab zero plus a merge to start, and which this one does not, since lazy per-range zeroing and a rebuild of the histogram bases only on node change make a block nearly free to begin.

106. Per-candidate min_data_in_leaf: declined by measurement (adopted)¶

Decision. bonsai keeps its node-level row floor; the per-candidate child floor LightGBM and XGBoost apply is not adopted. Measured at zero core cost via the min_child_hess identity (MSE writes a per-row hessian of 1.0, so min_child_hess = 20 is the feature for regression): on the 55-dataset quality suite over 3 seeds, the strict arm is net negative (20 datasets beyond the decision-55 band, 16 losses; AUC mean -0.008, rmse mean +0.65%), and XGBoost's own min_child_weight toggle loses the same way on the same datasets, so the floor is an over-aggressive regularization default rather than a bonsai deficiency. Sub-floor leaves are common (median 25% of leaves) but hold ~1% of training rows. Evidence in PR #379's record; the probe was deliberately not kept.

Consequence. The semantics live in one place: the interop tables document the non-equivalence at the LightGBM/CatBoost min_data_in_leaf rows, and the parameters page now states what the knob gates. min_child_hess = 20 expresses the strict form today, exactly a 20-row floor under squared error. Side finding: mapping min_data_in_leaf = 20 onto XGBoost's min_child_weight = 20 (decision 68's matched-knob rule) costs XGBoost beyond the band on 23 of 55 datasets; the rule stays, but the handicap is measured now, not assumed. Reopener: a workload whose sub-floor leaves carry real mass, or a row-expressed classification floor beating the default beyond the band.

106. The hist chunk axis fills the card (adopted)¶

Decision. launch_hist owns the chunk policy for all four of its callers: n_chunks is the larger of the row-split term (max_rows / 32768) and a fill term, ceil(sm_count * 4 / (grid_x * n_nodes)), clamped to [1, 64]. Both histogram kernels return before their shared zero when the chunk starts past the node's row count, a block-uniform exit that makes overshooting the chunk axis free on small nodes. The SM count is read once beside the shared-limit probe; a failed query disables the fill term rather than the launch.

Attribution. The small-cell fit-time trade published with the tiled plane (decision 103, issue #340) was occupancy starvation, not per-block fixed cost. Tiling divides grid.x by the tile width, and a shallow level over small data launches tiles x nodes blocks: a fraction of a 142-SM L40S, while the Orin's 8 SMs never notice. The three-arm A/B on the Orin (feature-major plane, tiled plane + per-feature kernel, tiled plane + tiled kernel; 3 interleaved reps) has the tiled kernel WINNING the exact cells that regress on the L40S, 7.33s to 5.41s of level-hist at 1M x 128, which rules the per-block cost story out on the device where fixed costs bite hardest.

Measured. Same-pod interleaved medians on one L40S (US-TX-3, 3 reps, r2_test a single value per cell with one 4th-decimal flutter of the issue #278 class): train at 262k x 128 falls 30.4% depthwise and 22.0% levelwise; 1M falls 6.3% and 8.1%; leafwise at 1M falls 35.3% (4.20s to 2.72s), because its per-round build launches one node's tiles and was the most starved caller of all; 4M and 16M read -0.6% to -0.0%, the no-regression bar met exactly. The Orin guard pair is a no-op (5.41s vs 5.40s), as the fill term predicts for a narrow device.

Rejected. Raising the small-node cutoff: the 2026-08-17 sweep measured every cutoff above 512 worse at every cell, monotonically. Per-node chunk lists: they shrink the chunk axis where the mechanism wants it grown, and the early exit already prices uniform overshoot at a block launch. The early exit alone: measured a no-op at uniform cells on the Orin (workless chunks are rare without skew), kept solely as the guard the fill term composes with. Routing more nodes to hist_small_kernel: unchanged at 512, per the same sweep.

107. Typed params are generated from the section registry (adopted)¶

Decision. The Python surface gains bonsai.Params, one frozen dataclass per config section with every field defaulting to None ("leave the library default"), plus a train() wrapper accepting Params or a dotted-key dict, both rendered to the unchanged (str, str) pairs wire format (accepted publicly in the first cut, retired the same day; the Consequence records the turn). The dataclasses are not written by hand: a _bonsai._params_schema() binding folds the same all_sections tuple dump_toml folds, emitting name, C++ type, and default per field, and scripts/gen_params_py.py renders bonsai/_params.py from it at build time, ordered before the nanobind stub the way the stub is ordered after the extension. Behavior (to_dict/from_dict, | merge with right-side-wins, the sparse repr) lives on a committed mixin, so the generator stays a renderer.

Consequence. The registry stays the single source: a new field<&SubConfig::name>() line appears in Params on the next build with its real C++ type, with no Python mirror to drift (the TOML-inference alternative types lambda_l1 = 0.0's whole-number rendering as int; the registry binding types it float). Misspelled knobs fail at Params/from_dict construction with the section's legal names in the message instead of at fit time in C++. Params | {"tree.max_depth": d} is the sweep idiom, and the dotted key doubling as the optuna trial name makes objectives one merge long. The pairs wire format is untouched, so every model-hash gate holds; data.* and CLI-only sections generate like the rest rather than maintaining a curated subset. The bench seating landed in the same change set: interop.to_* accepts a Params (the dict(pairs) normalization grows one isinstance), the *_core builders state their cells as Params.from_dict literals (retiring the _BONSAI_KEY hand mirror and _translated), BONSAI_CAMPAIGN_PARAMS derives its values from CAMPAIGN under registry validation, and a spec's defaults block rejects unknown keys at load. With bonsai's one production consumer being its own author, the pairs form then retired from the public wrapper in the same change set: train accepts Params | dict | None and raises on a pairs list with the dict(pairs) escape named, interop.from_* returns a Params with typed values (the round trip through to_* is now exact rather than stringified), and the pairs remain only as the native wire format under the wrapper. The config= argument then retired from train and the estimators (a mix of concerns, Daniel's call): Params.from_toml(path) carries only the keys a TOML file states — parsed by a typed_overrides walker in the C++ config layer, so no Python TOML dependency and no 3.11 floor — and from_toml(path) | overrides expresses the -c + --set layering through the one params argument. Params.from_model rides the same binding (the resolved config, every key set), closing that deferred item. The Dataset path's [bin_mapper] config-file check is subsumed: a TOML base arrives as ordinary pairs and hits the existing rejection. Still deferred: estimator re-seating (params= and fit accepting Params) and per-round callbacks for optuna pruning.

Rejected. A hand-written Python mirror (a second source of truth, the exact drift class bench/params.py's docstring documents). Runtime TOML parsing of default_config_toml() (needs tomli on 3.9/3.10, and inferred types carry the int-for-float trap). A committed generated file with a CI staleness check (reviewable, but adds a render ritual and install plumbing the stub precedent already avoids; Daniel's call, build-time only). Flat fields on one class (seed, random_seed, and feature_seed collide, and section names carry meaning the flat space loses).

108. Prediction and TreeSHAP serve the resident Dataset; SHAP takes a crown (adopted)¶

Status 2026-08-25: the oblivious half of the decline recorded here is lifted by decision 111; multiclass still declines.

Status 2026-09-04: the additivity watch item below is reached earlier than recorded: decision 111's L40S session measured the worst fp32 residual at 1.0e-5 depthwise and 7.8e-6 levelwise at 200 trees depth 10, not 500 trees depth 8. The escape is unchanged: pass the raw matrix and the fp64 host walk runs.

Decision. Every X-taking Model method accepts a Dataset, and the dispatch picks the strongest route the Dataset supports: the model's own cuts route bin space (exact, no raw rows, DLPack builds included), foreign cuts fall back to the retained host matrix, neither raises with both remedies. Width-1 dense-tree models additionally route two device planes: whole-ensemble predict (a route_add generalization over the resident bins, plan receipt keyed on a booster mutation epoch) and TreeSHAP (path decomposition into 8-byte u8 bin-interval elements evaluated by a division-free closed form, thread per row-path, fp32 walk under a double-precision host epilogue). Oblivious and multiclass models decline to the host bin walk. Design record: architecture/22.

Measured (RTX PRO 6000 Blackwell Server, same pod, interleaved arms, xgboost 3.3.0 on cuda:0 confirmed via save_config). The gpu-shap axis (500 trees, SHAP over the held-out test matrix per protocol, 2 repeats, sha ffce72d): bonsai beats xgboost's GPU engine in every cell, 0.57s vs 2.38s at 1M x 128 d6 (4.2x), 1.34s vs 5.61s at 4M x 128 (4.2x), 0.84s vs 2.57s at 1M x 512 (3.0x), 2.46s vs 7.46s at 1M x 128 d8 (3.0x); context arms: CatBoost CPU SHAP 3.2-8.4s, LightGBM CPU SHAP 173-378s with the d8 cell timing out at 900s. A deeper hand ladder over the full training matrix at the same shapes: 2.32s vs 11.44s, 8.59s vs 46.25s, 10.43s vs 44.02s (4.2-5.4x). Additivity residuals same order both arms (bonsai 8.2e-6 to 1.38e-5, xgboost 3.8e-6 to 5.8e-6); bonsai's closed form is exact in exact arithmetic where xgboost's QuadratureSHAP (its 3.3 replacement for GPUTreeShap) is a quadrature approximation, and the fp32 gap against bonsai's own fp64 host walk measured 7.9e-6 at 200 trees. Predict from the resident Dataset 0.011-0.038s against xgboost inplace_predict 0.40-1.75s from host numpy; that column measures the resident-loop workload (xgboost pays the host-to-device movement per call, which is the cost this design removes), not a kernel-versus-kernel claim. Device parity held on both sm_87 and Blackwell: predict bit-equal after spelling the fma rounding out, SHAP within two orders of its tolerance. Host-side wins ride along: the per-row bias walk deleted (leafwise pred_contribs -30% on a Mac at 300k x 32 x 200) and the oblivious re-densify cached per epoch.

Consequence. The select-then-refit loop runs end to end on the resident bins: bin once, fit, eval, explain, filter outside, refit, with the only recurring host traffic being the results. The u8 bin compare and the epoch-keyed plan cache are edges no competitor algorithm swap removes: xgboost dequantizes its ellpack per read and recompresses its model every ShapValues call. Watch items, recorded not hidden: the fp32 additivity residual grows with tree count and crosses 1e-5 at 500 trees depth 8 (the escape today is passing the raw matrix, which runs the fp64 host walk; an fp64 kernel instantiation is the fix if a workload needs the device at that fidelity), and the v1 kernel geometry left shared-memory row staging, atomic-contention mitigation, and K=32 register pressure unpriced. The gpu-shap axis publishes the category with a fidelity column beside the throughput race.

Rejected. Thread-per-row iterative DFS (local-memory bound, quadratic in depth, priced in architecture/22); warp-per-path GPUTreeShap geometry at bonsai's merged path lengths (shuffle machinery to parallelize six iterations, lanes idle); a rows= filter parameter (filtering happens outside in numpy against cheap predictions); a device predicate mini-language (same reason); folding the bias into the kernel (it is one fp64 scalar per model).

Standings: gpu-shap

A per-member call-site census of Dataset's nine change reasons, production separated from tests, preceded the cut (the census itself was working material, not kept). The cut: BinStore owns the binned matrix (columns, plane, lazy host bins, width flag, the row mirror) and the cuts, because bins are unreadable without them; Dataset remains the composite consumers hold: labels, weights, and which rows, over a shared_ptr<BinStore const>. The name does not invert: the census showed every production consumer reads bins and labels through one object, and the Python Dataset plus 175 doc mentions make the composite the public noun. Forwarders keep all existing call sites and keep bin_at header-inline.

The behavioral half re-keys the caches. Bins key by the store's address, corroborated by geometry, and on the adopted-plane path the key holds the plane alive, so that address cannot be recycled under it. Labels and the resident arming key by MINTED tokens (LabelsId per labels block, FitId per fit specification: new for every factory product and every view, shared by copies), never by address: a first cut keyed labels by the Meta block's address and review round 3 showed allocator reuse re-creates exactly the trap being removed, one level down, and that the host-side resident_train_ pointer had the same defect with deterministic stack-slot reuse. Tokens are monotone and zero is never minted, so equality means identity with no reuse caveat: a view shares its parent's LabelsId (skips the label re-upload) but carries its own FitId (re-arms the resident state, whose row list is per-fit).

Measured: interleaved same-machine A/B (B,A,B,A blocks, 2M x 32 x 150 iters), populate min 3.45s on both arms, finalize 0.49-0.53s on both; the forwarder hop is paid per column because the fills hoist their spans. Wire identity 55c6fe308852d9bb unmoved. Rejected: renaming the store Dataset and the residue Fit (inverts the public noun against every consumer's read pattern); moving the row view to a grow() argument (churns the IBooster seam for zero census evidence); std::function in the mirror's mint seam (type erasure on a minting path for one caller).

111. The device plan input owns what it lends (adopted)¶

Decision. DevicePlanInput, the one seam both device planes read, carries an optional owner beside its span<DenseTree const>. A dense booster lends a view of its own ensemble and leaves the owner null; a levelwise booster attaches the epoch-cached dense equivalent that its host TreeSHAP path already builds, and points the span at that. Nothing else changes: the packers take DenseTree whichever grower produced it, and they copy and upload at pack time, so the owner has only to outlive the pack call, which the caller's own local already does. Levelwise width-1 models therefore take device predict and device TreeSHAP from a resident Dataset with no change above the seam. Multiclass keeps declining, through the empty default and through the width gate above it.

Why it was ever declined. Not for a kernel reason. dense_equivalent has always expanded an oblivious tree into the shape TreeSHAP's cover-weighted walk is written against, which is how levelwise pred_contribs works on the host. What blocked the device was purely lifetime: the seam's contract was that the span stays valid until the booster mutates, which a booster lending its own vector can promise and one converting into a cache cannot. Recording the reason matters because the shape recurs: a borrow-versus-own mismatch at a type-erased seam reads like a missing capability from either side of it.

Consequence. Densification mints a perfect tree, so a depth-d levelwise tree packs 2^d paths whatever its live coverage. An oblivious tree splits one feature per level and may split the same feature at two levels, so the expansion contains corners asserting f <= t_i and f > t_j at once: the dead slots are unreachable by construction, not merely unvisited, and a measured fixture has 106 of its 256 leaves dead with no input of either distribution routing to one. What the device therefore meets is the unsatisfied case, a zero-cover element in a path the row does not follow, and it meets it constantly. The two walks reach it differently: the host form divides by the cover fraction and so has to guard the branch off, while the device closed form only ever multiplies by it and needs no guard. That they agree anyway is what the parity fixtures check, and a host case pins the structural claim so the device fixtures cannot quietly stop carrying dead leaves. Correctness is device-verified on the Orin, and the throughput question the 2^d raised is now measured and answered the other way. Same L40S, arms interleaved, 1M x 128 at 200 trees, TreeSHAP over the held-out matrix, the levelwise device path against the released wheel where it declines: 22.0x at depth 4, 50.8x at 6, 75.2x at 8, 83.4x at 10, the ratio growing with depth. Levelwise on the device also beats DEPTHWISE on the device from depth 6 up, 5.40s against 10.24s at depth 10, which is the opposite of what packing 2^d paths predicts. The cause is the structure that mints the dead slots: an oblivious tree repeats features across levels, a path merges one element per DISTINCT feature, so its merged paths are short where a depthwise path can carry one element per level. Merged length picks the kernel template and the polynomial degree, so levelwise packs more paths and pays less for each, and the per-path saving wins. The same shortness shows up in fidelity: worst additivity residual 7.8e-6 against depthwise's 1.0e-5 at depth 10, which is decision 108's watch item reached at 200 trees rather than 500. The control is depthwise, which takes the device path in both arms and reads -0.1% to -1.0% across the cells that resolve, so the levelwise effect is not pod drift. One host-side cost is charged here rather than left to be discovered: the plan input stopped being free for an oblivious booster, so predict_on_device and contribs_on_device now ask for it only after the gates that do not need it, or a levelwise model predicting from a host-binned Dataset would build and retain a dense ensemble it never packs.

Rejected. Making the input own always, with dense synthesizing a non-owning aliasing shared_ptr (dresses a borrow as ownership and churns every consumer for a distinction none of them reads); a variant of span and owner (forces visitation at each read site, and the owner is a lifetime anchor rather than state a reader branches on); pruning the dead paths at pack time (a row can route into one, and the contributions must still sum to the prediction).

112. Benchmark data is drawn in fixed blocks, and the recipe is numbered (adopted)¶

Decision. bonsai.bench.synth.gen_data draws its rows in N_BLOCKS fixed contiguous blocks, one spawned SeedSequence stream each, over a thread pool; a stream of its own carries the whole-matrix draws (the informative-feature choice, and the sigma the noise is scaled by). The block count is a constant in the module, never a core count, so the same cell produces the same bytes on a 27-core pod and a 128-core one; the worker count is sized from runlog.cpu_quota() and never reaches the output. Both properties are tests, not comments. DATA_RECIPE is 2, every emitted row carries it, and it is part of the BONSAI_BENCH_DATA_CACHE key.

Why re-anchor rather than parallelize bit-exactly. PCG64 exposes advance, so workers could jump to their block offsets and reproduce recipe 1 byte for byte. That version would hard-code how many raw draws numpy consumes per generated element, which is exactly the thing NEP 19 reserves the right to change: RandomState is frozen, Generator streams may move across feature releases. The archive's byte-identity already holds only within a numpy version, which is why every row records libs.numpy. A numbered recipe with a deliberate, recorded break is more honest than an implicit dependency that a routine numpy bump can shift underneath the goldens.

Measured (M2, 8 cores, interleaved arms, min of 3): 1M x 100 recipe 1 0.34s against recipe 2 0.08s (4.1x), 2M x 128 0.77s against 0.16s (4.7x). The block fill scales near-linearly because numpy's Generator releases the GIL in its bulk fill path and each block owns its own bit generator, so the per-generator lock is never contended: measured 1.86x at 2 threads, 5.57x at 8, flattening at 16 where the fill is memory-bandwidth-bound. What this buys is not the seconds: at the extreme cell the old recipe spent roughly 30 to 45 minutes of a 48-minute sweep drawing random numbers on one core, and a shorter sweep is a smaller window for a pod to die mid-run, which has cost a session once already.

Consequence. Recipe-1 and recipe-2 rows are not comparable and the protocol says so where the data is described. The registry does not gate on this: plane_digest covers src/, include/, and scripts/model_hash.py, so a recipe change staleness-flags nothing, and the discipline that keeps a results directory honest is landing the change with a full refresh rather than between refreshes. scripts/model_hash.py's frozen linear variant is untouched by design; the wire-identity gate is unaffected and 55c6fe308852d9bb is unmoved.

Rejected. Bit-exact parallelism via advance (couples the goldens to numpy's per-element draw count, above); deriving the block count from the host's cores (a 27-core pod and a 128-core pod would then generate different data for one cell, which is strictly worse than generating serially); processes instead of threads (the extreme cell's matrix is 64GiB and would have to cross a pickle boundary, and the GIL is already released where the time goes); blocking the train and test ranges separately to make the train rows independent of n_test (the old recipe did not have that property either, since y.std() spans both, and nothing reads it).

113. The refresh is re-anchored on one card and one draw per cell (adopted)¶

Decision. Three changes to the standings refresh, taken together because a full re-measure was already due and a comparability break is paid for once. The extreme axis moves to a single card. Its RAM floor was 320GB, sized to catboost's 196.3GB of ingest copies rather than to bonsai, which peaks at 66.7GB on the same 2^34-cell input; that floor forced a two-GPU draw at twice the rate. The floor is now 150GB, one 188GB card clears it, and catboost publishes an OOM beside xgboost's, which the protocol already treats as a result because capacity is a claim. The data cache is exported, and loads whole. It was refused entirely because it handed back memmaps, which fault pages inside fit() and so moved measured fit_s unequally across libraries, only some of which re-touch the raw arrays after ingest. A full read completes before worker() opens a timer, so the fit sees memory indistinguishable from freshly drawn arrays; a cell too large to hold twice declines to the generator, since the cache is an optimization and never a reason to run out of memory. The pod waits are bounded. measure polls for the DONE marker under a per-axis deadline and a consecutive-unreachable limit, and the create ladder is walked to a deadline with backoff instead of twice.

Why now. Every axis was stale after decision 111, so the release refresh re-measures all nine whatever else changes. Landing the recipe re-anchor (decision 112), the extreme axis's host of record, and the cache in the same sweep costs one break in comparability instead of three, and each of them alone would otherwise have had to wait for a refresh someone was already paying for.

Consequence. Rows before this refresh and after are not comparable, which was already true of the recipe change and is now true for one more reason on the extreme axis: its host of record is a 188GB single-card container rather than a 351GB two-card one, and catboost's row there changes from a finish to an OOM. The published extreme table keeps a throughput ratio, since lightgbm's 116.2GB still fits, so the axis reads as bonsai against a survivor plus two arms that cannot run rather than as bonsai alone. Cost falls roughly in half on that axis, and the per-worker regeneration the cache removes was the sweep's largest untimed cost after decision 112 halved it: on gpu-tall, twelve draws become one.

Rejected. Sizing the extreme cell to bonsai's own ceiling so every competitor OOMs (about 1.6 to 2.2x the current rows before host generation binds, and it costs the ratio: one number and three OOMs is a weaker table than a measured 7.2x against a survivor, and a cell chosen so competitors fail invites the objection that it was); keeping the 320GB floor and accepting the two-GPU rate (it buys nothing bonsai needs); a memmap-mode cache with a correction factor (the distortion is unequal across libraries, so no single factor exists).

114. The architecture directory is dissolved; claims are routed and linted (adopted)¶

Decision. docs/architecture/ (23 files, 3,264 lines) is deleted. A five-way audit measured it 76% cuttable, found four docs asserting the opposite of the shipped code (a storage design recorded as rejected that shipped as BinColumns; a 230-line deliberation over a four-axis dispatch that shipped as three; an API sketch with zero matching symbols; stale "not here" lists naming shipped features), and mapped 19 claims restated across 94 sites. Content now routes by the table in docs/STYLE.md "Where things live": conventions to CLAUDE.md, single-file constraints to comments at the definition site, cross-file contracts to docs/invariants.md under a docs-check lint that fails when a cited path or symbol stops resolving, rationale here, pedagogy to guide/ and learn/. This supersedes the "design rationale lives in docs/architecture/, told once" convention.

Consequence. The two homes that stay must be kept honest by machine: invariants.md by the resolution lint, this log by append-only banners (this entry adds four). Historical citations of architecture docs resolve via links pinned to the last commit carrying the directory. The docs corpus drops by roughly a third; the failure mode where a normal PR strands three or more restatements of one claim now fails docs-check instead of waiting for a reader.

Rejected. Moving architecture content into docstrings: no pipeline renders docstrings to the site, so the move would hide the content, and the comment convention is constraint statements, not essays. Consolidating into this log: entries 26 and 27 were stale in lockstep with the docs they ratified, so the archive is a record, not a maintenance surface. Keeping the directory with a staleness lint: the lint would have flagged most of the directory, which is the same verdict as deletion with extra machinery.

115. Levelwise honours monotone constraints by projecting its leaves (adopted)¶

Decision. The levelwise (oblivious) grower no longer rejects monotone_constraints. Enforcement is a weighted isotonic projection of the finished leaf table (src/monotone.cpp, project_monotone), not a veto during growth: the tree's structure is exactly what the unconstrained search would have chosen, and only the values at the bottom move. A levelwise leaf index is the bit vector of level outcomes, so constrained levels induce a partial order over leaves and free levels cut them into independent groups; pool-adjacent-violators along a linear extension of each group, weighted by the leaves' hessians, is the nearest table satisfying every constraint. Hessians are the right weights because weighted isotonic regression on the Newton step is the constrained minimiser of the same second-order objective the splits were scored under. Interaction constraints stay rejected: they constrain which features may share a path rather than how leaf values are ordered, and no projection expresses that.

Consequence. The guarantee is exact rather than approximate. Monotone functions are closed under addition and positive scaling and a boosted model is init + lr * sum(trees) with lr > 0, so per-tree projection makes the ensemble monotone with a worst violation of 0.0, measured on all three growers. Both planes get it for no kernel work, because the leaf table is built on the host in the CUDA path too and only then uploaded, so the projection lands before finalize_leaves. Unconstrained fits are byte-identical: scripts/model_hash.py holds at 55c6fe308852d9bb. Priced against each grower's own unconstrained baseline at 200 rounds, lr 0.05, depth 6: California Housing with +1 on MedInc costs depthwise 1.73% RMSE (corroborating decision 35's recorded ~2% for the veto scheme) and levelwise 3.38%; a synthetic whose truth really is monotone costs depthwise 1.90% and levelwise -0.06%; a deliberately misspecified constraint whose truth runs the other way costs 518% and 546%. Projection is free when the constraint matches the data and costs a little more than the veto when it fights it, which is the trade the two mechanisms make in opposite directions: the veto declines splits, the projection moves values.

Two limits are stated rather than hidden. With two or more constrained features the leaves form a partial order and the projection runs along one linear extension, so every constraint holds but the result is not the L2-nearest monotone table; a single constrained feature is exact. CatBoost's SymmetricTree path accepts the same approximation. And leaf renewal (MAE, Huber, Quantile) overwrites leaf values after the grower returns, so levelwise projects a second time over the renewed table with row counts as weights. Measuring that turned up a pre-existing defect: the node-splitting growers have no second pass and do violate under those objectives today, tracked as issue #442.

Rejected. Rejecting the renewing objectives outright instead of projecting twice: the rejection would have to live in the booster, which is where the second projection lives anyway, so it costs the same and delivers less. Porting the projection to depthwise and leafwise: their leaves are an irregular partition with no lattice to project onto, which is the same reason CatBoost cannot offer the feature on their non-symmetric policies. Expressing the old rejection as a compile-time typelist filter, proposed by the retired dispatch doc: the rejection was never a property of the grower, only an unexamined gap, and decision 35 recorded the behaviour without a reason.

116. Monotone dispatch stays a value, not a type (declined by measurement)¶

Decision. monotone_constraints is checked at runtime, not dispatched on at compile time. No MonotoneObliviousGrower beside ObliviousGrower, and no monotone axis in the registry's typelist product.

Measured ceiling. The question is whether a templated grower could delete the guarded block in update_best_for_feature_for_node (src/split.cpp), which sits in the innermost split loop and runs roughly 32k times per node at 64 features and 255 bins. Patching mc to a compile-time zero hands the optimizer exactly what a <false> specialization would, so the resulting delta is the most any compile-time dispatch could ever buy. Same-host interleaved, 1M x 64, 100 rounds, 8 threads: depthwise 4.835s with the branch against 4.850s without it (-0.30%, inside its own 0.81% and 1.36% spreads), leafwise 6.164s both ways (+0.00%). The ceiling is zero. The branch is loop-invariant and perfectly predicted, which is the branch predictor doing for free what a template would buy at twice the binary. Levelwise never had a hot-path check at all: its projection gate runs once per tree, measured at +0.02% of train_s on an L40S.

Consequence. The registry stays at 7 objectives x 6 growers x 3 samplers, 126 booster instantiations; a monotone axis would double it to 252, doubling template compile time and the booster share of a 9.6MB CLI and 4.9MB extension, for a measured zero. The rule this case establishes, against sampler_traits as the contrast: compile-time dispatch pays when the property belongs to the TYPE, since AllRowsSampler reads no gradients under any configuration and the compiler can know it. A monotone direction belongs to the DATA, so the projection from a per-feature vector to a boolean has to happen at runtime whatever the grower's type is; a type parameter moves that check earlier and duplicates everything downstream rather than removing it. The registry is also keyed by name (dispatch.grower_name), and a monotone axis has no name to dispatch on.

Rejected alternatives. Reading the disassembly to decide whether clang unswitches the loop: that answers a proxy, where forcing the constant answers the design question directly and stays valid whichever way the unswitching goes. Templating the split finder alone on a bool Monotone, one axis on one small type rather than a grower axis on the cartesian product: still the correct shape if the branch ever costs anything, and recorded here as the reopening condition, but nothing to buy at the measured ceiling.

What would reopen this. A profile showing the monotone guard above the noise floor on a real workload, or a future constraint whose per-candidate work is more than one predicted branch.

117. +inf is a very large number, not a missing value (adopted)¶

Decision. BinMapper::transform bins +inf into the FLT_MAX top band, so NaN and only NaN bins as missing. This completes decision 74 rather than reversing it: 74 added the FLT_MAX closer to remove a train/predict routing skew, where a leaked row trained down the learned default_left branch but predicted right of every threshold by raw comparison, and it closed that skew for every finite value. +inf was the residual case, because it compares equal to the sentinel cut and lower_bound therefore returned the missing bin while the raw walk still routed it right. 74's title claim, that the missing bin is NaN-only on every path, is true only now.

The divergence, reproduced. One model, one row, two answers: predict on a raw numpy array returned 0.884 for a +inf feature (routing right, as inf > threshold), and the same row through a prebinned Dataset returned -0.505, bit-identical to what NaN returns. The prebinned path is a documented performance escape hatch, so a user who took the advice silently got different predictions. The binned-walk-bit-identical-to-raw-walk invariant asserted this could not happen; its probe generator skipped the closer and the sentinel, on the reasoning that one ulp above FLT_MAX is infinity and no fitted domain contains it. The probe now covers both, which is what makes the invariant pin its own claim.

Consequence. A feature value of +inf now trains and predicts as a very large number everywhere, which is the arithmetic every other path already assumed. Models change only where training data contained +inf, so the canonical hashes are unmoved (55c6fe308852d9bb). -inf was never affected: it sorts below the first cut, bins to 0, and routes left on both paths. The clamp is one std::min against the top band and costs nothing measurable on a path already dominated by the lower_bound.

Rejected alternatives. Making the raw walk treat +inf as missing instead, so both paths agree the other way: it makes infinity and unknown the same claim about a feature, which is false, and it would have required the raw walk to test for non-finite values on the hot path. Refusing non-finite features at the boundary: strictly more disruptive to callers whose pipelines emit infinities from a divide or an overflow, and it answers a question about semantics with an error message.

118. A config file states a key as strongly as `--set` does (adopted)¶

Decision. config::stated_keys collects the dotted keys an invocation names from both sources, the TOML file and the --set overrides, and the warm-start reconciliation treats them identically. Writing quantile_alpha = 0.5 in a config file against a model fit at 0.9 now refuses, exactly as --set objective.quantile_alpha=0.5 already did.

Why it was split. The reconciliation distinguishes a value the user chose from one that is merely sitting at its default, because the default triple is indistinguishable from silence by value alone. It learned which keys were chosen from the --set list, which carries key spellings, while a parsed TOML had already been flattened into a Config where a stated default and an unset field look the same. The result was invisible: the same value, in the same invocation, refused through one channel and silently inherited the model's through the other.

Precedence is unchanged. resolve still applies the file first and the overrides on top, last write wins, and stated_keys collects key names only, never values. A file saying 0.5 with --set saying 0.9 resolves to 0.9 and passes against a 0.9 model; the reverse resolves to 0.5 and refuses. The reconciliation compares the already-resolved config, so the two mechanisms never contend.

Rejected alternatives. Leaving the split as a documented asymmetry, on the reasoning that a file is a weaker statement of intent than a flag: nothing else in the config layer makes that distinction, and a user cannot see it. Threading explicitness through resolve itself by returning both the config and its stated keys: a wider signature on the one function every entry point calls, to serve the single caller that needs the extra return.

119. Predict walks are packs, and each tree family gets its own loop shape (adopted)¶

Decision. Batch predict no longer walks one tree at a time over the ensemble. Each tree family packs its trees once per booster epoch into a walk object (DenseWalk for depthwise and leafwise, ObliviousWalk for oblivious) and the pack chooses the loop shape its cost model wants. The contracts, bit-identity with the per-tree walk and the epoch-keyed staleness rule, live above the two classes in include/bonsai/tree.hpp; this entry holds the measurements and the shapes that lost.

Dense trees: lockstep groups, not loop inversion. A dense tree's cost is the chain of data-dependent node loads, not the loop shape, so inverting the ensemble loop to rows-outer alone buys nothing: 2231 vs 2206 ns/row on M2 (1 thread, 64 cols, depth 8, 100 trees). The lever is walking eight trees per row in lockstep so the core always has independent load chains in flight, which took the same cell from 2206 to 1004 ns/row at batch against xgboost inplace_predict at 1015. Leaves point to themselves and a leaf's threshold slot holds its value, so a walk padded to its group's deepest tree self-loops harmlessly and the sum reads the node the walk ends on. The padding tax is bounded by the group's deepest member, and groups bound their own depth, so leafwise's ragged best-first trees tax only their own group.

Oblivious trees: rows outer, 64-row blocks transposed. Every tree's level splits pack contiguous and every leaf table concatenates, so the walk runs rows-outer in one parallel section over cache-resident arrays instead of opening a parallel region per tree. Full 64-row blocks are flipped into feature-major scratch once, so each level step is one broadcast compare over 64 contiguous floats, which vectorizes at baseline SIMD widths with no gathers; the sub-block tail walks scalar. Row-wise blocking is the one vectorization the bit-identity contract permits, because each row's accumulator remains its own tree-order fold. Measured at 1 thread on the same cell: single-row 1053 to 610 ns on M2; the blocked batch walk reaches 181 ns/row on M2 and 179 on EPYC 9654, from 505 and 874 scalar, against catboost's same-pod 784. The two hosts converging says the block walk is memory bound, not ISA bound.

Rejected alternatives. A scalar baked-constant code generator, probed at 293 ns/row: row blocking beats that ceiling along the dimension the probe did not price, and the tree stays free of a code generator. An isnan test in the walk: the two comparison forms v > t and !(v <= t) agree on every non-NaN value and disagree exactly on NaN, so default_left selects the form and the test disappears (the header guards the IEEE assumption with a preprocessor check against -ffinite-math-only). Rows-outer inversion for dense trees, refuted above.

Reopener. A host where the blocked oblivious walk stops converging across ISAs, which would mean it has become ISA bound and a wider block or an explicit SIMD path is worth pricing; or a code generator that also blocks rows, which would need to beat 181 on the same pod.

120. A row view hands its ids to the device walks, and the eval plane adopts the device plane (adopted)¶

Decision. The scoring side of a Dataset row view goes to the device the way the training side already did. cuda_predict, cuda_pred_contribs and the resident eval walk take a std::span<row_id_t const> of view ids (empty meaning every plane row), the kernels read bins at the mapped plane row and write at the view position, and the two identity checks that sent a non-identity view to the host walks (ValidationScorer::arm_device, Model::device_ready) are gone. The staged map has one home, RowMap in src/cuda/detail/device_buffer.cuh, a nullable device pointer over the plane the way RowIndex is a nullable index over the host, and the three kernels share mapped_row. Separately, eval_begin adopts a validation plane that already lives on the device through matching_plane instead of retiling it through bin_at, and retiles only the view's rows when it cannot.

Measurement. Same-pod L40S, min of three interleaved reps, 4M x 64 train (device-resident), one 2M holdout binned against it, 100 rounds at depth 6, buckets from BONSAI_FIT_PROFILE and BONSAI_CUDA_PROFILE:

build	holdout arrangement	fit	eval-route	eval-loss	eval-arm	predict 1M
main	two Datasets	1.00s	0.02s	0.00s	0.12s	0.00s
main	one Dataset, `subset(rows=)` views	1.78s	0.67s	0.24s	0.00s	0.58s
main	views with `columns=slice(None)`	1.00s	0.02s	0.00s	0.12s	0.00s
this change	two Datasets	0.89s	0.02s	0.00s	0.00s	0.00s
this change	one Dataset, `subset(rows=)` views	0.89s	0.02s	0.00s	0.01s	0.00s
this change	views with `columns=slice(None)`	0.88s	0.02s	0.00s	0.00s	0.00s

The view arm's eval-route and eval-loss were the per-round host walk and host loss; its predict was the parent plane pulled home to mint the row-major mirror. All three vanish. The eval-arm 0.12s on the other two arms was the host retile of a plane already on the device, and adoption removes it from every CUDA fit with a device-resident eval set, which is the 11% the arms that were never slow gained. Predictions through a view, a copied Dataset and the raw array are bit-equal (max abs diff 0.0 over 1M rows); SHAP through the view differs from the copy by 1.4e-5, the per-path atomic accumulation order the SHAP kernel has between any two calls. Every run of this change printed cuda eval plane armed: ... adopted=1 and a cuda-predict: line with rows under plane on the view arm; no main view run printed either.

Rejected alternative. A range-offset row map for contiguous views, which would skip the ids upload for a slice. The upload does not show in the profile at 1M view rows (upload=0.000s on every cuda-predict: line above), so a second map form would be a concept without a measurement in which the plain form loses. Reopener: upload= becoming a visible share of cuda-predict: or cuda-shap: at some larger view, which would price the offset form against the ids.

Lifetime. The adopted plane is held by the eval seam until the next eval_begin or the context is torn down, the same lifetime the own tiled copy had, with one copy less on the device.

121. A renewed leaf is clamped into the fence the grower left for it (adopted)¶

Decision. The depthwise and leafwise growers carry each leaf's monotone fence out of grow() in GrowResult::leaf_bounds, indexed by node id, and the booster's renewal clamps every renewed value into it before writing the leaf. The levelwise grower returns no fence and keeps the leaf-table reprojection of decision 111. The invariant renewal-keeps-monotone pins both node-splitting growers for MAE, Huber and Quantile on an eight-row case where the Newton leaves are ordered and the renewed ones are not.

Why a clamp is exact. Growth leaves every leaf in a left subtree with hi at most the split's midpoint and every leaf in the right subtree with lo at least that midpoint, for each constrained split on the path, so any assignment of leaf values inside their fences is monotone. Each renewing objective minimises a convex one-dimensional loss over its residuals (absolute, Huber, pinball), and the minimiser of a convex function restricted to an interval is its unconstrained minimiser clamped into that interval. The clamp is therefore the fenced optimum, not an approximation of it.

Rejected alternatives. Recomputing the fence in the booster from the finished tree, which would copy the arithmetic of propagate_monotone_bounds into a second home and, under row sampling, run it over row sums the grower never saw. Storing the fence on DenseTree beside split_gains and covers, which are serialized, so a per-round datum would move the model format. Reprojecting dense leaves by pool-adjacent-violators over the leaf partial order, the levelwise tool, which solves a problem the fence already solves and would change values the fence does not need to touch.

Vocabulary. LeafBounds adds bounds to the public vocabulary (vocabulary_singletons 146 to 147). It is the word SplitInput's contract already uses for lo and hi; the nearest existing words name other things, Range a run of row positions and interval a logging cadence. Reopener: a second lo/hi pair type above the engine seam, at which point the two share this name.

122. The release A/B is anchored to one fixed wheel and a moved verdict needs a citing decision (adopted)¶

Decision. Every standings refresh fits three arms at the tall cell of each plane, interleaved on the pod that measures the plane: the previous release wheel, one fixed anchor wheel (1.15.0), and the commit under refresh. The statistic is the min over repeats. HEAD is read inside 5% of the previous release and inside 2% of the anchor. The rows are committed as ab-<plane>-<date>.jsonl, registered with the plane's tall axis so they supersede with it, rendered as the release drift section of the perf page, and gated: make docs-check refuses a moved cell until a decision entry tagged Standings: <axis> cites the file. The tagged entry is the one decision 103's gate already holds against the axis's as_of_decision, so one entry carries the explanation and the bump.

Why an anchor. The 2.1.0 refresh read a 1 to 2% loss on cpu-tall inside a band that is 5% wide, and a same-host Mac A/B could not resolve it. A two-arm A/B compares each release only with the last, so a loss under the band passes every time and the bands compound: 1.5% a release is 35% over twenty. Against a fixed wheel the comparison is cumulative, and the same drift shows by the second release. The 2% anchor band is the size the 2.1.0 session's min-over-reps spread on the gpu plane supports; the cpu plane's spread was wider than its band at 2 repeats, which is why cpu-tall takes 4.

Why a gate and not a warning. The driver already printed the verdict, and the 2.1.0 refresh shipped its moved cells on a reading in the PR body that the PR body alone recorded. A verdict recomputed from the committed rows by check_standings.py, in the lowest module so the driver, the renderer and the gate read one answer, cannot be read differently by the next refresh, and a citation in a Standings: entry is the form the standings already use to say why a number changed.

Rejected alternatives. A moving anchor (the release before last), which compounds at half the rate and hides the same drift over forty releases. Failing the refresh on a moved verdict, which would block a deliberate trade (a slower fit for a faster predict) that the decision entry exists to record. Keeping the A/B file out of the tree and gating on the PR body, which is what made the 2.1.0 reading unrepeatable. Reopener: the anchor wheel no longer installs on the pod image (a Python or CUDA floor moves past it), at which point the anchor advances to the oldest wheel that does and the band is re-derived from that session's spread.

123. The core tree stays array-of-structs; the structure-of-arrays probe is declined (adopted)¶

Decision. DenseTree::Node keeps its array-of-structs layout and no structure-of-arrays probe is run. The question came from the TreeSHAP paper, whose tree is six parallel vectors, and asked whether that layout would simplify split and grow.

Why. The paper's vectors and Node carry the same information grouped differently, and the tree already keeps covers_ and split_gains_ as parallel vectors beside the node array (include/bonsai/tree.hpp). Where an access pattern rewards structure-of-arrays the code already has it: the histogram grad and hess arrays on the hot path, the whole CUDA plane, and the oblivious per-level arrays. What reads as elaborateness in split and grow is orchestration (frontier transactions, buffer recycling, the histogram lifecycle, dispatch), and a layout change removes none of those responsibilities; it converts struct cohesion into index discipline across arrays tied together by convention. CPU predict is a single-row branchy traversal, where array-of-structs keeps a node in one cache line and the alternative scatters it across five. The node layout is frozen into the model format, so a change ripples through serialization, SHAP, the CLI, and the fixtures. The 2026-08-27 core design review reached the same verdict for HistCell: the fill and the scan touch both fields, so splitting them buys nothing.

Rejected alternative. The one-afternoon probe the issue pre-registered: a structure-of-arrays DenseTree behind the unchanged interface, measured on core LOC, mean cyclomatic complexity, and single-thread predict throughput, with the prediction "LOC flat, CCN flat, predict a few percent slower". Declined because no measured split, grow, or predict cost has ever been attributed to node layout, so the probe would price a rewrite against a prediction no evidence contradicts. Reopener: a profile bucket attributes fill, split, or predict time to node-field gathers, at which point the probe runs as the issue wrote it, against the ledger baselines of that day.

124. Device histogram cells are int64 fixed point, so a device fit is bit-reproducible (adopted)¶

Status 2026-09-05: the CPU-finder move to double totals named under "Why fixed point" is declined in decision 125; the host finder keeps float node totals.

Status 2026-09-05: measured on an RTX PRO 6000 Blackwell (sm_120, driver 595.91), main before the change and after it interleaved on one pod, min over three reps. The two-word shared add lowers to native ATOMS.ADD there as on Ada (36 ATOMS.CAST.SPIN before, 72 ATOMS.ADD after), so the reopener is not tripped. Tall 16M x 128: depthwise 4.870 s to 4.282 (-12.1%), leafwise 7.336 to 6.614 (-9.8%), levelwise 4.837 to 4.294 (-11.2%), the populate bucket -25 to -28%. Wide 131072 x 16384: depthwise 28.214 to 27.442 (-2.7%), levelwise 30.237 to 30.687 (+1.5%), leafwise 87.678 to 89.500 (+2.1%). The wide cells spend 22 to 84 s of the fit in find, and the finder's added cost is the per-bin int64 to double dequantise (I2F.F64 count 1 to 5 in find_kernel, 0 to 4 in level_find_kernel) on a part whose fp64 pipe runs at 1/64 of fp32 rate; the fill is faster there too. Every rep agreed in sign at under 0.5% spread; the before arm printed up to three r2 per wide cell, the after arm one. Three cuda_depthwise fits hashed e1a5391a, the L40S value, which is an observation and not a contract. The 2.2.0 refresh prices the wide cells against the anchored band; the lever if they trip it is fp32 gain math from the int64 prefix split into 32-bit halves, which changes the device model and is priced as its own decision.

Decision. The three CUDA fill kernels accumulate gradient and hessian in int64 cells rather than float. Once per tree and per component the fill reads M = max|g| * n_rows with a warp-reduced fabsf and one atomicMax on the float bit pattern (non-negative floats order as unsigned integers and max is order-free, so the reduction is exact), and derives one power-of-two scale 2^e with e = min(61 - ilogb(M), 126); each visit adds __float2ll_rn(v * 2^e). Any subset of the tree's rows sums to under 2^62 + n/2, so every partial, prefix and sibling difference fits int64 in any order, and the readers dequantise the prefix sums once, by one multiply, before the gain math. The contract flips from "tolerance-equal, never bit-exact" to cuda-training-bit-reproducible: three fits of one dataset serialize to identical bytes on the depthwise, oblivious and leafwise planes, with the device objective and with the host objective forced. Not claimed: cross-device or cross-toolkit identity (the gradient kernels use device transcendentals and compile with contraction on), and host-vs-device identity, which stays at 1e-4 because the host accumulates float cells.

Why fixed point. A cell is the exact integer sum of the node's quantised rows, so the chunk count (which follows sm_count), block scheduling, atomic landing order and plane choice all give the same integer. The per-row error is one rounding at max|g| * n / 2^62, which at 16M rows is 2^-38 max|g|; the float shared stage it replaces rounded every add at the running partial's ulp. Parity error is now dominated by the host's own float HistCell, so the 1e-4 host-vs-device bound stays and the leafwise flake at 1.06e-4 (one run in five, the tail of the old spread) can only shrink. The new plane test (65536 rows, min_data_in_leaf = 4) then failed at 2.4e-4 on main as well as on the fixed-point cells, and the cause was a second float on the device path: node totals crossed the host as float HistCell and were re-uploaded as the parent totals the find kernels subtract a left prefix from, so each right-child edge re-read a total rounded one level up. Totals now travel as a double NodeTotals; the CPU finder still reads float totals from its own cells, so CPU model bytes are unchanged, and moving it to double is a separate change because it moves the hash. The tile plane's shared footprint doubles (8 * stride int64, 32640 bytes at 255 bins where the float tile took 16320), so its ceiling moves from 768 to 384 bins, above which a feature takes the per-feature plane; that plane holds one int64 replica where it held two float replicas, the same bytes, so the 4096-bin case still fits the 48 KiB static budget.

Measurement. One L40S, main and branch interleaved on the same pod, the 16M x 128 ledger cells at 255 bins, depth 8, 100 trees, min over three reps (two for the 511-bin cell); seconds, hist is root_hist + adv_hist from cuda-round-decomp. The first form, a native 64-bit shared atomicAdd, missed the pre-registered bar (fit +13 to +17%, hist +26 to +30%): the SASS showed it lowered to ATOMS.CAST.SPIN.64, a compare-and-swap loop, where a 32-bit integer add is one native ATOMS.ADD. Two levers took it: the shared stage adds each cell as two 32-bit atomics with the carry taken from the returned low word (root fill 1.89 s to 0.68 s), and the tile block grows to 512 threads so three 32 KiB blocks keep 1536 threads resident (0.68 s to 0.60 s). The branch then beats main on every cell:

cell	arm	fit	train	root_hist	adv_hist	hist	r2
depthwise	main	9.379	8.163	1.09	3.88	4.97	0.8793, 0.88
depthwise	branch	8.597	7.411	0.60	3.62	4.22	0.8797 x3
leafwise	main	13.020	11.729				0.88
leafwise	branch	12.034	10.569				0.8797 x3
levelwise	main	8.704	7.481	1.09	3.23	4.32	0.877
levelwise	branch	7.953	6.760	0.60	2.99	3.59	0.877 x3
depthwise 511 bins	main	11.710	10.184	1.35	5.05	6.40	0.8796
depthwise 511 bins	branch	10.754	9.376	0.86	4.74	5.60	0.8796 x2

Fit total -7.6 to -8.6%, hist -12 to -17%. Main's r2 moved between reps on the depthwise cell (0.8793 and 0.88); the branch printed one r2 per cell on every rep. Three cuda_depthwise fits hashed 17d29e48, 982c3d37, 525539e2 on main and e1a5391a three times on the branch, with the [cuda] suites at 350305 assertions passed. The engagement line fixed-point scale_g=2^{e} appeared in every branch run and in no main run.

Rejected alternatives. Widening the parity bound, which absorbs the tail of a distribution the test was written to bound. int32 shared cells with a second scale: chunks reach 2^18 rows, so a shared-only int32 scale leaves 12 bits of per-row resolution against float's 24, plus a requantisation at the merge. A deterministic reduction library (decision 40 declined cub, and it fixes reduction order, not scatter associativity). A native 64-bit shared atomicAdd as the shipped form: it was the first form and lost to the CAS loop above, so the two-word add stays as long as the SASS shows ATOMS.ADD for it. Two levers were left unpriced because the bar was cleared before them: a width-4 tile instantiation for stride > 512 and k_fill_blocks_per_sm 4 to 3. Reopener: a device where a 32-bit shared integer add is not one native instruction, or a cell where the fill loses to the float tile at the same shared bytes, at which point the two unpriced levers run first.

125. The CPU finder keeps float node totals; the double-totals move is declined (adopted)¶

Decision. Histogram::totals() keeps returning a float HistCell, cell_totals keeps handing the split finder a float pair, and the CPU model hash stays at 55c6fe308852d9bb. Decision 124 named the move to double as a separate change because it moves the hash; it was priced on 2026-09-05 and does not pay.

Why. The device bug that decision 124's NodeTotals fixed was a chain: the device planes derive each right child as the parent total minus the left prefix, so one float rounding at the root reached a four-row leaf 2.4e-4 off. The host finder has no such chain. Each node's total is summed in double from that node's own histogram and rounded to float once, the scan accumulates left_grad in double, and the right side is that node's own total minus its own prefix, so the only float in the node math is one rounding of the node's own total, a relative 6e-8 in its score. A laptop probe made the total double end to end (a totals_exact() sibling on Histogram, SplitInput::totals() reading it, NodeTotals through cell_totals and both finder entry points) and compared predictions against main on two CPU cells, depthwise, 255 bins, 8 threads, 200k held-out rows:

cell	main r2	double totals r2	rows whose prediction moved	max abs move
500k x 100, depth 8, 20 trees	0.9409917	0.9409934	31325 of 200000	3.3
2M x 64, depth 12, 100 trees	0.9881329	0.9881342	56550 of 200000	3.8

The shape of the delta is the verdict: a mean move near 1e-4 with a maximum over 3 means a handful of near-tie splits changed rank and rerouted their subtrees, which any ulp-level change to a gain does, and the r2 movement (1.3e-6 and 1.7e-6) sits two orders of magnitude under the 0.001 chance band the quality work treats as threshold-placement luck. The move would spend the hash gate that every PR relies on, and rewire a few dozen splits per model, to buy precision the float histogram cells do not carry.

Rejected alternative. Shipping the move anyway for type consistency, so that one NodeTotals flows through both finders. Consistency at the seam is already there: SplitInput::totals() returns NodeTotals on both planes, and the float narrowing is one function, cell_totals, at the host finder's entry. Reopener: a host-plane cell where a float node total is shown to reach a leaf, which needs the host to adopt a parent-minus-left chain, or a host-vs-device parity failure attributed to the host total rather than to the host's float cells.

126. Twenty-seven levers on the wide CUDA cell: block-parallel small fill and reduce, a blocked mapper-fit gather, fill-targeted slot zeroing, the sibling subtraction fused into the finder, a level finder that sweeps cuts instead of nodes, a mapper fit that sorts its gathered block in place, finders that spend fewer fp64-pipe ops per cut, an fp32 interval screen ahead of the node finder's fp64 scores, a prefetched cut chunk, a screen that divides and converts on the fp32 pipe, a node finder that derives the large sibling in its own sweep, and a screen that reduces on the certified bound, compares in fp32, takes its node constants from the host and its inverse scale from the quantiser, a node finder that prefetches its strip into L2 ahead of its sweep, a tile fill whose shared histogram spreads each bin over all 32 banks, a tile sixteen features wide filled by 1024 threads, a partition whose warps own contiguous rows and rank by ballot, leafwise row segments that keep their buffer side, one partition kernel with a decoupled look-back, a count plane for the unit-hessian tile fill, the small child's fill queued behind the leaf partition with its segment chosen on device, and the leaf round's arguments riding the launch with its best split delivered by event; the wide fit drops 63 to 90% and the tall train 22 to 31%, and the same screen over the level finder, the approximate divide, two cuts per lane, the occupancy bounds, 64-bit shared atomics, the wide tile without its launch bound, a root bulk zero of the leafwise pool and a 16-byte slot zero are refuted (adopted)¶

Decision. Twenty-seven behaviour-preserving changes, one commit each, the seventh replacing the sixth's code, the eighth widening the third's, the fourteenth taking the fifth's derivation into the finder's sweep, the fifteenth through eighteenth cutting what the node finder's screen still spent on the fp64 pipe and per block, the nineteenth issuing the strip's L2 prefetch ahead of the sweep, the twentieth laying the tile fill's shared histogram out as word planes, the twenty-first widening that tile to sixteen features under a 1024-thread launch bound, the twenty-second giving each partition warp 512 contiguous rows and folding the segmented scan into the scatter, the twenty-third letting a leafwise split read its parent's rows from whichever buffer holds them and leave the children where it scattered them, the twenty-fourth replacing the partition's route count and scatter with one kernel whose blocks claim tiles in scheduling order and read their prefix from the tiles before them, the twenty-fifth filling the hessian plane with a row count when the resident objective is unweighted MSE, the twenty-sixth queueing the small child's fill behind the leaf partition, its segment written by the partition's last tile and its row count read through mapped memory after an event, and the twenty-seventh passing the leaf round's partition op and finder nodes as kernel parameters and delivering the leaf's best split through mapped memory after an event. partition_kernel's last tile writes the small child's segment, the fewer rows of left and right with their offset and the fresh slot the host chose before the launch, into a three-word device record (SmallChildDev); when the popped leaf's children lie below max_depth, leaf_split enqueues that slot's zero and its fill (leaf_enqueue_fill) behind the partition, records an event (StreamFence) and waits on it with the left count read from mapped pinned memory (MappedBuffer) instead of a blocking copy, so leaf_build is a no-op for that round and the stream no longer drains between the partition and the fill; hist_tile_kernel takes the direct-global branch fill_direct for a node under 512 rows, the fill hist_small_kernel runs, so the small-node choice moves on device with the segment. partition_kernel and find_kernel are templates on an accessor type: the level path passes pointer-form accessors over its device tables (PartOpTable, FindNodesRef), the leaf path passes value-form accessors built on the host stack (PartOpValue, LeafFindNodes) that the launch copies into the kernel's parameter space, so the five pinned uploads a leaf round staged before its partition and finder could launch (the op, the sums and bounds, the screens, the slots, the sibling derivation) are gone; LeafFindNodes selects between its two nodes with a compare rather than a dynamic index, which keeps the parameters in constant memory. The second find_kernel instantiation cost sweep_strip its single caller: LLVM outlined it and its NodeCut and ScreenNode arguments crossed a 256-byte local frame, 91 STL per kernel on sm_87 against 6, so the finder chain from screen_node down is forced inline and each instantiation is one 22k-line body with 6 to 9 STL, the shape of the single finder it replaces. The leaf's reduce_kernel writes its best split into mapped pinned memory (MappedBuffer, which also carries the left count) and the host waits on an event for it instead of a synchronous cudaMemcpy into a pageable vector. hist_small_kernel fills each node under 512 rows over the tile kernel's block geometry, one block per (feature tile, node), instead of one block per node. reduce_kernel picks the best of the per-feature candidates with a two-stage block reduction ordered by first_max_better (larger gain, then smaller feature slot on an exact tie) instead of one thread walking the row. BinMappers::fit gathers eight adjacent columns per row pass with the row sixteen ahead prefetched, instead of walking each column down the row-major matrix alone. zero_slots_kernel zeroes only the histogram slots a fill will land in (the small slot of each level triple, the fresh slot of a leafwise split) where the level advance zeroed every child slot and the leafwise root zeroed the whole pool; the large slot is never read before it is derived. The two subtract kernels are gone: each find block writes the large strip it is about to read, parent minus small, before its scan, from a per-node SiblingDerive table the level plane stages with its triples and the leaf plane resolves from the pending pair leaf_build records; advance_level therefore leaves a level complete only once the find that always follows has run on it. A level too narrow to hide that work (fewer than k_derive_blocks_per_sm find blocks per SM, 4) derives with a warp-per-strip kernel ahead of the finder instead. level_find_kernel runs 256 threads per feature with a thread per cut: for each node the block reads the strip's 256 consecutive cells in one coalesced sweep, forms the per-cut prefix with a warp shuffle-up scan plus cross-warp totals (LevelPrefixTotals, double buffered by node parity, one barrier per node), scores its cut, and keeps the per-cut sum over nodes in registers, with the per-node scalars staged 256 nodes at a time into shared memory; one warp per feature with a lane per node had walked every cut serially, each deep-level load touching 32 node strips 2 * stride cells apart. The fp64 sum over nodes visits them in bit-reversed order within each batch of 32 and folds them by ShuffleTreeSum, a binary-counter merge of five partials that adds the same pairs in the same order the old shfl_down tree did, inactive nodes contributing +0.0 as the idle lanes did. The mapper fit gathers 32 columns per row pass into one block whose column streams sit rows + 32 floats apart, drops a NaN with an unconditional store and a conditional advance, and hands each column to BinMapper::from_sample as a span it sorts in place; the radix sort takes all four byte histograms from its key transform pass and skips a scatter whose byte every key shares, and the run lengths are written as run ends with no branch, where the third lever's gather had pushed eight columns into growing vectors that from_sample took by value and sorted with a count pass per byte. The finders spend fewer fp64-pipe ops per cut, every one bit-identical to the old arithmetic: split_sums_dev selects between pg and pg + miss instead of adding a selected 0.0 to every sum (x + 0.0 == x for every x but -0.0, and no gradient or hessian sum reaching it is -0.0: the cells are integers times a positive scale, real - p with equal operands rounds to +0.0, and the root accumulator starts at +0.0); cut_score<k_l1> specialises lambda_l1 == 0, the default and the ledger setting, so the two-compare soft threshold becomes an integer-pipe NaN test that keeps the NaN-to-zero semantics, dispatched once per launch in the find_kernel and level_find_kernel wrappers so no launch site changes; ShuffleTreeSum::push walks the binary counter with an early return instead of five predicated adds (31 DADDs per 32 pushes where there were 160); level_pass_sums adds the tree total only on the lane that holds it and guards the second direction on n_dirs, so a feature with no missing bin skips that direction's two divides. bounded_leaf_weight is untouched, since a leaf value keeps the sign of its zero. find_node bounds every (cut, direction) score in fp32 before it scores it in fp64: directed-rounding intrinsics (__fadd_rd, __fmul_ru, __fdiv_rd and their partners) on the sums, the squares and the quotients give an interval that contains the exact fp64 score, the warp keeps a running certified lower bound l_star, and a cut whose upper bound falls below l_star, or whose child hessian surely misses min_child_hess, never reaches the fp64 pipe; the survivors are compacted per lane with a ballot and a shuffle so the fp64 scoring sees a dense list. The screen is a proof and not a heuristic, since an interval that contains the exact value can only drop a cut that cannot win, and it disarms itself (everything survives) when a node sum is not finite, when float(inv) is inexact, or under BONSAI_CUDA_FINDER_EXHAUSTIVE=1, which the test uses as its oracle; a mutated screen that drops at 2 * l_star fails all six sections. sweep_cuts loads each lane's cut as one 16-byte longlong2 (a strip is (g, h) pairs, so every cut pair sits on a 16-byte boundary, pinned by a static_assert on the pair width) and issues the load for the next chunk before it scans the current one, so the load's latency overlaps the two shuffle scans instead of heading them. The screen's directed quotients come from a plain fp32 division bracketed by one float step in each direction (the operands are non-negative and the round-to-nearest quotient lies within half an ulp of the true value, so the next float up and down bracket it, infinity returned as is and the zero quotient floored at zero), and its int64 cells reach fp32 as a signed high int32 and an unsigned low uint32 recombined by one directed fma (hi * 2^32 is exact in fp32 for any int32, so the fma's single rounding is the only one and the bound keeps its side); __launch_bounds__(32, 16) pins find_kernel at 128 registers. The finder dispatches the sibling blocks of one feature adjacently (find_grid is 2 * n_sel by ceil(n_nodes / 2), node = 2 * blockIdx.y + (blockIdx.x & 1), the root's odd block returning), and the large sibling's block forms parent minus small per longlong2 chunk in registers during its sweep, storing the difference only where a later reader exists (always in leafwise, whose slot outlives the find, and in depthwise only while a next level derives from it, tracked by a depth counter on the level pipeline), with the store test a template argument so the sweep loop carries no runtime branch; the bins past the sweep and the strips of nodes that do not sweep are derived by a plain strided loop, and the separate derive pass of the fifth lever is gone from the node finders (the level finder keeps derive_level_strips). warp_max_nonnegative folds the warp's certified lower bound with one REDUX.MAX.U32 on the float bit pattern (a non-negative float orders as its unsigned bits, and the bound is floored at zero) instead of a five-step shuffle tree. screen_node widens each node's thresholds once into a directed fp32 Interval (the score to beat, min_child_hess, min_gain, each end rounded outward) and screen_cut compares in fp32 against the conservative end, certifying against the rounded-up threshold and dropping against the rounded-down one, so a cut in the ulp between the two ends falls through to the exact path; that replaces six F2F.F64.F32 and six DSETP per cut per direction. The per-node screen constants come from the host: a NodeScreen per node (score, and the outward-rounded fp32 brackets of the score and the node sums) is filled where the node sums are already staged, a ScreenConst of the four config brackets rides as a kernel argument, and the device keeps only the brackets that depend on the histogram; the host score is the CPU finder's score(g, h, l1, l2), bit-identical to the device cut_score. Winning splits compare by their int64 bit pattern (split_better, key_of_positive_gain), since a gain is stored only when positive and __double_as_longlong is order-preserving there. GhQuant carries the fp32 inverse scale beside the double (inv_f, written once per tree by gh_quant_kernel), because SASS showed the compiler rematerialising float(inv) as two F2F.F32.F64 per 32-cut chunk inside the sweep. find_node issues prefetch.global.L2 over the whole (node, feature) strip right after it opens it, before the miss load and the sweep: each lane prefetches one 128-byte line, so one warp instruction covers 4 KB and a loop covers longer strides, for the cells and the small strip alike, and the sweep's chunk loads then hit L2 in place of a chain of eight DRAM misses per block. hist_tile_kernel keeps its shared histogram as four planes of 32 words per 32-bin group (gradient low and high word, hessian low and high word) in place of adjacent int64 cells, so the 32-bit atomics of bin b hit bank b mod 32 where they hit bank 4b mod 32 before, eight bins to a bank; the plane offsets are constants the compiler folds into the atomic's address immediate, so one base pointer per feature stays live, the tile is padded to a multiple of 64 cells per feature (tile_stride), and the merge reassembles each cell from its two words before the global add, so the global slots keep their layout. hist_tile_kernel<16> visits sixteen features per row from one 16-byte strip load, so each row's strip load, gradient load and two quantise ops serve twice the features; its 64 KiB shared tile is one block per 100 KiB SM, and that block runs 1024 threads under __launch_bounds__(k_tile_fill_threads), which holds the kernel at 64 registers with no spills where the 72 it takes unbounded cannot launch 1024 threads at all; launch_hist admits the tile against the device's opt-in shared limit, which init_shared_limit raises for the tile kernel with the attribute call the per-feature kernel already had. route_count_kernel gives each warp 512 consecutive rows with lane l on row j * 32 + l (part_lane, part_row), loads the sixteen row ids into registers before any bin gather so the sixteen gathers are in flight together, stores the flag bytes coalesced, sums the flags by warp shuffle (warp_sum_u32) and reduces the eight warp sums through shared once (warp_prefix_in_block), so a block writes one count where each thread had walked sixteen consecutive rows and every warp instruction touched 32 sectors; scatter_kernel derives its segment's prefix and total from the op's block counts in-block (segment_lefts), takes a ballot per row group, ranks each lane by __popc under its lane mask, and writes rows and gradients coalesced to lefts + rank or total + (i - here), so the segmented scan kernel and its launch are gone and the scatter writes n_left itself. The leafwise plane keeps a side bit per histogram slot (slot_in_b): a split reads the parent's rows and gradients from the buffer its slot names and scatters both children into the other, both children take the flipped side, leaf_build fills from the small slot's side, and leaf_stamp stamps each side's slots in its own launch, so the two device-to-device copies per split that moved each child's segment back to the front buffer are gone with their stream serialisation. Integer histogram cells commute (decision 124), a fixed-point difference is the same bytes wherever it is taken, the tie order is the serial walk's, each column's sample keeps its row order and its sort is a total order on the same floats, the untouched slots were dead bytes, and the streamed tree sum is the shuffle tree's pairs in the shuffle tree's order, each finder cut is the same double as before, the screen only removes cuts the exact score would have rejected, a fixed-point difference formed in a register is the bytes the derive pass wrote, the halves recombine to a bound on the same side, the warp maximum of non-negative floats is the same float by either reduction, an outward-rounded fp32 threshold admits every cut the fp64 compare admitted, the host score is the device score's bytes, an int64 compare of positive doubles is their fp64 order, a power-of-two inverse is exact in fp32, a prefetch moves no byte into a register, a cell reassembled from two words is the same integer wherever those words sit, a wider tile adds the same rows' integers into the same slots, a partition that ranks by ballot keeps each side's rows ascending as the scan did, a segment holds the same rows in the same order on either side, the one-pass partition reverses the right side's rows but nothing reads a segment's order (the fill adds integers per row, the finder reads histograms, the stamp is per row), the count plane multiplies by the hessian quantum, a power of two, and the queued fill adds the same rows into the same fresh slot (the device picks the child with fewer rows as the host did, an event wait and a mapped read deliver the count the copy did), the kernel parameters carry the values the uploads carried and the mapped best split is the record the copy returned, so the histogram bytes are those of the general path; the model hashes are unchanged: CPU 55c6fe308852d9bb, device e1a5391a7beea349 (depthwise), d4d1539061bd3d36 (leafwise), 56c3d85ec0e37094 (levelwise), each equal between arms on the RTX PRO 6000 and on sm_87.

Why. Decision 124's Blackwell status banner left the wide cell (131072 x 16384) at -2.7 to +2.1% with the finder named as the suspect. A per-kernel decomposition (nsys, one RTX PRO 6000 pod, used to price the levers and not quoted as a delta) found two serial kernels ahead of it: the single-block small fill at 18.8 ms per launch (405 launches, 7.6 s of a 27.4 s depthwise fit; 6817 launches, 44.4 s of the 88 s leafwise fit) and the single-thread reduce at 3.6 ms per launch (840 launches, 3.0 s depthwise; 12386 launches, 27.0 s leafwise). One SM, or one thread, did work the whole device waited on. With those gone the fit's own profile put the next two in the ingest and the memsets: ingest_fit (the host mapper fit, 5.9 s of every wide fit, dbin a further 0.55 s) and the level memset at 1.00 s, streaming 134 MB per op over 12700 ops, which is 1.7 TB in 1.00 s, the GDDR7 roofline, so only traffic reduction could move it.

The same-pod A/B, RTX PRO 6000 Blackwell Server Edition, driver 595.91, three arms interleaved (before be39d55, the two kernel levers at a834923, and the mapper gather at 688c925), min over three reps, r2_test equal per plane in all arms; train is fit minus ingest:

cell	plane	fit before	fit kernels	fit gather	train before	train kernels	train gather	ingest before	ingest gather
16M x 128	depthwise	4.397 s	4.394 s (-0.1%)	4.378 s (-0.4%)	3.657 s	3.661 s	3.658 s	0.739 s	0.713 s
16M x 128	leafwise	6.685 s	6.547 s (-2.1%)	6.479 s (-3.1%)	5.958 s	5.798 s	5.753 s	0.728 s	0.722 s
16M x 128	levelwise	4.435 s	4.423 s (-0.3%)	4.373 s (-1.4%)	3.674 s	3.681 s	3.684 s	0.749 s	0.680 s
131k x 16384	depthwise	26.510 s	17.649 s (-33.4%)	14.092 s (-46.8%)	20.880 s	11.708 s	11.708 s	5.631 s	2.367 s
131k x 16384	leafwise	88.189 s	20.920 s (-76.3%)	17.418 s (-80.2%)	82.012 s	14.995 s	14.961 s	6.103 s	2.412 s
131k x 16384	levelwise	30.265 s	21.659 s (-28.4%)	18.149 s (-40.0%)	24.417 s	15.726 s	15.716 s	5.847 s	2.431 s

The split says which lever did what: the kernel levers moved train alone (wide depthwise 20.88 to 11.71 s, ingest 5.63 to 5.92 s, the difference rep noise) and the gather moved ingest alone (5.92 to 2.37 s, train 11.708 s in both arms to the millisecond). The tall rows sit inside the 2% band on train throughout (leafwise -3.4% is the small fill, which that plane launches per split); the tall ingest is 0.7 s on every arm because the sampled mapper fit was already cheap there.

The slot zeroing ran as its own two-arm session on the same pod, 688c925 against 688c925 plus the zeroing, min over two interleaved reps, wide cell only (the tall cell's slot is 0.5 MB and its memset is under the profile's resolution): train depthwise 11.711 to 11.200 s (-4.4%), leafwise 15.077 to 13.973 s (-7.3%), levelwise 15.714 to 15.165 s (-3.5%); ingest inside rep noise (2.30 to 2.42, 2.39 to 2.45, 2.38 to 2.38 s). The priced saving was half the 1.00 s level memset for the two level growers and most of the leafwise per-tree pool memset (256 slots of 67 MB, 17 GB, against about 123 fresh slots actually filled); measured 0.51, 0.55 and 1.10 s. Composed across the two sessions, the wide fit sat at about 13.6 s depthwise (-49%), 16.4 s leafwise (-81%) and 17.5 s levelwise (-42%) against the before arm.

The fused subtraction ran as two further sessions on the same pod type. The first, the four-lever arm (af16942) against it plus the fusion, min over three wide and two tall reps: wide train depthwise 11.575 to 10.054 s (-13.1%), leafwise 14.410 to 13.414 s (-6.9%), levelwise 15.809 to 14.635 s (-7.4%), priced at the subtract's 1.65, 1.39 and 1.79 s; the depthwise finder bucket moved +0.05 s while its wait moved -1.57 s, so the derivation hides under the fp64-bound scan almost entirely, where leafwise and levelwise show +0.4 and +0.5 s of visible kernel time against -1.4 and -1.7 s of wait. The tall cell read depthwise -0.3%, leafwise 0.0% and levelwise +2.0% (3.719 to 3.836 s): level_find_kernel runs one warp per selected feature, so 128 warps on a 188-SM part derived every large strip of a level serially by node. The second session added the narrow-level policy as a third arm, min over two reps of each cell: tall levelwise 3.724 / 3.800 / 3.735 s (before / fused / policy, +0.3% at the end), wide levelwise 15.534 / 14.378 / 14.373 s (-7.5%), the other four rows unchanged between the last two arms (tall depthwise 3.698 / 3.687 / 3.689, leafwise 5.920 / 5.928 / 5.928; wide depthwise 11.432 / 9.895 / 9.893, leafwise 14.280 / 13.256 / 13.297). Both branches of the policy are hash-gated: the 100-feature hash cell takes the pre-kernel on Blackwell (threshold 752 warps) and the fused path on the 8-SM Jetson (threshold 32). Composed, the wide fit sits at about 12.1 s depthwise (-54%), 15.5 s leafwise (-82%) and 16.3 s levelwise (-46%) against the before arm.

The finder lanes ran as a sixth session, be22344 against it plus the remap, min over two interleaved reps, r2_test equal per plane: wide levelwise train 14.414 to 13.272 s (-7.9%), tall levelwise 3.685 to 3.563 s (-3.3%), the four other rows inside +0.6%; the whole delta sits in the device bucket (11.36 to 10.24 s) with ingest unmoved. The lever was priced from lane-iterations, 3048 to 2025 per feature per tree on the depth-8 wide cell (the five levels with n <= 16 were 1275 of them, most idle), which would have been a third of the finder; it returned a tenth, because the count treats a deep level's iteration as a shallow one's when a deep level's load touches 32 node strips where a shallow level's touches one. Composed, the wide fit sits at about 12.1 s depthwise (-54%), 15.5 s leafwise (-82%) and 15.2 s levelwise (-50%).

The cut sweep ran as a seventh session, 5ab8363 against it plus the rewrite, min over two interleaved reps, r2_test equal per plane: wide levelwise train 13.266 to 12.139 s (-8.5%), tall levelwise 3.571 to 3.267 s (-8.5%), the four other rows inside +0.2%; grow_find carries the whole delta (-1.12 s and -0.31 s) with every other bucket flat. The rewrite removes the sixth lever's lane remap along with the serial walk it was patching, since a block that owns cuts has no idle lanes at any level; the sixth session stays in the record as the measurement that priced the deep levels. The kernel's fp64 scratch buffer (level_score, one double per node per cut) goes with it, since the sum lives in registers and only the per-feature best is written. Kernel resources on sm_87 read 128 registers, no stack, 11200 bytes of shared memory. The first build lost the candidate's feature index on lane 0 of the winning warp, whose in-warp reduce shuffles only gain, bin, direction and validity, and split on feature 0 with feature 2's bin; the Jetson parity suite caught it at the fifth level of a 4096-row scenario, and the fix initialises every thread's candidate with its block's feature. Composed, the wide fit sits at about 11.9 s depthwise (-55%), 15.3 s leafwise (-83%) and 14.5 s levelwise (-52%); train 20.9 to 9.7 s, 82.0 to 13.0 s and 24.4 to 12.1 s.

The in-place mapper fit ran as an eighth session, 988f2e6 against it plus the rewrite, min over two interleaved reps, r2_test equal per plane: wide ingest 2.317 to 1.865 s depthwise (-19.5%), 2.363 to 1.815 s leafwise (-23.2%), 2.414 to 1.836 s levelwise (-23.9%), with ingest_fit 1.89 to 2.08 s per rep before and 1.38 to 1.49 s after on every plane and dbin unmoved at 0.55 s; wide train inside -0.9% and the tall rows inside the 2% band on fit, train and ingest. The wide fit reads 12.211 to 11.590, 15.592 to 14.996 and 14.717 to 14.020 s. The lever was priced on the M2 at -43% of the mapper fit (0.91 to 0.61 s per Dataset at 32768 x 16384, 8 threads) and returned -30% on the pod's 16 vCPU host, whose caches differ and whose normal data takes no uniform-byte skip. Composed, the wide fit sits at about 11.6 s depthwise (-56%), 15.0 s leafwise (-83%) and 14.0 s levelwise (-54%); ingest 5.6 to 1.8 s on every plane.

The finder op cuts ran as a ninth session, 0725c04 against it plus the cuts, min over two interleaved reps, r2_test equal per plane: wide fit 11.149 to 10.283 s depthwise (-7.8%), 14.843 to 13.765 s leafwise (-7.3%), 13.748 to 10.835 s levelwise (-21.2%); wide train 9.670 to 8.751, 13.286 to 12.058 and 12.194 to 9.264 s; the tall rows inside the 2% band on fit, train and ingest, and the wide ingest (host code the lever does not touch) spread 1.48 to 1.73 s across reps in both arms. The lever was priced from static op counts at -1, -1 and -2 s and returned -0.9, -1.1 and -2.9 s: GB202 runs fp64 at 1/64 rate, so on the wide cell the finders are bound on that pipe, and the level finder, which scores every cut of every node per pass, carried the most of it. The SASS on sm_87 for the lambda_l1 == 0 instantiation against the old kernel reads DADD 43 to 29 and DSETP 49 to 39 in level_find_kernel, DADD 41 to 29 and DSETP 47 to 43 in find_kernel, with no local memory in either and 128 and 90 registers as before. Composed through nine levers, the wide fit sits at about 10.3 s depthwise (-61%), 13.8 s leafwise (-84%) and 10.8 s levelwise (-64%).

The interval screen ran as a tenth session, 6593222 against it plus the screen, min over two interleaved reps, r2_test equal per plane: wide fit 10.564 to 9.036 s depthwise (-14.5%), 13.714 to 11.943 s leafwise (-12.9%), 11.226 to 11.124 s levelwise (-0.9%, that plane's finder untouched and the delta inside the ingest's rep spread); wide train 8.735 to 7.229, 11.762 to 10.086 and 9.254 to 9.250 s; find_kern 5.27 to 3.76 s on the depthwise plane and the leafwise cuda_gpu bucket 7.86 to 6.28 s, every other bucket flat; the tall rows within 0.4% on train. Device hashes hold with and without the toggle on Blackwell and on sm_87 (on a Python module relinked for the gate: the earlier Jetson gates rebuilt the test binary and not the module, so their suite evidence stands and their hash lines do not); the sm_87 SASS carries the rounding-mode variants, no local memory, and find_kernel grows from 90 to 124 registers. The lever was priced at -4 s on find_kern from the fp64 op count and returned -1.5 s; this entry first read the shortfall as a byte floor (the finder issues about 42 GB of strip reads per tree, 2.6 s at the memset's roofline), and the fourteenth session below refutes that reading: at a 64 MB node histogram the 128 MB L2 absorbs the finder's re-reads, so the kernel is bound by latency at 16 warps per SM and the shortfall was the screen's own instructions, which the twelfth and thirteenth levers cut. The fill's gpu_wait (2.87 s) sits at about 45% of its own byte floor (9.6 GB of bins and 8.5 GB of strips per tree), so that bucket is bound on its int64 shared atomics and not on bytes. Composed through ten levers, the wide fit sits at about 9.0 s depthwise (-66%), 11.9 s leafwise (-86%) and 11.1 s levelwise (-63%). Composed through fourteen levers, the wide fit sits at about 8.6 s depthwise (-68%), 11.2 s leafwise (-87%) and 11.1 s levelwise (-63%); train 20.9 to 6.7 s, 82.0 to 9.2 s and 24.4 to 9.3 s.

The prefetch ran as an eleventh session, 544f691 against it plus the prefetch, min over two interleaved reps: wide depthwise find_kern 3.76 to 3.69 s (-1.9%, every changed rep under every base rep), train 7.218 to 7.146 s; wide leafwise train 10.116 to 10.039 s, cuda_gpu flat; the tall rows flat. find_kernel reads 124 to 131 registers on sm_87 with no local memory and 128 with no stack on sm_120. The session also priced the finder's occupancy: find_kernel at 128 registers runs 16 warps per SM, and a dummy dynamic shared allocation that caps it at 11 and at 7 warps (the probe prints find_kernel occupancy N warps/SM) reads 4.19 and 5.84 s on the same cell, so each resident warp is worth about 0.1 s of the finder and the kernel is latency-bound, not byte-bound.

The fp32-pipe division ran as a twelfth session. On sm_120 clang lowers __fdiv_ru and __fdiv_rd to a 116-instruction software subroutine (CALL.REL.NOINC, looped, the slow-path FCHK inside), four calls per cut per lane, where a plain fp32 division is an inline MUFU.RCP plus FFMA chain. The inline division costs two registers, and find_kernel at 130 rounds to 136 per lane, 15 warps per SM instead of 16; measured alone that step cost more than the subroutine saved (wide depthwise find_kern 3.69 to 3.85 s, +4.3%). With __launch_bounds__(32, 16) pinning 128 registers and 16 warps the lever pays: find_kern 3.69 to 3.63 s, depthwise train 7.15 to 7.09 s, leafwise train 10.06 to 9.91 s, tall flat; sm_87 also compiles to 128.

The 32-bit conversion ran as a thirteenth session, on top of the twelfth. interval_of took each int64 prefix cell to fp32 with the directed 64-bit conversions, four per cut per lane, and on sm_120 I2F.S64 runs on the same 1/64-rate pipe as fp64 arithmetic. Same pod, interleaved, two reps each: wide depthwise find_kern 3.63 to 3.25 s (-10.5%), depthwise train 7.09 to 6.71 s, leafwise train 9.91 to 9.40 s, leafwise cuda_gpu 6.00 to 5.65 s; tall flat. The SASS reads I2F.S64.RM/RP gone from find_kernel with I2F.RM/RP and I2F.U32.RM/RP in their place, 128 registers held on sm_120 and sm_87.

The paired finder ran as a fourteenth session, 74298db against it plus the pairing, min over two interleaved reps: wide depthwise find_kern 3.25 to 3.19 s, train 6.71 to 6.65 s; wide leafwise train 9.44 to 9.18 s (-2.7%), cuda_gpu 5.6 to 5.35 s; tall flat on both planes. It was priced from bytes at about 1.3 s (the derive pass's write and read-back plus the last level's 4 GB of writes per tree) and returned 0.06 s on the depthwise finder, which is the refutation of the byte floor: the node histogram is 64 MB against a 128 MB L2, so the write-then-read-back and the small strip's second read were already L2 hits, and the only DRAM traffic the lever removes is the last level's write. A variant that carried the store decision as a runtime pointer test inside the sweep loop measured the same (3.20 s) with 72 bytes of stack per thread; the template form spills 48 on sm_120 and 8 on sm_87 at 128 registers, where the base spilled none, because the derived sweep carries three strip pointers and two prefetched chunks under the register cap.

The certified-bound reduce ran as a fifteenth session, 140428a against it plus the reduce, min over two interleaved reps, every changed rep faster than its interleaved base rep, r2_test equal per plane: wide depthwise find_kern 3.28 to 3.20 s, train 6.789 to 6.686 s (-2.4% / -1.5%); wide leafwise cuda_gpu 5.44 to 5.39 s, train 9.367 vs 9.397 s; tall flat (1.16 = 1.16 s, 3.658 vs 3.640 s). It was priced at -10 to -25% from the shuffle count and returned -2.4%.

The fp32 thresholds ran as a sixteenth session, c3e7118 against it plus the thresholds, min over three interleaved reps, every changed rep faster than every base rep: wide depthwise find_kern 3.20 to 2.03 s, train 6.683 to 5.500 s (-36.6% / -17.7%); wide leafwise cuda_gpu 5.49 to 4.06 s, train 9.363 to 7.999 s (-26.0% / -14.6%); tall flat on one rep. SASS on sm_120: F2F.F64.F32 50 to 14 in the module, DSETP 359 to 323, find_kernel STACK 48 to 16 bytes at 128 registers. That corrects the fourteenth session's reading of the spill: the fp64 threshold compares held the registers, not the strip pointers.

The host-staged screen ran as a seventeenth session, 927dd4d against it plus the staging, min over three interleaved reps, every changed rep faster than every base rep on the four wide metrics: wide depthwise find_kern 2.03 to 1.84 s, train 5.502 to 5.321 s (-9.4% / -3.3%); wide leafwise cuda_gpu 4.09 to 3.84 s, train 8.048 to 7.812 s (-6.1% / -2.9%); tall 3.625 vs 3.638 s. The int64 key compare measured alone was flat (2.01 vs 2.03 s) and stays because the measured arm carries it and it removes 84 DSETP. SASS: F2F 54 to 20, DSETP 323 to 237, STACK 16 to 24.

The fp64-pipe throughput model those three levers were priced on (16 SM-cycles per warp fp64 instruction on sm_120, a removed per-cut instruction worth its count times the cut count) predicted the fifteenth and sixteenth within 10% and priced the seventeenth at -20% for a -9.4% return: a per-block prologue overlaps with the other resident warps, where a per-cut instruction sits on every warp's chain. A counting probe on the seventeenth arm (a direct worker run of the wide depthwise cell) read 385.7M sweeps, 393.8M flushes (1.02 per block), 2.06G survivors (5.3 per block, 2.1% of the cuts) and 3.086G chunk directions; the exhaustive finder reads 5.65 to 5.68 s against 1.89 to 1.95 s screened, so a warp flush costs about 610 SM-cycles, about 38 fp64 instructions, and the exact path is about a quarter of the finder.

SASS of that arm's chunk loop (788 instructions on sm_120) carries 230 NOPs, all but four in the exact path as runs of four WAIT15 stalls behind each dependent DFMA, DMUL, DADD or DSETP: fp64 dependent-op latency written out by the scheduler. The screened path is 342 instructions with four NOPs, about 0.3 instructions per scheduler-cycle at the measured 230 to 300 SM-cycles per chunk, so the finder is neither issue-bound nor fp64-bound; it is bound by each warp's latency chain at 16 warps per SM. The same dump showed float(inv) rematerialised per chunk as two F2F.F32.F64 from uniform registers, which the eighteenth lever removes.

The fp32 inverse ran as an eighteenth session, four arms interleaved over three wide reps and one tall rep, f4c8c54 against it plus the inverse and against two occupancy arms, every changed rep faster than every base rep on depthwise find_kern and train and on leafwise train: wide depthwise find_kern 1.84 to 1.77 s, train 5.316 to 5.246 s (-3.8% / -1.3%); wide leafwise cuda_gpu 3.84 vs 3.82 s, train 7.848 to 7.738 s (-1.4%); tall 3.611 vs 3.631 s. SASS: F2F 20 to 4 in find_kernel, STACK 24 to 40.

The L2 prefetch ran as a nineteenth session on the same pod type, four arms interleaved over three wide reps and one tall rep: aec2e9e (the eighteenth lever), it plus the L2 prefetch, it plus a two-deep register prefetch, and both. Min over reps: wide depthwise find_kern 1.77 to 1.60 s (-9.6%), train 5.238 to 5.087 s (-2.9%), fit 6.876 to 6.800 s; wide leafwise cuda_gpu 3.70 to 3.53 s (-4.6%), train 7.533 to 7.440 s (-1.2%), fit 9.318 to 9.246 s; tall depthwise train 3.615 vs 3.611 s and leafwise 5.695 vs 5.713 s. Every wide rep of the prefetch arm beats every base rep on depthwise find_kern (1.60, 1.60, 1.62 against 1.77, 1.77, 1.80 s), depthwise train (5.087 to 5.104 against 5.238 to 5.287 s) and leafwise cuda_gpu (3.53 to 3.68 against 3.70 to 3.87 s); leafwise train separates on the min only (one prefetch rep at 7.579 s sits above the base's 7.533 s). r2_test reads 0.8597 on every record, and all 24 device hashes are equal across the four arms in both finder modes. SASS: four CCTL.PF2 in find_kernel, 128 registers and 40 bytes of stack unchanged, STL 26 to 22. The gain is the per-block DRAM chain the sweep serialised: eight chunks per strip, each load a DRAM miss the sweep waited on before issuing the next, and the prefetch turns that into one miss and seven L2 hits, which is what a finder bound on each warp's latency chain at 16 warps per SM should return. The prefetch arm's bench processes took about 9 s longer in wall clock than the other three arms in every rep, all of it before the first timed record (process start, data load, the untimed micro-fit), while its timed fit, predict and ingest read equal or better and the arm carrying the same lever plus the register form showed no such gap; it is read as a per-directory artefact of the pod, not the code, and is recorded because it was not explained.

The word planes ran as a twentieth and a twenty-first session on the same pod type. The first, cf468ec against it plus runtime-addressed planes (four plane bases per feature computed from stride / 2) and against it plus 64-bit shared atomics on the same planes, three wide and two tall reps interleaved: the runtime form read wide depthwise train 5.072 to 4.967 s (-2.1%), gpu_wait 2.87 to 2.81 s, wide leafwise train 7.475 to 7.341 s and tall depthwise 3.608 to 3.553 s, every changed rep under every base rep on depthwise train, but its SASS read 94 registers against 36 (the compiler hoists 8 features x 4 plane pointers), one 512-thread block per SM in place of three, so the bank win was capped by the lost residency. An nsys pass on that pod put the wide depthwise kernel time at hist_tile_kernel 2.04 s (39%, 828 launches at 2.47 ms), find_kernel 1.64 s (31%), hist_small_kernel 0.82 s (16%), zero_slots_kernel 0.52 s (10%) and bin_rows 0.14 s, and the tall depthwise time at hist_tile_kernel 2.10 s (60%), the scatter 0.57 s and the route count 0.55 s; the gpu_wait bucket (2.87 s) is the two fills (2.04 + 0.82 s) to the hundredth, so the parts explain the whole, and the tile fill reads its 2 GB of bins per level at about 0.87 TB/s on a 1.6 TB/s part. The second session, cf468ec against the runtime form and against the grouped form (32-bin groups of four 32-word planes, constant plane offsets), three wide and two tall reps interleaved, every grouped rep under every base and runtime rep on train in all four cells: wide depthwise train 5.073 to 4.854 s (-4.3%), fit 6.726 to 6.467 s; wide leafwise train 7.390 to 7.226 s (-2.2%), fit 8.971 to 8.850 s; tall depthwise train 3.608 to 3.318 s (-8.0%), fit 4.240 to 3.949 s; tall leafwise train 5.738 to 5.450 s (-5.0%), fit 6.344 to 6.073 s. grow_populate carries the whole delta (wide 0.54 to 0.32 s, tall 0.70 to 0.48 s) with find_kern and r2_test (0.8597 wide, 0.8797 tall) equal on every record, and the six device hashes are equal in both finder modes. SASS on sm_120: 66 ATOMS.ADD in hist_tile_kernel<8> as before, LDS 8 to 16 for the merge, 37 registers against 36, no stack; sm_87 reads 37 registers where the runtime form read 64.

The wide tile ran as a twenty-second and a twenty-third session on the same pod type, the tall cell (16M x 128) first. The first, 754c335 against it plus a 16-feature tile at 512 threads, plus the same at 1024 threads unbounded, and two bound arms (the atomics replaced by a per-thread register sink, and the atomics with their return discarded), three tall and two wide reps interleaved. The bound arms change the tree, so their timed rows are not evidence, but their nsys root fills are: with no atomics the 16M-row root fill reads 1.65 ms where the head reads 5.48 ms, so the strip loads and the quantise run near the byte floor and the shared atomics are about 70% of the fill. The 16-feature tile at 512 threads read tall depthwise train 3.333 to 3.099 s and hist_tile_kernel 1.83 to 1.44 s (829 launches, 2.20 to 1.74 ms), its 64 KiB shared tile one block per 100 KiB SM in place of three, so the per-row fixed cost (one strip load, one gradient load, two quantise ops per sixteen features instead of per eight) outweighed the lost residency; the unbounded 1024-thread arm failed every fill launch (72 registers x 1024 threads is more than the 65536-register block file, too many resources requested for launch). An nsys pass on that pod put the tall depthwise kernel time at hist_tile_kernel 1.83 s (57%), the scatter 0.58 s, the route count 0.55 s and the route add 0.10 s, and the tall leafwise at hist_tile_kernel 1.83 s (49%, 12672 launches at 144 us), the route count 0.75 s (26400 launches), the scatter 0.66 s and find_kernel 0.17 s, 3.73 s of kernels under a 4.53 s train: 0.8 s of the leafwise plane's 1.2 s over depthwise is launch gaps across about 132000 launches per 100 trees (two route counts, two scatters and two segmented scans per split), the rest more partition kernel time. The second session, 754c335 against the 512-thread tile, against a 768-thread tile (72 registers x 768 fits the file unbounded) and against a 1024-thread tile under __launch_bounds__(k_tile_fill_threads) (64 registers, no stack), three tall and two wide reps interleaved, every 768 and 1024 rep under every 512 rep and every 512 rep under every base rep on train in all four cells: tall depthwise train 3.319 to 3.088, 2.901 and 2.879 s (-13.3%), fit 3.939 to 3.497 s; tall leafwise train 4.523 to 4.394, 4.208 and 4.204 s (-7.1%), fit 5.165 to 4.831 s; wide depthwise train 4.901 to 4.667, 4.578 and 4.562 s (-6.9%), fit 6.297 to 5.972 s; wide leafwise train 6.154 to 5.857, 5.636 and 5.617 s (-8.7%), fit 7.553 to 7.020 s; the wide levelwise plane was not timed in these sessions. gpu_wait (the fills) carries it: tall depthwise 1.56 to 1.07, 0.99 and 0.98 s, wide depthwise 2.86 to 2.55, 2.50 and 2.50 s; nsys reads hist_tile_kernel 1.23 s on the bounded 1024-thread arm's tall depthwise (1.48 ms per launch, root fill 3.24 ms against the 1.65 ms no-atomics bound) and, on the 512-thread arm's wide depthwise, 1.83 to 1.55 s with hist_small_kernel<16> 0.83 to 0.88 s. The strip is now 16 bytes per row, two rows per 32-byte sector where there were four, and the partition kernels that gather one byte per row pay for it on every 16-wide arm alike: the tall route count 0.55 to 0.69 s, the route add 0.10 to 0.12 s, the gpu bucket 1.16 to 1.30 s on tall depthwise. find_kern and r2_test (0.8797 tall, 0.8597 wide) are equal on every record, and the six device hashes are equal in both finder modes on every arm. SASS on sm_120: hist_tile_kernel<16> 130 ATOMS.ADD, 32 LDS, no stack, 72 registers unbounded and 64 under the bound; sm_87 reads 62 under the bound, no stack.

The coalesced partition ran as a twenty-fourth session on the same pod type, ee4ae3e against it plus the partition, three tall and two wide reps interleaved, min per arm. Tall depthwise train 2.886 to 2.256 s (-21.8%), fit 3.506 to 2.885 s; tall leafwise train 4.185 to 3.506 s (-16.2%), fit 4.803 to 4.127 s; tall levelwise train 2.755 to 2.112 s (-23.3%), fit 3.378 to 2.737 s; wide depthwise train 4.497 to 4.534 s (+0.8%, inside the two-rep spread, the partition under 0.06 s at 131072 rows) and wide leafwise 5.584 to 5.524 s (-1.1%). The gpu bucket carries it (tall depthwise 1.30 to 0.66 s, leafwise 2.37 to 1.69 s, levelwise 1.39 to 0.75 s) with gpu_wait and find unmoved. nsys on the tall depthwise arm reads route_count_kernel 0.693 to 0.388 s (825 to 462 us over 840 launches), scatter_kernel 0.577 to 0.250 s (687 to 298 us), the segmented scan gone and hist_tile_kernel 1.229 s unmoved, 2.78 to 2.15 s of kernels; on the tall leafwise arm the route count 0.907 to 0.482 s and the scatter 0.663 to 0.435 s over 26400 launches each, the 26400 scan launches (0.042 s) gone, 3.42 to 2.72 s of kernels with hist_tile_kernel 1.34 s (49%) and find_kernel 0.166 s. The saving was priced at about 0.45 s from the sector count alone and measured 0.63 s on tall depthwise train, the rest being the sixteen loads in flight where the one-loop form serialised them behind the flag store it could not prove disjoint. r2_test (0.8797 tall, 0.877 tall levelwise, 0.8597 wide) is equal on every record and the six device hashes are equal in both finder modes on both arms. SASS on sm_120: route_count_kernel<uint8_t> 30 to 56 registers, 5 SHFL, 1 barrier where the old form had 9, scatter_kernel 40 to 64 registers, 16 VOTE, 31 POPC, 3 barriers where the old form had 17, no stack on either; sm_87 reads 48 and 64, no stack.

The leafwise buffer side ran as a twenty-fifth session on the same pod type, bac9715 against it plus the side, three reps interleaved on each leafwise cell, min per arm. Tall leafwise train 3.505 to 3.282 s (-6.4%), fit 4.052 to 3.843 s, the gpu bucket 1.69 to 1.51 s with gpu_wait 1.13 to 1.09 s; wide leafwise train 5.381 to 5.335 s (-0.9%), fit 6.693 to 6.639 s. nsys on the tall leafwise cell reads 52905 device-to-device copies of 3.3 us (0.177 s) on the base arm and 105 of 90 us (0.009 s) on the arm carrying the side, cudaMemcpyAsync 146100 to 93300 calls, the kernel sum 2.66 to 2.60 s with hist_tile_kernel 1.31 s (49%), route_count_kernel 0.48 s and scatter_kernel 0.42 s over 26400 launches each, find_kernel 0.16 s. The saving was priced at the copy time and measured 0.22 s, the difference being the copies' place in the stream between the scatter and the fill of the same slot. r2_test is equal on every record and the six device hashes are equal in both finder modes on both arms. The api sum names what the leafwise plane still spends: 41181 blocking cudaMemcpy of 50.6 us (2.08 s) and 13382 cudaDeviceSynchronize of 84.7 us (1.13 s) per 100 trees, one fetch per split for the child counts and one per find for the winner, which only a grow loop that takes several splits per fetch removes.

The one-pass partition ran as a twenty-seventh session on the same pod type, 51d0d42 against it and against it plus the count plane, three reps interleaved on each tall cell and two on each wide cell, min per arm, then two more sessions over the tile shape. The partition's two kernels were 0.890 s over 26400 launches each on the tall leafwise cell and 0.634 s over 840 on the depthwise, and every launch re-gathered the split feature's bins. A single kernel that claims a tile from a device counter, classifies, publishes an aggregate, walks the status words of the tiles before it 32 at a time until an inclusive entry, and scatters lefts up from the segment start and rights down from its end, is priced at the second gather and half the launches. At 16 rows per thread the rows, bins and ballot masks held across the look-back cost 107 registers on sm_120, two blocks per SM, and the tall train read 2.239 to 2.412 s depthwise (+7.7%), 3.944 to 3.840 s leafwise (-2.6%) and 2.110 to 2.244 s levelwise (+6.4%): the level partitions at 16M rows are bandwidth-bound and lost half their occupancy, the leaf partitions are launch-bound and gained. The count plane on top of that form took the tall train down a further 0.063 s depthwise, 0.079 s leafwise and 0.065 s levelwise, about 3% each, hist_tile_kernel 1.273 to 1.170 s leafwise and 1.212 to 1.135 s depthwise, and the later arms all carry it. Against 51d0d42 reading 2.240 / 3.984 / 2.106 s, __launch_bounds__(256, 4) gave 64 registers with 40 bytes of stack and 2.279 / 3.782 / 2.123 s, (256, 3) gave 80 registers and 2.292 / 3.753 / 2.135 s, and 8 rows per thread gave 64 registers with no stack and 2.249 / 3.542 / 2.102 s. Against 51d0d42 reading 2.245 / 4.004 / 2.115 s, 4 rows per thread gives 40 registers and 2.236 s (-0.4%), 3.499 s (-12.6%) and 2.108 s (-0.3%), partition_kernel 0.605 s leafwise against the 0.890 s of the two kernels and 0.715 s depthwise against 0.634 s, the kernel-time loss at the level partition paid back by the launches it no longer makes; 8 rows at a 512-thread block reads 2.230 / 3.578 / 2.088 s, the leafwise 0.08 s behind the 4-row form and the others inside the rep spread. The wide cells read 4.491 to 4.438 s depthwise and 6.337 to 6.164 s leafwise on the 8-row arm; the engagement line under BONSAI_CUDA_PROFILE ends in unit-hessian count plane on the arm that carries it and not on the others (the bench harness keeps the profile counters and drops the text lines, so the line was proven on a direct worker run). r2_test is equal on every record and the six device hashes are equal in both finder modes on every arm.

The queued fill ran as a thirtieth session on the same pod type, f7b4946 against it plus the queue, three reps interleaved on each tall cell and two on each wide cell, min per arm. Every leafwise expansion had drained the stream twice, leaf_split fetching the left count with a blocking copy and leaf_build waiting on its segment upload before the fill could launch, so the tall leafwise train spent about 1.3 s of 3.5 s idle with the partition bucket reading 1.11 s against 0.605 s of kernel. With the partition's last tile writing the small child's segment and the fill queued behind it, the tall leafwise train reads 3.021 to 2.758 s (-8.7%), fit 3.667 to 3.402 s, the gpu bucket 1.29 to 1.11 s, populate 0.48 to 0.41 s and partition 0.96 to 0.83 s; the wide leafwise train reads 5.390 to 5.134 s (-4.7%), populate 0.37 to 0.29 s and partition 0.48 to 0.33 s. The depthwise and levelwise trains never take the leaf path and read 2.275 to 2.273 s and 2.141 to 2.160 s tall and 4.496 to 4.529 s wide, inside their rep spread; the 16-feature tile kernel, which gained the sub-512-row branch, reads 63 registers on sm_120 against 62 and 64, and partition_kernel 40 against 40. The engagement line under BONSAI_CUDA_PROFILE ends in nl by event on the arm that carries it and not on the other (proven on a direct worker run). r2_test is equal on every record and the six device hashes are equal in both finder modes on both arms and on sm_87.

Both halves of the launch lever ran as a thirty-first session on the same pod type, 71dec59 (k0) against it plus the kernel parameters (l27a) and plus the parameters and the event (l27b), three reps interleaved on each tall cell and two on each wide cell, min per arm. Tall leafwise train 2.791 to 2.673 to 2.627 s (-4.2%, -5.9%), the finder bucket 1.46 to 1.37 to 1.32 s and the partition staging bucket 0.04 to 0.00 s; wide leafwise 5.103 to 4.967 to 4.887 s (-2.7%, -4.2%), its finder bucket 4.67 to 4.52 to 4.46 s. The depthwise and levelwise planes never take the leaf path: tall depthwise 2.295 / 2.281 / 2.284 s, tall levelwise 2.144 / 2.169 / 2.165 s, wide depthwise 4.484 / 4.466 / 4.469 s, inside the rep spread. On sm_120 find_kernel<LeafFindNodes> reads 128 registers with a 24-byte frame and find_kernel<FindNodesRef> a 32-byte frame against the single finder's 40, partition_kernel 40 registers on all four instantiations, level_find_kernel unchanged, no warnings. The engagement line under BONSAI_CUDA_PROFILE names the kernel parameters on l27a and l27b and the event on l27b only, never on k0 (proven on a direct worker run). r2_test is equal on every record and the six device hashes are equal in both finder modes on all three arms and on sm_87.

Refuted: 64-bit shared atomics on the planes. Native atomicAdd(unsigned long long) into structure-of-arrays int64 planes: sm_120 lowers a 64-bit shared atomic to a compare-and-swap spin as sm_89 does (32 ATOMS.CAST.SPIN.64, 59 to 61 registers, 16 bytes of stack), and it read wide depthwise train 5.46 s (+8%) and tall +13% against the base.

Refuted: occupancy bounds on the tile fill. On the runtime-addressed form __launch_bounds__(512, 3) gave 40 registers with 184 bytes of stack (45 STL, 45 LDL) and read wide depthwise train 5.454 s (+7.7%), wide leafwise 8.093 s (+8.5%), tall depthwise 3.915 s (+8.6%) and tall leafwise 6.566 s (+15%) against the base's 5.065, 7.459, 3.604 and 5.691 s; __launch_bounds__(512, 4) left the kernel at 114 registers (the cap is dropped rather than met) and read 4.954, 7.276, 3.511 and 5.604 s, the runtime form's numbers to the hundredth, so residency past 64 registers is one block per SM whatever the count, and the register diet had to come from the layout.

Refuted: the 1024-thread tile without its launch bound. The 16-wide tile compiles to 72 registers, and 72 x 1024 exceeds the 65536-register block file, so every root fill launch fails with too many resources requested for launch and the fit errors out; __launch_bounds__(k_tile_fill_threads) is what lets the block launch, not a tuning, and it lands at 64 registers with no stack on sm_120 and 62 on sm_87.

Refuted: a second cut chunk prefetched into registers. sweep_cuts holding two chunks ahead instead of one (after beside next) pushed the stack from 40 to 64 bytes (STL 40, LDL 48, LDG 54 to 60) at the 128-register cap and read find_kern 1.78 vs 1.77 s, train 5.248 vs 5.238 s depthwise and cuda_gpu 3.76 vs 3.70 s, train 7.574 vs 7.533 s leafwise: the extra chunk spilled, so the prefetch became a global load plus a local store and reload and hid nothing. Stacked on the L2 form it read 1.60 / 5.080 / 3.57 / 7.400 s with 56 bytes of stack, the L2 arm's numbers, so the register form adds nothing the L2 prefetch does not already give.

Refuted: the derivation fused into the sweep without the prefetch, and wider finder blocks. The first form of the fourteenth lever (the difference formed per cell inside the sweep, with leafwise deriving in place, and no prefetch) measured wide depthwise find_kern 3.76 to 3.98 s (+5.8%) and leafwise train 10.12 to 10.23 s on the eleventh session's pod, and 144 registers on sm_120 (14 warps per SM); the eleventh session split it, kept the prefetch, and the fourteenth rebuilt the derivation on top of the twelfth and thirteenth levers with the register cap. Four and two features per 128- and 64-thread find block, so a block amortises its launch over more warps, measured 4.09 and 4.00 s against 3.76 s (+8.8% and +6.4%) at equal occupancy (__launch_bounds__(128, 4), 128 registers with 16 bytes of stack), which places the finder's cost in each warp's own latency chain and not in block turnover; the first build of that probe stepped derive_large_strip by the warp width and made the 256-thread level finder derive each strip eight times over (wide levelwise +55%), an artefact the parameterised stride removed and the rerun confirmed flat.

Refuted: the same screen over the level finder. The level finder was 6.2 s of the levelwise grow after the ninth lever with the same 2.6 s byte floor, so it was the one finder still bound on the fp64 pipe, and the interval screen looked like the same -1.5 s or more. It was built in full: a first pass over every node bounds each cut's level gain in fp32 (IntervalTreeSum, the shuffle tree over intervals, with the feature-independent parent sum hoisted into a one-block level_parent_kernel that replays the walk's push order so the bytes stay the walk's), a keep mask, and a second pass that scores only the survivors exactly, with per-(node, warp) exclusive prefix checkpoints so the second pass can skip a cut's neighbours; the level suite passed on sm_87 and all six hashes held with and without the toggle on Blackwell. Same pod, 951f8ce against it plus the level screen, min over two interleaved reps: wide levelwise fit 11.140 to 14.230 s (+27.7%), train 9.314 to 12.419 s (+33.3%), cuda_gpu 6.27 to 9.38 s, grow_find 9.20 to 12.42 s; tall levelwise fit 3.934 to 4.018 s (+2.1%, cuda_gpu 1.13 to 1.19); the depthwise and leafwise rows flat on train (wide depthwise fit -2.6% is the ingest's rep spread). The mechanism is the shape of a level gain: it is a sum over 2^d nodes of split scores less a parent sum of the same size, two nearly equal totals, and the fp32 interval's width grows with the node count while the gap between competing cuts at depth does not, so at the deep levels that carry the cost almost nothing certifies (gain.hi < lower rarely holds), the second pass rereads every strip it would have read anyway, and the first pass (a full strip read plus the interval arithmetic) is pure overhead. A node's score is one quotient and an interval screens it well; a level's gain is a difference of two long sums and an interval cannot. Two facts from the build stand on their own: ptxas targets 128 registers for a __launch_bounds__(256) kernel so that two blocks fit an SM, and spills to reach it (the two-pass kernel read 392 STL/LDL at 128 registers, 49 once the parent sum was hoisted; __launch_bounds__(256, 1) gave 157 registers and no spills at half the occupancy), and the hoisted parent sum is a self-contained cut of about 2% of the level kernel's arithmetic that was not measured on its own and is not adopted here.

Refuted: the screen's divisions on the approximate divide with a certified bracket. The screen's four fp32 IEEE divisions per chunk (MUFU.RCP, an FFMA refinement and an FCHK guard with a slow-path call) were replaced by the two-instruction approximate divide bracketed two float steps each way, twice on the same pod type: on the fifteenth arm 3.21 vs 3.28 s find_kern, train 6.729 vs 6.765 s depthwise and 9.408 vs 9.371 s leafwise; on the sixteenth arm 2.04 vs 2.02 s, 5.520 vs 5.500 s, leafwise 8.014 = 8.014 s with cuda_gpu 4.08 vs 4.19 s. MUFU.RCP rose 112 to 136 and nothing moved past rep noise: the guard's slow path is never taken on these operands, so the IEEE divide already costs its reciprocal.

Refuted: two cuts per lane. A sweep with two longlong2 per lane, 64 cuts per chunk and one scan over the pair, so each warp holds two independent chains: at 128 registers it spills 96 bytes of stack and reads 2.20 vs 2.03 s find_kern, train 5.667 vs 5.510 s depthwise and 8.193 vs 7.941 s leafwise (cuda_gpu 4.32 vs 3.97 s); under __launch_bounds__(32, 12) it holds 168 registers with 8 bytes of stack and reads 2.10 / 5.570 / 8.068 s (cuda_gpu 4.08 s); tall flat. The doubled chain either spills under the cap or costs four resident warps without it, and either loses more than the second chain hides.

Refuted: the occupancy bounds. On the eighteenth arm __launch_bounds__(32, 24) gives 80 registers with 184 bytes of stack (154 STL, 174 LDL) and reads 1.78 s find_kern, 5.257 s train depthwise, 4.11 s cuda_gpu and 8.052 s train leafwise against 1.77 / 5.246 / 3.82 / 7.738 s: the spills eat the eight extra warps. __launch_bounds__(32, 32) does not cap at 64 registers; ptxas chose 144 with no stack, 14 warps per SM, and it reads 2.10 / 5.581 / 4.06 / 8.100 s. Two fewer resident warps cost 0.26 s of a 1.84 s finder, about 0.13 s per warp, which confirms the eleventh session's ladder. All six device hashes are equal across the four arms in both finder modes.

Refuted: the last level's finder reading the sibling difference. At the last level of a tree the children are leaves and their histograms are never read again, yet find_node derived every larger sibling (parent minus smaller) into its slot before sweeping it, a write of the whole strip and a read back: 64 of the 126 derives per tree at depth 8, 16384 features by 4 KB, about 4.3 GB of writes per tree and 0.27 s per fit at the memset's roofline. It was built: find_node took a StripSource (a cell pointer and an optional subtrahend), the pipeline counted its own depth, and at the last level the finder pointed at the parent and the small sibling and formed the difference per cell as it loaded, storing nothing; the leafwise finder always stored, since its slots outlive the find. The suites passed and the device hashes held in both toggle modes on Blackwell and, on a relinked module, on sm_87. Same pod, 113a421 against it plus the change, min over two interleaved reps, r2_test equal per plane: wide depthwise fit 9.097 to 9.480 s (+4.2%), train 7.292 to 7.672 s (+5.2%), find_kern 3.81 to 4.15 s; wide leafwise fit 12.274 to 12.715 s (+3.6%), train 10.398 to 10.973 s (+5.5%), cuda_gpu 6.41 to 7.16 s; wide levelwise flat (+0.5%, its finder untouched); the tall cells within the band on every plane, and every base rep faster than every changed rep on the two moved buckets. The mechanism is where the subtrahend test landed: the null-tested pointer sits inside the per-cell load, so both sweep passes at every level evaluate it, find_kernel grew from 124 to 129 registers on sm_87, and the leafwise plane, which never takes the new path, paid the same tax, which places the cost in the load and not in the difference. That tax is several times the priced saving, so whether any write saving sat underneath cannot be read from the session. A StripSource<bool> that compiles the test away was named and not built: the last level's sweep would then issue two loads per cell in each of its two passes on a kernel near its byte floor, which is the write it removed, read twice. Reopener: a finder that reads each strip once (a sibling pair per block, the parent and the small child loaded once and both children scored from them), where the last level's difference comes free. The fourteenth lever takes this reopener with the sibling blocks adjacent in the grid rather than sharing one, and the byte saving it priced did not appear, for the reason the fourteenth session records.

Refuted: a root bulk zero of the leafwise pool. One cudaMemset over every slot a tree can reach at leaf_begin_root, so leaf_build launches no zero kernel until a slot is reused: priced at the 0.30 s of zero launches on the wide cell, it read wide leafwise train 5.335 to 6.325 s (+17.5%) on the twenty-fifth session, tall leafwise 3.282 to 3.278 s (noise), the other planes flat. grow_populate carried 0.38 to 1.32 s of it, the memset of about 2 GB per tree sitting on the critical path where the per-build kernels ran between fills, and hist_small_kernel carried the rest, 78.8 to 129.0 us per launch (0.54 to 0.88 s over 6817 launches): a slot zeroed by a kernel just ahead of its fill is L2-resident (128 MB of L2 on this part) for the small fill's scattered int64 atomics, and a slot memset at the root is DRAM-cold by the time its fill lands. The zero kernel is a prefetch as much as a write, and the fill-targeted form of the fourth lever stays.

Refuted: a 16-byte slot zero. zero_slots_kernel storing ulonglong2 pairs over a 1024-block grid in place of scalar int64 over 256 blocks: sm_120 SASS shows the pair store lowered to two STG.E.64 ... RZ at offsets 0 and 8 (registers 12 to 14, no STG.E.128), and a twenty-sixth session on the same pod type, four arms three reps interleaved on the wide leafwise cell, reads the kernel at 24.2 us per launch on the base grid, 28.3 us for the pair form, 22.8 us for the scalar store on the wider grid, and 25.5 us for an inline st.global.v4.u32 that does emit STG.E.128, over 12281 launches; train min 5.333, 5.329, 5.306 and 5.297 s in that order, inside the 3% rep spread. The wider grid alone is worth 0.017 s per 100 trees, under the bar for a commit, and neither vector form beats the scalar store, so the kernel is unchanged.

On the eighth arm the small fill reads 2.0 ms per launch, the reduce 0.05 ms, and kernel time sums to within a second of the depthwise grow, so the parts explain the whole. What remains of the wide cell is the finder (find_kernel 6.6 s of the depthwise grow and level_find_kernel about 9 s of the levelwise grow on that arm, less the 0.9 and 2.9 s the ninth lever returned, the 2.1 s the tenth through fourteenth returned, the 1.4 s the fifteenth through eighteenth returned and the 0.17 s the nineteenth returned on the node finder, which now reads 1.60 s; the node finder is bound by the latency of each warp's chain at 16 warps per SM (the screened path issues about 0.3 instructions per scheduler-cycle, and the exact path, a quarter of the finder, stalls four deep behind each dependent fp64 op), since each resident warp is worth about 0.13 s on the occupancy ladder, and the level finder keeps one barrier per node per pass and its fp64 gain math on a 1/64-rate pipe) and an ingest of about 1.4 s of mapper fit plus 0.55 s of dbin, both host work whose remaining cost is the sort of 16384 columns of 131072 floats. The tile fill was 2.04 s of the wide depthwise kernel time and 2.10 s of the tall before the twentieth lever, at about half its byte floor; the small fill (0.82 s) and the slot zeroing (0.52 s) are the next two kernels on the wide cell by the nsys decomposition.

Two build facts the kernel levers carry. The strip loader unpacks its vector load from registers by shift rather than storing through the bin array, because LLVM's SROA pre-split that store into the array's two-byte reads and the uint16 tile kernel lost its single 16-byte load; on sm_87 and sm_120 both tile kernels keep LDG.E.128 (uint16) and LDG.E.64 (uint8) with the 32-atomic inner loop at 245 instructions against 244. The block-uniform full-strip test is hoisted out of the row loop as a template flag so the hot loop holds one load shape; the partial-strip copy runs only on the last tile of a feature count the tile width does not divide. The gather's shape was swept on an M2 at 32768 x 16384 (blocks of 8, 16, 32, 64 and 1024 columns read 0.29, 0.38, 0.46, 0.49 and 0.65 s for the gather; 4, 8, 16 and 32 rows ahead read 0.53, 0.29, 0.29 and 0.38 s), a laptop sweep used to choose the constants, not quoted as a delta. That sweep's ordering was an artefact of the write side: at a power-of-two row count an unpadded dense block puts every column stream in one cache set, so 16 and 32 columns read 0.44 s where 8 read 0.26 s; with the streams 32 floats past a multiple of the row count apart, 16, 32 and 64 columns read 0.121, 0.101 and 0.116 s and the block width settled at 32. Single-thread sort plus run lengths on 256 columns of 131072 normal rows read 0.247 s with a count pass per byte and a push_back run-length loop, 0.133 s with the four histograms from the key pass and run ends written without a branch (0.478 to 0.157 s on integer-valued columns, where two of the four scatters skip); a per-block gather buffer costs 5% over a thread-local one and keeps no scratch alive after the fit.

Rejected alternatives. Routing every node through the tile kernel, so the small kernel disappears: the tile kernel zeroes and merges a shared histogram per block, which for a 20-row node is more work than the atomics it saves, and the 512-row threshold is the one decision 106 (the hist chunk axis) swept, every cutoff above it worse at every cell. A wider small block (1024 threads) without the tile split: one SM still carries the node. A warp-shuffle reduce with no shared stage: 16384 candidates over 32 lanes is 512 strided steps per lane before the shuffle, where 256 threads take 64 steps and an 8-level tree. Reducing in find_kernel itself: it would put the cross-feature order inside the per-feature kernel and end the one-launch-per-node shape the leafwise plane relies on. A dense mapper-fit write with one NaN compaction pass per column instead of push_back: 0.51 s against 0.29 s on the M2 sweep, and the unpadded dense stride that sweep used is the cache-set aliasing the eighth lever's padding removes. Copying each column out of the gathered block into its own vector so from_sample could keep taking a vector by value: +0.032 s on the 32-wide gather sweep and a second copy of every column, where a span over the block sorts the bytes where they landed. Raising the subtract kernel's 256-block grid cap: the subtract already streamed 12700 x 200 MB in 1.65 s, the memory roofline, so more blocks could not move it, and the fusion removed the pass instead. A wider level_find_kernel block so more warps derive, while the lanes still owned nodes: the finder's registers are allocated per block, so a 128-thread block whose three extra warps exit after the derivation would have cut the resident warps four-fold on the kernel that was 11 s of the levelwise grow; the seventh lever widens the block to 256 threads that all do finder work instead. Shuffling the feature index in the warp reduce to fix the lost candidate: the index is block-uniform, so initialising it makes the loss impossible where a fifth shuffle would only carry it. Filling both children directly when both are under 512 rows, so the op needs neither subtract nor a zeroed large slot: 28.3% of the wide cell's ops (3316 of 11721, a Model.dump() cover walk), but a scattered int64 atomic costs about 8x a streamed byte, so the direct fill only wins when the large child is under about 50 rows, and at level 6 the small child alone averages 585. A ShuffleTreeSum whose five partials sit in an array walked by an early-return loop: the compiler collapsed the loop into a count-trailing-zeros index and put the array in local memory (166 STL and 20 LDL in the level finder), where five named scalars and a hand-unrolled chain compile to a uniform branch chain with one DADD per level and no local memory. Erasing + 0.0 under fast-math instead of the select form: device code builds with -ffp-contract=fast and no fast-math, so x + 0.0 is not foldable, and the select is the only form that removes the add while the sign argument holds. Specialising lambda_l1 in bounded_leaf_weight too: a leaf value keeps the sign of its zero, so the integer NaN test would change the bytes of a zero leaf. Scoring the node finder's survivors in fp32 instead of screening them: it changes which cut wins on a near tie, so it is a model change and its own decision. Letting the inline division take its two registers (136 per lane, 15 warps per SM): +4.3% on the finder, more than the 116-instruction subroutine cost, so the register cap is part of the lever. Deriving the sibling without the prefetch, or in one block per sibling pair: the first measured +5.8% and the second is the wider-block probe measured at +6 to +9%. A bulk L2 prefetch (cp.async.bulk.prefetch.L2) in place of the per-lane form: it is sm_90 and later, and the sm_87 gate would lose the kernel. Word planes addressed at runtime, or bounded to three or four blocks per SM: 94 registers, spills, or an ignored cap, each measured above. The 16-wide tile at 512 threads, or at 768 without a bound: 0.21 s and 0.02 s behind the bounded 1024-thread form on tall depthwise train, and the bound is what makes 1024 threads launch at all. Reopener: a wide cell where the level finder's fp64 gain math is priced against an fp32 form (model-changing, its own decision), or a level screen that bounds each node's contribution against a per-node bound rather than the level total, or a mapper fit whose remaining 1.4 s (the sort of 16384 columns of 131072 floats) is priced against a device sort of the gathered block, which would trade the host's per-column radix passes for one upload of the sampled columns, or a node finder that fits more than 16 warps per SM without spilling (the 24-block bound spilled 184 bytes and two cuts per lane spilled 96 or cost four warps, so the register diet has to come from the exact path's live state), since the occupancy ladder prices each resident warp at about 0.13 s of the 1.6 s finder, or an exact path with a shorter dependent fp64 chain, since the flush stalls four deep behind each of its 38 fp64 instructions and is a quarter of the finder, or a tile fill whose shared atomics are the last half of its root fill (3.24 ms against a no-atomics bound of 1.65 ms at 16M rows), or the partition kernel's level form (0.715 s at 16M x 128 against 0.634 s for the two kernels it replaced, the look-back's serial chain over 16384 tiles of 1024 rows being the cost that its halved launches pay for), or the leafwise plane's blocking fetches (about 39800 per 100 trees at 16M x 128, one per split for the child counts and one per find for the winner, 2.08 s of cudaMemcpy and 1.13 s of cudaDeviceSynchronize in the api sum, removable only by a grow loop that takes several splits per fetch, which is a change to the shared grow loop and its own design), or the 0.54 s of small fill and 0.30 s of slot zeroing the wide leafwise decomposition names, or the 1.2 s of tile fill (52% of the tall leafwise kernel time) whose shared atomics are its last half, three per row-feature under the count plane.

127. The 2.2.0 refresh ships the tall CUDA cells 34 to 60% faster; the CPU cell read over its anchor is one repeat's spread (adopted)¶

Decision. The 2.2.0 standings supersede 2.1.0's on every axis with every moved cell of the release A/B shipping as measured: the four gpu-tall cells sit 33.6 to 60.3% under the previous release and 33.7 to 60.0% under the 1.15.0 anchor, and the cpu-tall depthwise cell reads 3.7% over the anchor against a 2% band while reading 0.3% under the previous release. The device moves are decision 126's twenty-seven levers reaching the tall cell, which is what that entry measured (tall train 22 to 31% on its own same-pod ladder) and what this refresh confirms on the standings host. The CPU reading is not re-measured before the release: the anchor arm's four depthwise repeats span 10.58 to 11.41 s, a 7.8% spread, and the min-over-repeats statistic the gate reads took the low outlier; the new arm's four repeats (10.97 to 11.16 s) all sit inside that spread, the previous release's (11.01 to 11.19 s) likewise, and medians read anchor 11.03 s against new 11.09 s (+0.5%). The 2.1.0 release settled the same question on a CPU pod with no quota (cumulative 1.15.0 to 2.1.0 read -0.2 to -0.7%), and 2.2.0's diff against 2.1.0 is the device plane, with the wire identity unmoved.

Why the CPU band cannot be read on this host. A GPU pod meters its CPU plane by a bandwidth quota (13.6 cores against the 128 nproc advertises), so a host measurement there wanders between repeats in a way a CPU pod's cpuset does not; the leafwise cell in the same file reads +4.5% over the previous release and -0.4% over the anchor, and the levelwise cell 0.0% and -0.5%, three growers whose readings against two references disagree by more than the band in both directions, which is the host's spread and not three different engines. The gate is kept at 2%: on a CPU pod it resolves that band (the 2.1.0 session's per-rep spread was under 1%), and a refresh whose CPU plane runs on a GPU pod ships its moved cell under this entry's reading, per repeat and against both references, rather than under a wider band that would also hide a real 3% on the pod that can see it.

Measured (RTX PRO 6000 Blackwell Server Edition, one pod, ab-gpu-2026-09.jsonl and ab-cpu-2026-09.jsonl, min over interleaved repeats): gpu-tall cuda_depthwise 1M x 100 anchor 0.50 s, old 0.51 s, new 0.29 s; cuda_levelwise 0.93, 0.93, 0.37 s; 16M x 100 cuda_depthwise 3.91, 3.90, 2.59 s; cuda_levelwise 3.99, 3.97, 2.51 s; peak host RSS -4.5% at 1M and -0.3% at 16M. cpu-tall 2M x 128 depthwise 10.58, 11.01, 10.97 s; leafwise 10.83, 10.32, 10.79 s; levelwise 10.08, 10.03, 10.03 s. Parity PASS at the anchor cell (fused 2.60 s against two-step 2.58 s, -0.8%).

Rejected. Re-renting a CPU pod for the cpu-tall cell before tagging: the previous-release reading (-0.3%) is the one that carries this release's change, the anchor reading is one repeat of a quota-metered host, and the CPU-pod session that would settle it was run for 2.1.0 on the same anchor. A wider CPU band on GPU-pod refreshes: the band would then also be the band on the CPU pod, where it is the drift the gate exists to catch. Reading the mean instead of the min: noise on a fixed workload only adds time, so the min is the statistic nearest the engine, and the fix for a lucky repeat is more repeats on a host that resolves the band, not a statistic that averages the host in.

Reopener. A CPU-pod three-arm session that reads the depthwise cell more than 2% over the 1.15.0 anchor, at which point the reading is the engine's and gets its own entry.

Standings: gpu-tall, cpu-tall

128. The device quality axis is gated inside the host's own seed spread per task; the pre-registered fixed 1e-3 band is refuted by the metric's resolution (adopted)¶

Decision. quality-grinsztajn-gpu sweeps the Grinsztajn suite (55 tasks, 3 seeds, the campaign knobs) with bonsai's three CUDA growers alone, ships as its own standings axis on the gpu plane, and is read against quality-grinsztajn pair by pair: same suite, dataset, seed, and growing strategy. A pair holds when its gap, device metric minus host metric, sits inside the host's seed-to-seed spread of that task and strategy (max minus min of the metric over the CPU rows' seeds) plus 1e-4, the host-vs-device prediction tolerance, so a task measured at one seed still has an allowance. The gate runs at claim time and release time, and the quality page tables per grower the mean gap, the pair nearest its allowance, and the spread it is read against. A pair past its task's spread is a device-plane bug until an entry here says otherwise.

Why the fixed band failed. The gate was pre-registered at 1e-3 in metric units, ten times the prediction tolerance, before the first sweep. The sweep (RTX PRO 6000 Blackwell Server Edition, 495 pairs, quality-grinsztajn-gpu-2026-09.jsonl against quality-grinsztajn-2026-09.jsonl) put 33 pairs past it: 15 depthwise, 15 leafwise, 3 levelwise, none at 1e-2, 100 pairs at exactly zero, and the per-grower mean gaps -2.9e-5, -4.1e-5 and -1.2e-5. The largest, -8.1e-3 on wine seed 0 for both depthwise and leafwise, is one task whose three host seeds read 0.4556, 0.3779 and 0.4603, a spread of 8.2e-2, so the device sits inside the host's own scatter by a factor of ten. That holds across the 33: each one is inside its task's seed spread, at ratios from 1.15 (road-safety seed 2, leafwise: 4.7e-3 against 5.4e-3) to 145 (median 10), and over all 495 pairs one exceeds its spread, levelwise on Bike_Sharing_Demand seed 1 at 1.40e-4 against 1.33e-4, by 7e-6. The mechanism is the metric's resolution: these are fits capped at 10k training rows (wine has 2043) scored on held-out rows, the device's fixed-point cells and tie order pick a different cut where two are near-tied, and on a small task one such cut moves the held-out score by more than 1e-3, which is the same mechanism that scatters the host's seeds on those tasks. A fixed band in metric units cannot see the difference between a device that moves quality and a task that resolves coarsely; the host's seed spread is measured on the same rows and carries exactly that difference.

Rejected. A fixed band widened to 1e-2: it would pass this sweep and be blind on the quiet tasks, where the spread is 1e-4 and a 5e-3 device move would be a real regression. Gating the mean gap per grower: the means are 1e-5 and a per-task bug of 5e-3 on one task hides under 164 zeros. Folding the CUDA rows into the CPU quality file: the CPU axis's rows are read by the standings pages as the host's standing, and a device row in that file would need every consumer to filter it; a second axis with a partner name is the same join the ab files already use. Reading the largest gap as the worst pair: under a per-task allowance the largest gap (wine, 8e-3 inside 8e-2) is the pair that matters least, and the pair pressing the gate is the one nearest its allowance, so the table shows that one with its spread.

Reopener. A sweep where a pair exceeds its task's spread by more than 1e-4, at which point it is a device-plane bug to be found, not a band to be widened; or a suite whose tasks are single-seeded, where the floor alone gates and this entry's allowance is re-derived from a seed sweep of that suite.

129. The GPU plane has its own Grinsztajn standings, every library on its GPU build; an arm whose library does not take the campaign knobs on its device is measured but not ranked (adopted)¶

Decision. quality-grinsztajn-gpu sweeps the suite (55 tasks, 3 seeds, the campaign knobs) with six arms on one GPU host: bonsai's three CUDA growers and the three references on their own GPU builds, xgboost device=cuda, lightgbm device_type=cuda, catboost task_type=GPU at border_count=254. The quality page and the README rank the GPU plane by the rule the CPU table already uses (decision 68: mean over seeds per task, best variant per library, average rank across tasks, a win is rank exactly 1), so the two tables read in the same words and each ranks one plane against its own peers. The drift table reads every device arm against its own library's CPU rows task by task, and only bonsai's three arms are gated by decision 128's allowance; a reference's gap is shown with the same columns and verdict wording but never fails a build, because its distance from its own CPU build is that library's to explain. An arm whose library does not apply the campaign knobs on its device is swept and read in the drift table but left out of the ranking, with the rank it would have taken stated under the table (QUALITY_UNRANKED in scripts/check_standings.py); a rank at matched knobs cannot hold an arm at other knobs. A reference arm that silently falls back to the CPU fails its row instead of ranking a host fit under a device label (xgboost's post-fit save_config check, the guard the perf runner already carries).

Measured (RTX PRO 6000 Blackwell Server Edition, driver 595.91.07, one pod, xgboost 3.3.0, lightgbm 4.7.0, catboost 1.2.10, quality-grinsztajn-gpu-2026-09.jsonl, 990 rows, 0 errors). The ranked GPU table: bonsai 1.24 with 42 outright wins, xgboost 2.29 with 7, catboost 2.47 with 6; per suite bonsai 1.00 on cat_clf, 1.31 on cat_reg, 1.33 on num_clf, 1.20 on num_reg. The CPU table on quality-grinsztajn-2026-09.jsonl reads bonsai 1.49 with 35, lightgbm 2.40 with 6, xgboost 2.93 with 5, catboost 3.18 with 9. Ranked among all six arms the table would read lightgbm 1.71 with 38, bonsai 1.93 with 11, xgboost 3.11 with 2, catboost 3.25 with 4, and the drift table is where that reading falls apart: lgbm_cuda against lightgbm's own CPU rows over 165 pairs has mean gap +6.5e-3 and worst +2.6e-2 (compass seed 0, 0.8008 to 0.8270) against a host spread of 8.3e-3, with 123 pairs past 1e-3 in magnitude, where bonsai's three arms read -2.9e-5, -4.1e-5 and -1.2e-5, xgboost's -5.8e-4 and catboost's +2.7e-4. That is not a device rounding effect. On the laptop, lightgbm 4.6.0 and 4.7.0 CPU give identical numbers to four decimals on wine_quality, compass and Mercedes_Benz_Greener_Manufacturing, so the version step is not the cause; and setting max_depth=-1 alone on the CPU build reproduces the CUDA numbers (wine_quality seed 0: 0.5399 against CUDA 0.5363, host as-is 0.4992; compass seed 0: 0.8271 against 0.8270, host 0.8008; Mercedes seed 0: 0.5642 against 0.5586, host 0.5897), where deterministic=False, lambda_l2=0, min_data_in_leaf=1, max_bin=63 and force_col_wise each leave the as-is value. lightgbm 4.7.0's source agrees: the only max_depth check is SerialTreeLearner::BeforeFindBestSplit (src/treelearner/serial_tree_learner.cpp, if (config_->max_depth > 0) against leaf_depth), and CUDASingleGPUTreeLearner::Train loops num_leaves - 1 splits without calling it; cuda_single_gpu_tree_learner.cpp, cuda_best_split_finder.cpp, cuda_best_split_finder.cu and cuda_leaf_splits.cpp contain no occurrence of depth. At the campaign knobs (depth 6, num_leaves 63) the CUDA arm therefore grows 63 leaves at any depth where every other arm is capped at depth 6, which is a different model class, not a device build of the same one. Its rows stay in the file and in the drift table, which is where a lightgbm release that applies the cap will show the arm holding its host spread.

Rejected. Ranking bonsai's CUDA arms against the references' CPU rows: the first GPU file did that by construction (bonsai's three CUDA growers alone, 495 rows) and a table built from it compares a device fit to a host fit, which is the drift question, not a standings question. Ranking lightgbm's CUDA arm at its own capacity: it would crown a 63-leaf unbounded-depth learner over five depth-6 learners and call it a GPU build effect; the mean rank the rule declines (1.71) is printed under the table so the reader has the number with the reason. Restating the knobs so lightgbm's CUDA arm takes them (a num_leaves that emulates depth 6): there is no such value on a leafwise learner, and it would move lightgbm's CPU rows off the campaign the other libraries run. Gating the references' device drift: catboost's GPU build quantises with border_count capped at 254, lightgbm's CUDA learner is its own tree learner, and a gate on either would fail bonsai's release for someone else's regression. One file holding both planes: the CPU axis's rows are the host's standing and every consumer would filter on a plane column; a second axis with a partner map is the join the ab files already use (decision 128).

Reopener. A lightgbm release whose CUDA learner applies max_depth, seen as the drift table's lgbm_cuda row holding its host spread, at which point the arm leaves QUALITY_UNRANKED and enters the ranking; a reference's device gap past its own host spread on a task where bonsai's arms hold, reported with the task named and the library's tracker cited in the PR, never gated; or a fourth reference library, which enters the partner map in scripts/check_standings.py with its CPU spelling and its own device arm in the bench registry.

Standings: quality-grinsztajn-gpu

130. bonsai leafwise meets lightgbm's CUDA learner at its own cap: a leaf-capped regime on its own GPU axis, one parameter change, ranked head to head (adopted)¶

Decision. python -m bonsai.bench.grinsztajn --device cuda --regime leaf-capped sweeps the suite (55 tasks, 3 seeds) with the campaign's 63 leaves under a depth cap of 62, the deepest a 63-leaf tree can reach, so the leaf count is the only binding cap on both learners. That is one parameter change on bonsai's leafwise grower, max_depth=62 in place of 6, and lightgbm's own max_depth=-1 fits the same trees (wine_quality seed 0 reads 0.5399 on the CPU at both). Only the learners a leaf count alone can cap run in the regime, bonsai_cuda_leafwise and lgbm_cuda on the GPU, bonsai_lw and lgbm on the CPU; a depthwise or symmetric tree at depth 62 is a different experiment, so bonsai's other growers, xgboost, and catboost are absent by design. The rows ship as quality-grinsztajn-leaf-capped-gpu, an axis of its own on the gpu plane, ranked by the standings rule (decision 68) with lgbm_cuda in the ranking, because under these knobs the arm takes every knob it is given. The quality page carries the ranked table, the per-suite table, and a task-by-task head-to-head table (wins, mean gap, widest lead and deficit, gap as bonsai's mean over seeds minus lightgbm's); the README carries the two-line table. The regime is never merged into the campaign standings and never drift-paired: its question is head to head on one device, not device against host.

Measured (RTX PRO 6000 Blackwell Server Edition, driver 595.91.07, one pod, lightgbm 4.7.0, quality-grinsztajn-leaf-capped-gpu-2026-09.jsonl, 330 rows, 0 errors). Ranked: bonsai 1.42 with 32 outright wins, lightgbm 1.58 with 23; per suite bonsai leads on cat_clf (1.29 against 1.71, 5 tasks to 2), num_clf (1.40 against 1.60, 9 to 6), and num_reg (1.35 against 1.65, 13 to 7), lightgbm leads on cat_reg (1.38 against 1.62, 8 to 5). Mean gap +0.0006 over the 55 tasks, median absolute gap 0.0006, widest lead +0.0256 (sulfur, num_reg), widest deficit -0.0130 (house_16H, num_reg). What the parameter change bought: paired on the same 55 tasks under the campaign knobs (quality-grinsztajn-gpu-2026-09.jsonl), lightgbm's uncapped trees beat bonsai leafwise at depth 6 on 41 tasks to 14 with a mean gap of -0.0056; freeing the depth cap moved bonsai leafwise by +0.0063 on average (up on 38 tasks, from -0.0311 to +0.0436) and flipped the count to 32 to 23. Lightgbm's own rows moved by 0.00006 on average and at most 0.0011 between the two sweeps, one at max_depth=6 and one at 62: the on-GPU confirmation of decision 129, the knob does nothing to its CUDA learner. On 54 of the 55 tasks the gap between the two means sits inside the wider arm's own spread over its three seeds (medical_charges is the exception, a lightgbm lead of 0.0005 r2 against spreads of 0.0005 on both sides, at the metric's resolution); the ranking is real but the margin is thin, and the honest summary is parity with a lean to bonsai.

Rejected. Ranking lgbm_cuda inside the campaign GPU table at its own capacity (decision 129's rejected alternative, unchanged). Setting max_depth=-1 on bonsai: the leafwise grower takes a depth cap, and 62 is the value at which the cap cannot bind for 63 leaves, so the equivalence is exact and stated rather than a sentinel to explain. Adding bonsai's depthwise or levelwise growers to the regime: a depth-62 depthwise tree has no leaf cap to meet and would measure something else. Running the regime on the CPU plane as a second axis: it would answer the drift question the regime is not asking; the CPU arms exist so a laptop can check the equivalence, not for a table.

Reopener. A lightgbm release whose CUDA learner applies max_depth, at which point the campaign table ranks it (decision 129's reopener) and this regime becomes a second view rather than the only one; a third learner a leaf count alone can cap on its GPU build, which enters the regime's arm map in python/bonsai/bench/grinsztajn.py with a test pinning its knobs; or a gap on a task past its seed spread on either side, which is a quality finding for that learner, named in the ledger, never a gate.

Standings: quality-grinsztajn-leaf-capped-gpu

131. One wait per leaf expansion: declined by measurement (adopted)¶

Decision. The CUDA leafwise plane keeps decision 126's round: partition, a wait for the left count, fill, find, a wait for the best split. Lever 28 of that decision, resolving each child's histogram slot on device so partition, fill, find and reduce enqueue together behind one wait, was built and measured (PR #478, closed unmerged, branch perf/leaf-fused-wait at 8baad6a on origin). It was priced at 0.2 to 0.4 s of the tall leafwise train from decision 126's residual; it measured 0.04 s on the standings card, inside one arm's spread over three repeats, and nothing on the L40S.

Measured (RTX PRO 6000 Blackwell Server Edition, driver 595.91.07, one pod, interleaved k0/fw/fw/k0/k0/fw, min over repeats, train_s; k0 is main at 681b4f2, fw is the branch). Tall leafwise (gpu-tall, 16777216 x 128) 2.969 against 2.925, -1.5% or 44 ms, with k0's three repeats spanning 37 ms (2.969, 2.994, 3.006) and fw's 58 ms (2.925, 2.981, 2.983); fit 3.649 against 3.606. Wide leafwise (gpu-wide, 131072 x 16384, two repeats) 5.660 against 5.636, -0.4%; fit 7.474 against 7.435. Controls: tall depthwise 2.252 against 2.260, tall levelwise 2.130 against 2.129; ingest flat. On an L40S (driver 570.195.03, one pod, same interleave, never compared across pods): tall leafwise 5.616 against 5.631, +0.3%, with k0's repeats spanning 125 ms; depthwise 4.729 against 4.732, levelwise 4.306 against 4.313; the wide spec is refused by the leaf plane on that card's 48 GB on both arms (the pool would exceed a quarter of free memory), so it has no wide number. The buckets say why nothing moved. On main the leafwise host reads partition 0.89 to 1.01 s, find 1.4 to 1.5 s and populate 0.46 to 0.50 s on the standings card, and of the 2.4 s of leaf rounds the engine's gpu_wait already accounts for 1.1 to 1.2 s against 1.16 to 1.28 s of cuda_gpu: the second wait was a wait on a busy device, not on an idle one, because lever 26 had already queued the fill behind the partition. The branch folds the two host buckets into one expand bucket reading 2.39 to 2.44 s, with cuda_gpu at 2.16 to 2.19 s and gpu_wait at 0.04 to 0.05 s: the same kernel time, now behind one wait instead of two. On the L40S the shape is the same (k0 gpu_wait 2.5 s of 4.5 s of rounds, fw expand 4.47 to 4.55 s with cuda_gpu 4.08 to 4.17 s). The 0.2 to 0.4 s price assumed device idle between the fill and the find; there was none left to remove. Engagement was proven by a direct worker run on both cards (the fused line prints twice on fw, for the micro-fit and the fit, and never on k0, which prints decision 126's lever 26 and 27 lines); r2_test is equal on every record (0.8797 tall depthwise and leafwise, 0.877 levelwise, 0.8597 wide); the three device growers hash equal across arms on both cards; the branch's [cuda] suite passes in both finder modes on both pods and on sm_87.

What the branch also changed, kept out of the tree with it. The fused round reports one host bucket, expand, in place of partition and find, and its part_stage counter includes the round's fill and find launches: a reader of a profile from that branch is looking at different buckets, which is why the comparison above is read as sums. Reordering the driver to inherit the parent's state before the partition returned exposed SplitInput::totals() gating the cached sums on a positive row count; a device child's sums arrive before its count in that order, so the branch gates on the histogram being present instead and pins it with a test. Main's order propagates after the partition and never trips the gate, so no fix ships with this entry; the branch carries it for the day the order is revisited.

Rejected. Merging the lever for its 44 ms: the number is inside the spread and the branch is 368 lines added over 14 files across the engine seam, the leaf driver and the profiler, which is a cost the residual does not pay for. Re-measuring at a larger cell: the buckets on both cards already put the device, not the host, on the critical path of the leaf rounds, and a larger cell moves more work to the device side. Keeping the totals() gate change alone: it changes nothing observable on main.

Reopener. A card or a cell where main's leafwise gpu_wait reads near zero over the leaf rounds, meaning the host and not the device paces the expansion; then the branch at 8baad6a is the implementation to re-measure same-pod, and the profile-bucket change it carries is the first thing to state in the entry that adopts it. The device-bound residual itself (2.2 s of kernel time across the leaf rounds on the standings card, find the largest share) is the next target, and it is a kernel question, not a host-latency one.

132. A leaf budget that cannot bind takes the depthwise plane (adopted)¶

Decision. A leafwise configuration whose budget cannot bind (max_leaves of 0, or of 2^max_depth and above) is grown by the depthwise grower of the same engine. resolve_dispatch(cfg) is the one home for the rule; make_booster, save_dispatch and load_dispatch search the dispatch table with it. cfg.dispatch and the saved model keep the name the user wrote, so a warm start, Params.from_model and a loaded model see leafwise unchanged and a load searches with the same routed name the save did. BONSAI_GROW_PROFILE=1 prints the route once per booster.

Why. A depth-D tree has at most 2^D leaves, so a budget of 2^D or above never stops an expansion: every leaf with positive gain splits, which is exactly the set the depthwise grower splits in one level round. Leafwise grew that tree one leaf at a time, paying a find, a partition and a fill round per leaf (255 rounds at depth 8 against 8), and the leaf pool it sizes for 2^D live slots refused the wide cell on a 48 GB part where the depthwise plane fits. Every gpu standings leafwise cell uses num_leaves = 2^depth, so the leafwise rows were paying the per-leaf tax to grow the depthwise tree. On the host the two planes agree to float rounding across the options that could have separated them: 4096 x 6 at depth 5 with a budget of 32 or 0, with and without min_data_in_leaf, monotone bounds, interaction groups and a feature draw, every sampled leaf value within 1e-5 and the same leaf count and depth; a budget of 15 under depth 4 still binds (16 against 15 leaves). At 100k x 32 and depth 6 the planes differ 4.8e-7 at 64 leaves and 0.34 at 63.

Measurement. Same-pod on one RTX PRO 6000 Blackwell (US-NC-1), main 681b4f2 against aad4e66, interleaved reps, min over reps, fit total in seconds; the depthwise rows are the control.

cell	plane	main	routed	delta
tall 16M x 128	leafwise	3.694	2.971	-19.6%
tall 16M x 128	depthwise	2.968	2.974	+0.2%
wide 131k x 16384	leafwise	7.521	6.141	-18.3%
wide 131k x 16384	depthwise	6.187	6.172	-0.2%
extreme 16M x 1024	leafwise	15.357	14.843	-3.3%
extreme 16M x 1024	depthwise	14.841	14.982	+1.0%

On train alone the leafwise rows read -24.4% tall, -21.8% wide and -4.7% extreme (2.993 to 2.262, 5.688 to 4.450, 11.352 to 10.823 s). r2 is identical to four decimals across both arms and both planes at every cell (0.8797, 0.8597, 0.8788), and the three device model hashes (depthwise, leafwise, levelwise) are equal between the arms. The tall and wide reps were three and two, extreme one, which is why its 1% is read as the spread and not as a cost. The routed leafwise row lands on its depthwise row because it is the depthwise plane: on every gpu standings cell the leafwise row now measures depthwise, and the results pages say so. The leaf-capped regime (decision 130, 63 leaves under depth 62) is unaffected because that budget binds.

Rejected alternatives. Batching a level's worth of expansions inside the leafwise grower when the budget cannot bind: it rebuilds the depthwise plane's level transaction a second time behind the leaf interface, and the depthwise plane already exists. Speculative batched expansion for a binding budget (grow a level, then prune back to the budget): the pruned tree is not the leafwise tree, since a leaf that would have been expanded ahead of its siblings under the gain order can lose its slot to a sibling split first, so it changes the model and needs its own quality study. Refusing a budget that cannot bind: a user who writes max_leaves=256 at depth 8 has asked for the depthwise tree, and refusing is the answer nobody wants.

Reopener. A binding budget's per-leaf residual (the leaf plane's find, partition and fill rounds against the depthwise plane's level rounds at the same knobs) is the remaining gap and is measured, not routed: when a leafwise cell with a binding budget lands within the depthwise row's noise, the leaf plane has closed it; while it does not, the leaf plane is the target.

133. Device ingest above 8 GiB stages raw chunks through a pinned ring (adopted)¶

Decision. When the raw float matrix exceeds 8 GiB, cuda_ingest copies each 64 MiB chunk into one of three pinned host slots with a parallel host memcpy, uploads it with cudaMemcpyAsync on the slot's own stream, bins it on that stream and records a fence; a slot is reused only after its fence has been waited on. Below 8 GiB each chunk uploads with a pageable cudaMemcpy through one device slot on the default stream, the path that shipped before. Both overloads (row-major features_view, feature-major ColumnBatch) share the ring, and BONSAI_CUDA_INGEST_RING forces it below the threshold for the invariant test. Binning per element is unchanged, so the bin planes are byte-identical to the host fill on both paths (the chunk-reuse invariant pins it across four chunks row-major and sixteen columns feature-major, with and without the ring).

Why. Every gpu standings cell paid ingest_dbin for a pageable upload: extreme 3.58 s of a 14.98 s fit (68.7 GB raw at 19.2 GB/s), tall 0.39 of 2.99, wide 0.53 of 6.17, read from the branch arm of decision 132's session on one RTX PRO 6000. Pinned staging was priced at the extreme cell's copy rate against the tall cell's, about 12% of the extreme fit if the extreme's upload reached the tall's rate.

Measurement. Same-pod on one RTX PRO 6000 Blackwell Server Edition (US-NC-1, driver 595.91.07), main 681b4f2 against the ring at f159ece (before the threshold), interleaved reps, min over reps, seconds; ingest_dbin from the ingest profile, fit total from the bench row, depthwise plane, r2_test identical per cell on both arms (0.8788, 0.8797, 0.8597) and the three CUDA model hashes equal.

cell	ingest_dbin main	ingest_dbin ring	fit main	fit ring	fit delta
tall 16M x 128 (8 GiB)	0.32	0.33	2.908	2.967	+2.0%
wide 131072 x 16384 (8 GiB)	0.43	0.42	6.162	6.251	+1.4%
extreme 16M x 1024 (64 GiB)	3.57	2.78	14.871	14.121	-5.0%

The extreme cell gains 0.79 s of ingest (-22%), half the priced figure. At 8 GiB the ring's rate matched the pageable rate on both cells (tall 27 GB/s on both arms, wide 20 GB/s on both), so those cells had nothing to take from it; at 64 GiB main's upload fell to 19 GB/s and the ring held 25 GB/s. Their fit deltas sit inside the rep spread (main's tall ingest reps span 0.66 to 0.78 s). A 128 MiB engagement cell (1M x 32, 10 rounds) read ingest_dbin 0.01 s pageable and 0.06 s through the ring, three 64 MiB pinned slots allocated for a two-chunk matrix; that 0.05 s is why the ring engages only above 8 GiB, where the cells that measured flat keep the shipped path and the cell that measured a gain takes the ring.

Rejected alternatives. cudaHostRegister of the caller's array in place: page-locking the source is the same work the slots amortise, paid once per byte for a single upload. One pinned slot with no overlap: the DMA rate without the host memcpy running under it. Larger chunks: 64 MiB already amortises launch and fence cost, and three slots hold 192 MiB of pinned memory. A ring at every size: the 128 MiB cell above. Priced dead on this session's buckets, not built: fp32 gain math in the finders, which would change the model: the tall finder is 0.03 s of the fit, and the wide finder (1.6 s of 6.16) already screens in fp32 and scores only the survivors in fp64, with its residual attributed to the warp latency chain rather than to bytes (decision 126), so no priced win remains that is worth a model change; a further split of the fill's shared atomics (the shipped fill already adds each 64-bit cell as two 32-bit ATOMS.ADD with a carry, on sm_120 as on sm_89, and the sm_120 SASS carries no CAST.SPIN, so the fill is at its atomic ceiling); a count plane for unit-hessian objectives, already shipped in the tile fill (decision 126).

Reopener. A host whose pageable upload reads below 20 GB/s at 8 GiB, where the threshold moves down on a same-pod measurement of that cell; a container whose pinned budget refuses 192 MiB, which is a fallback to the pageable path, not a redesign; a pinned slot cache that survives across ingests, which removes the allocation the 128 MiB cell paid and would let the threshold go, priced only once a workload of repeated small device fits shows the 0.05 s in its profile.

134. The fixed-point exponent stays at decision 124's sum bound; capping the row below 2^27 to skip the high-word atomic is declined by measurement (adopted)¶

Decision. The device histogram keeps e = min(61 - ilogb(max|g| * n), 126) from decision 124. Lever 30, capping each quantised row below 2^27 so that the shared fill's second 32-bit atomic fires only on a low-word wrap, was built and measured (branch perf/fixed-point-row-bits at 52ff11b on origin, one file, ten lines). It was priced against the word-split's own record: the split took the 16M-row root fill from 1.09 to 0.68 s on an L40S by replacing a compare-and-swap loop with two native adds, and with the sum-bound exponent the largest row lands near 2^61 / n (2^37 at 16M rows, 2^44 on the micro-fit's scale line), so the high add fired on every visit; removing it bounded the saving by the difference between one and two native adds per visit, a share of populate, which reads 0.43 s of a 2.98 s tall fit and 1.60 s of a 14.8 s extreme fit on the standings card. It measured at most two hundredths on populate and nothing on the fit.

Measured (RTX PRO 6000 Blackwell Server Edition, driver 595.91.07, one pod, arms interleaved k0/pr/pr/k0 per cell, min over repeats, seconds; k0 is main at 681b4f2, pr is the branch; buckets from the grow and engine profiles, fit is fit_s). Tall depthwise (16777216 x 128, three repeats each) fit 2.977 against 2.957, -0.7%, with k0's repeats spanning 6 ms (2.977, 2.980, 2.983) and pr's 36 ms (2.957, 2.965, 2.993); populate 0.43 against 0.42, find 1.01 both, gpu_wait 0.96 against 0.95, partition 0.74 both. Wide depthwise (131072 x 16384, two repeats) fit 6.107 against 6.256, +2.4%, all of it in ingest (dbin 0.44 against 0.53, with k0's second repeat reading 0.61): train 4.455 against 4.434, populate 0.28 against 0.27, find 4.31 against 4.30, gpu_wait 2.49 against 2.50. Extreme depthwise (16777216 x 1024, two repeats) fit 14.809 against 14.851, +0.3%; train 10.823 against 10.743; populate 1.60 and 1.60 against 1.64 and 1.51, so the -0.09 s minimum is one repeat against a 0.13 s spread on its own arm; find 8.43, gpu_wait 8.28 and partition 0.74 on both arms. Tall leafwise (two repeats) fit 3.663 against 3.675, train 2.965 against 2.968, populate 0.44 against 0.46, find 1.50 against 1.47. r2_test is equal to five digits on every record (0.8797 tall, 0.8597 wide, 0.8788 extreme). Engagement was proven by the profile's scale line on a direct worker run: the micro-fit prints scale_g=2^37 and the fit 2^44 on k0, 2^22 on pr, and the hash cell prints 2^41 against 2^25. The branch's [cuda] suite passes on sm_120 (382217 assertions) and sm_87, and its sm_120 SASS carries 398 ATOMS.ADD and no CAST.SPIN. The reading is that the shared add count was not what the fill was paying for: halving the atomics per visit, proven by the exponent, moved populate by one to two hundredths at three cells, inside the counter's resolution, while the extreme cell's fill of 1.60 s did not move outside one arm's spread. The 0.41 s the word-split saved came from the instruction class (a native add against a compare-and-swap loop), not from the count, and the count is not on the critical path at any standings cell.

What the branch also changed, kept out of the tree with it. The per-row rounding coarsens from max|g| * n / 2^62 to max|g| / 2^28, the subset-sum bound shrinks from 2^62 to 2^59 for 2^32 rows, and the model bytes move at the float32 ulp on some inputs: the three device growers hash equal across arms on scripts/model_hash.py's cell, and a 200000 x 40 fit of 20 trees at depth 8 differs in 46 of 5000 predictions by 5.96e-8, one ulp. The 1e-4 host parity bound and bit-reproducibility on one device hold on both arms, but a change to what the device fit computes is a wire change for the GPU standings hash set, which a refuted lever has no reason to carry.

Rejected. Merging the lever for its hundredths: the populate deltas sit at the profile counter's two-decimal resolution and the fit deltas inside spread, against a change that moves model bytes. Measuring on an L40S as well: the instruction class is the same (ATOMS.ADD on sm_89 and sm_120, per decision 124's status banner) and the standings card is where the claim would have been made. A middle exponent that keeps the high word zero for positive rows only: it has the same count on every visit that matters and the same byte movement.

Reopener. A cell or a card where populate is the largest bucket of the fit and its SASS shows two ATOMS.ADD per visit, at which point the branch at 52ff11b is the ten-line implementation to re-measure same-pod, with the byte movement above stated in the entry that adopts it. On the standings card the extreme cell's residual is gpu_wait at 8.28 s of 14.8 s behind a find of 8.43 s, so the next lever is in the finder, not the fill.

135. Each node's fill is chunked by its own rows, at the rows per block the largest node gets (adopted)¶

Decision. launch_hist keeps its grid (tiles x nodes x chunks, the chunk count sized from the level's largest node) and passes the kernels chunk_rows = max_rows / n_chunks, the rows one block of the largest node covers. Every block computes its node's own chunk count, clamp(ceil(count / chunk_rows), 1, n_chunks), exits when its z index is past that count, and strides the node's rows by that count. The largest node runs exactly the blocks it ran before, so its fill is bit-identical; a smaller node runs as many blocks as its rows fill at the same rows per block, and the rest of its grid slice exits before the shared-memory zero. The cells are int64 fixed point (decision 124), so the sum is the same integer in any block order and the model bytes cannot move: model_hash.py --grower cuda_depthwise reads e1a5391a7beea349 on both arms, r2 equal per cell. The per-level profile counters (rows_lN, blocks_lN on the cuda-level-decomp line, added in this change) now count active blocks, computed on the host from the level's row counts; a blocks_lN read from before this change counted grid blocks.

Why. The extreme cell (16M x 1024) spent 8.25 s of a 14.9 s fit in adv_hist, and the per-level split showed where: rows filled per level are flat at about 4.65M per tree (the smaller sibling at every split), yet the fill's time rose from 5.5 ms per tree at level 1 to 20.1 at level 7, and the block count rose 26x (3973 to 103823), because the chunk count was sized from the largest node and every block with any rows ran, at 45 rows per block by level 7 against 1171 at level 1. Each block zeroes and merges 8160 shared cells (16 features x 2 x 255 bins) whatever its row count, so level 7 merged 847M cells per tree to fill the same rows level 1 filled through 32M. Priced from a linear fit across the levels at 18 us per million merged cells, the lever was worth about 4 s of the extreme fit; the measurement below says the fit across levels overstated the merge and the deep levels carry a second cost.

Measurement. Same-pod on one RTX PRO 6000 Blackwell Server Edition (US-NC-1), 1986d78 (main plus the counters) against b8d7ef0, interleaved reps, min over two reps, seconds; find and adv_hist from the round decomposition, hist_l7 from the level decomposition, fit total from the bench row.

cell	fit A	fit B	fit delta	find A	find B	adv_hist A	adv_hist B	hist_l7 A	hist_l7 B
extreme 16M x 1024 dw	14.87	13.79	-7.3%	8.43	7.34	8.25	7.17	2.01	1.61
tall 16M x 128 dw	2.92	2.85	-2.3%	1.02	0.98	0.96	0.92	0.24	0.21
wide 131072 x 16384 dw	6.04	6.03	-0.2%	4.30	4.29	2.00	1.99	0.76	0.76
tall 16M x 128 lw	3.64	3.69	+1.3%	1.53	1.62

The extreme cell per level, blocks and merged cells per tree, min hist seconds over 100 trees:

level	rows/tree	blocks A	blocks B	Mcells A	Mcells B	hist A	hist B	delta
1	4651350	3973	3973	32	32	0.55	0.55	0%
2	4215091	7540	5412	62	44	0.61	0.60	-2%
3	4450903	14249	6891	116	56	0.79	0.76	-4%
4	4683568	25984	8198	212	67	1.10	0.99	-10%
5	4739758	43899	9362	358	76	1.44	1.23	-15%
6	4650977	68608	10185	560	83	1.74	1.43	-18%
7	4649717	103823	11596	847	95	2.01	1.61	-20%

Level 1 is unchanged in blocks and time, the largest-node identity the design promised. Level 7 dropped 752M merged cells per tree and 4.0 ms, so the merge costs about 5 us per million cells, a third of what the fit across levels implied; the other two thirds of that slope was a cost that rises with depth for a different reason. After the lever, level 7 still reads 16.1 ms per tree against level 1's 5.5 at the same rows and a comparable cell count (95M against 32M), so about 10 ms per tree of the deep-level fill is the row loop itself, three times the level-1 row loop for the same rows spread over the level's node segments instead of one. That is the residual and the next target; its mechanism is not measured here. On the tall cell the same shape at a tenth of the features (levels 5 to 7 read -6, -10, -12%) is 0.04 s of adv_hist and the fit moves at the edge of its rep spread (A 2.92 and 2.98, B 2.85 and 2.90). The wide cell is flat by construction: from level 5 every counted node already ran one chunk (blocks identical on both arms). Its level-7 cost, 7.6 ms per tree, is not attributed by these counters: the level timer spans the tiled fill and hist_small_kernel together, while rows_lN and blocks_lN carry only the nodes at or above the 512-row cutoff, so the 27k counted rows sit in about 16 nodes (16507 blocks over 1024 tiles) and the rest of the level's filled rows go through the small-node kernel uncounted. At 5 us per million cells the 135M counted cells are under 1 ms of the 7.6, so most of that cost is somewhere the counters do not reach. The leafwise fill takes the same kernels and its fit reads +0.05 s inside its own spread (A 3.64 and 3.75, B 3.69 and 3.78).

Rejected alternatives. A larger k_fill_chunk_rows (fewer chunks for every node): it shrinks the largest node's chunk count with the rest, and that count is what fills the SMs at the top levels, where one or two nodes carry the level. One launch per node sized to its own rows: 128 launches at level 7 per tree, and the largest node's chunking would move with it; the per-block exit gets the per-node count from the same grid at no host cost. Routing more nodes to hist_small_kernel by raising its 512-row cutoff: the 2026-08-17 sweep measured every higher cutoff worse at every cell, and the node this change leaves expensive at level 7 averages over 30k rows (4.65M rows over at most 128 nodes), far above any cutoff that sweep considered. Counting active blocks on the device: the host has the level's row counts already (row_counts.host), and the count is profile-only.

Reopener. The deep-level row loop, 10 ms per tree at level 7 on the extreme cell against 5.5 at level 1 for the same rows, which needs a counter that separates the row loop from the zero-and-merge before it is designed against; and the wide cell's deep levels, where the small-node kernel's rows are not counted and the level timer spans both kernels: a small_rows_lN counter and a timer split between the two launches come first, before any lever is designed for that cell.

136. The wide cell's deep-level fill is the small-node kernel; the extreme cell's is the tiled row loop (adopted)¶

Decision. The level decomposition splits its fill timer: small_lN is the device time of the hist_small_kernel launch and small_rows_lN the row sum of the nodes under the 512-row cutoff, so hist_lN - small_lN is the tiled kernels' share and rows_lN + small_rows_lN the level's rows. Read once on the three ledger cells, the split attributes the two deep-level costs decision 135 left open to different kernels, so they are two levers with two designs. On the wide cell (131072 x 16384) the small-node kernel carries 0.84 s of the 2.00 s adv_hist, and 5.1 of the 7.6 ms per tree at level 7, for under 10k rows per tree; on the extreme (16M x 1024) and tall (16M x 128) cells its share is 0.00 s at every level, so the deep-level residual there sits in the tiled kernel's row loop. The wide small-node path is the next fill lever, priced at 0.84 s of small-kernel time plus the 0.50 s wide adv_memset it shares a slot with, out of a 6.29 s fit; the extreme row loop follows, once a counter separates it from the zero-and-merge.

Why. The wide cell's level 7 read 7.6 ms per tree with 27k counted rows in about 16 nodes; the counted merge cells price at under 1 ms, and the rest of the level's rows sat in nodes the counters did not see, in a launch the timer spanned but did not separate. Each hist_small_kernel block owns one (16-feature tile, node) pair and adds every visit straight into the global slot with two int64 atomics, one per gradient component, with no shared stage and no merge; whether that path or the tiled kernel's merge carried the level could not be read from a timer over both.

Measurement. One RTX PRO 6000 Blackwell Server Edition (US-NC-1), head 8f60a76, one arm (the counters change no model byte; model_hash.py --grower cuda_depthwise on the pod build read e1a5391a7beea349), min over two reps, 100 trees. Fit totals 13.62 s extreme, 2.99 s tall, 6.29 s wide. Round decomposition (seconds over the fit): extreme root_sums 1.44, root_hist 1.36, adv_memset 0.03, adv_hist 7.16; tall 0.26, 0.19, 0.01, 0.92; wide 0.20, 0.20, 0.50, 2.00.

Wide, per level, rows and blocks per tree, seconds over 100 trees, tiled = hist - small:

level	tiled rows/tree	blocks/tree	small rows/tree	hist s	small s	tiled ms/tree	small ms/tree
1	32239	1454	0	0.08	0.00	0.8	0.0
2	35721	2427	7	0.10	0.00	1.0	0.0
3	35696	3953	75	0.13	0.01	1.2	0.1
4	34262	6820	268	0.18	0.02	1.6	0.2
5	33772	10670	1001	0.29	0.08	2.1	0.8
6	31321	14715	2624	0.46	0.22	2.4	2.2
7	27026	16507	5894	0.76	0.51	2.5	5.1

The small kernel's cost per row is flat where it is measurable, 0.80, 0.84 and 0.87 us at levels 5, 6 and 7, against 0.025 us per tiled row at level 1 and 0.09 at level 7 (merge included). At level 7 the 5894 small rows per tree are 18% of the level's filled rows and 67% of its time. Counting the atomics the kernel issues, 5894 rows x 16384 features x 2 components is 193M global int64 atomic adds per tree in 5.1 ms, 38 G per second; what bounds that rate (atomic unit throughput, or the DRAM sectors a random 8-byte read-modify-write touches across a slot 16384 features wide) is not measured here.

Extreme, the same split:

level	tiled rows/tree	blocks/tree	small rows/tree	hist s	tiled ms/tree
1	4651350	3973	0	0.55	5.5
2	4215091	5412	0	0.60	6.0
3	4450903	6891	0	0.76	7.6
4	4683568	8198	0	0.99	9.9
5	4739758	9362	2	1.22	12.2
6	4650977	10185	48	1.43	14.3
7	4649717	11596	395	1.61	16.1

Tall reads the same shape: small rows 0 through level 5, 57 and 412 per tree at levels 6 and 7, small seconds 0.00 at every level, hist 0.07 s at level 1 rising to 0.21 at level 7 for flat rows (4.3M to 4.8M per tree). On both cells the small-node kernel never reaches a hundredth of a second, so the rise from 5.5 to 16.1 ms per tree at equal rows is inside the tiled kernel, and decision 135 found the merge accounts for a third of it.

The wide adv_memset is the second bucket the read exposes: 0.50 s on wide against 0.03 s on extreme and 0.01 s on tall. zero_slots_kernel zeroes the two child slots of every split node at every level, each slot the full feature width times the bin stride, so its bytes per level scale with the feature count (16384 against 1024 and 128) and not with the rows; on wide that is 5 ms per tree, more than any single level's fill below level 7.

Rejected alternatives. Designing the wide lever against the level timer, on the assumption that the tiled merge carried level 7: the split shows the tiled share (row loop and merge) at 2.5 ms per tree and the small kernel at twice that. A same-pod A/B for this commit: the counters record only under the profile flag and change no model byte, so there is no arm to compare against and the read is one arm's attribution. Pricing the small kernel from the round decomposition alone: adv_hist at 2.00 s does not say which of two launches per level carries it.

Reopener. The wide small-node path: a node under 512 rows is filled into a slot 16384 features wide by global atomics, one block per tile with no shared stage, and 0.84 s of small-kernel time plus a share of the 0.50 s memset is the ceiling on the saving (the small-kernel time alone is 13% of the 6.29 s fit), less whatever the design costs. The design is chosen by a probe that separates atomic throughput from the sector footprint, on a pod, not in the tree. The extreme tiled row loop: 16.1 ms per tree at level 7 against 5.5 at level 1 for the same rows, of which the merge is a third; a counter that splits the row loop from the zero-and-merge comes before any design, as decision 135's did.

137. A small node's block owns its slot, so it stores the tile whole and the level memset skips that slot (adopted)¶

Decision. A node under the 512-row cutoff has exactly one block per (16-feature tile, node), so that block is the only writer of its slot's cells. On the tiled plane hist_small_kernel now builds the tile in shared memory (fill_plane, the same stage hist_tile_kernel runs) and stores every cell of the slot, zeros included (emit_plane with a plain store), instead of adding each row visit into a slot the level memset zeroed. zero_slots_kernel clears only the slots of the nodes the tiled kernel fills (lvl.slots, one per tiled node), since the small kernel's store is the zero for the rest. On the per-feature plane (a feature over about 384 bins, where the tile does not fit shared memory) the direct global-add kernel and the full memset stay as they were. The fill is a template parameter of one kernel (SmallFill::direct, store, store_unit_h) over one argument struct, and the profile plane line reports which path ran (small nodes stored whole or added in place). Integer sums commute, so no model byte moves: model_hash.py --grower cuda_depthwise read e1a5391a7beea349 on both arms of the pod build.

Why. Decision 136 read the wide cell's small-node kernel at 0.84 s of the 2.00 s adv_hist, flat at 0.80 to 0.87 us per small row, and the level memset at 0.50 s, and left open whether the atomic unit or the DRAM sectors under a random 8-byte read-modify-write bounded the kernel. The store design removes both mechanisms at once, and the memset of the same slot with them, so that probe was not needed to choose it: on wide a slot is about 64 MiB (16384 features at stride 2 x 256 cells of 8 bytes), the memset writes it once, the small kernel then touches it again at random, and the store writes it once in tile order with the memset gone.

Measurement. Same-pod on one RTX PRO 6000 Blackwell Server Edition (US-NC-1), 04da8e4 (A, the decision 136 head) against e9d35f4 (B), interleaved reps, min over two reps, seconds; find from the grow profile, adv_memset and adv_hist from the round decomposition, small the sum of small_lN over the levels, fit total from the bench row. r2 equal to four places on every cell and rep; B printed small nodes stored whole on every run and A never did.

cell	fit A	fit B	fit delta	find A	find B	adv_memset A	adv_memset B	adv_hist A	adv_hist B	small A	small B
wide 131072 x 16384 dw	6.08	5.45	-10.3%	4.30	3.50	0.50	0.22	2.00	1.51	0.84	0.36
tall 16M x 128 dw	2.84	2.89	+1.6%	0.98	0.98	0.01	0.01	0.92	0.92	0.00	0.00
extreme 16M x 1024 dw	13.85	13.68	-1.2%	7.34	7.35	0.03	0.02	7.16	7.17	0.00	0.00
wide 131072 x 16384 lw	7.33	7.37	+0.5%	5.18	5.14

Wide per level, small rows per tree, seconds over 100 trees, and the small kernel's cost per small row where the count is measurable:

level	small rows/tree	small A	small B	hist A	hist B	us/row A	us/row B
3	75	0.01	0.00	0.13	0.12
4	268	0.02	0.01	0.18	0.17
5	1000	0.08	0.03	0.29	0.24	0.80	0.30
6	2624	0.22	0.09	0.46	0.33	0.84	0.34
7	5894	0.51	0.23	0.76	0.47	0.87	0.39

The B small_lN seconds include the slot bytes the store now writes in place of the memset, so the two buckets move together: the memset gave up 0.28 s and the small kernel 0.48 s, 0.76 s in all against the 0.80 s the grow profile's find lost and the 0.83 s the fit profile's grow lost (4.64 to 3.81). The fit total moved 0.63 s; the other 0.2 s sits in the ingest profile (mapper-fit 1.28 to 1.38, dbin 0.46 to 0.53), host-side stages the change does not touch, read here as the run's spread and not attributed. Tall and extreme have no small rows to speak of (decision 136), and the lever leaves them where they were: tall's +1.6% is 0.045 s between arms whose own reps spread 0.05 s, with adv_hist and find identical. Wide leafwise does not run the level kernels and reads flat.

Rejected alternatives. The decision 136 reopener's probe (atomic throughput against sector footprint, on a pod) before designing: both mechanisms are the same code path, the random global read-modify-write into a slot the memset had just written, and a store into an owned slot removes the path rather than tuning it, so the probe would have chosen between two costs that both go. Keeping the memset and storing only nonzero cells (the tiled kernel's if (v != 0) emit): the store's bytes are the memset's bytes, so the full store costs nothing the memset did not, and it frees the memset. A second kernel for the store path: it duplicated the small kernel's prologue and the design lint's clone count rose 108 to 110; one kernel templated on the fill brought it back to 108. Extending the owner store to the per-feature plane: that plane holds a single feature's cells in shared memory per block, so the tile it could store is one feature wide and the memset it could skip is one feature of the slot; not priced, no ledger cell runs there.

Reopener. The wide cell's find is 3.50 s of a 5.45 s fit: root_sums 0.19, root_hist 0.19, adv_memset 0.22, adv_hist 1.51, and find_kern 1.61 s (the finder, which reads the parent and small-child strips, derives the large sibling on the fly and stores it while its children are still to be found, then the reduce), which this lever does not move. By the slot arithmetic above the finder touches on the order of 30 GB of slot bytes per tree (one strip read per small child, two read and one written per large child, 254 children found per tree at 64 MiB a slot) in 16 ms per tree, which would put it at the DRAM ceiling of the part; that is arithmetic, not a measurement, and the next step is the finder's bytes against its time on wide, before a byte-cutting design is chosen. The same owner argument applies to a tiled node whose chunk count is one (decision 135 gives each node its own count): its blocks are also the only writers of its slot, so its tiles could be stored whole and its slot dropped from the memset, priced at the 0.22 s of memset that remains on wide.

138. The wide finder is DRAM-bound at the subtraction scheme's byte floor, and the sibling block pairing is its L2 hit (study)¶

Decision. The profile gains two counters per level, find_lN (seconds the level's find_splits_many launch takes, from the profile-only device sync that already timed find_kern) and find_gb_lN (the slot strip bytes that launch touches: one strip of n_selected x stride x 8 bytes for a node whose slot the fill wrote, three for a node derived from its sibling while its own children are still to be found, since it reads the parent, reads the sibling and stores itself, two at the last level where it does not store). find_kern is unchanged and the seven per-level arrays of the profile became one LevelCounters struct per level. No byte-cutting finder design is chosen: the read below puts the finder at the DRAM ceiling of the part on the bytes the subtraction scheme needs at 8 bytes a cell, and every design that cuts counted bytes either cuts none from DRAM or costs more in the fill than it saves. The finder's __launch_bounds__(32, 16) stays: the two alternatives measured below both lose.

Why. Decision 137 left find_kern at 1.61 s of the wide cell's 5.45 s fit and estimated the finder near the DRAM ceiling by slot arithmetic. A design chosen against that estimate would have been a bet on which of three mechanisms (bytes, latency, occupancy) bounds the kernel, so the counters came first, then two pod probes that are not kept in the tree.

Measurement. One RTX PRO 6000 Blackwell Server Edition (US-NC-1), the counters commit, depthwise, 255 bins, depth 8, 100 trees; wide 131072 x 16384 read fit 5.373 and 5.521 over two reps with find_kern 1.61 and 1.63 s. Per level, seconds and counted GB over the 100 trees, then the DRAM share the pairing paragraph below establishes (three of four counted strips at levels 1 to 6, two of three at level 7) and the rate that implies. Levels 1 and 2 sit at the two-decimal resolution of the lap timer and are not read.

level	find_s	counted GB	counted GB/s	DRAM GB	DRAM GB/s
1	0.02	26.7		20.0
2	0.04	53.5		40.1
3	0.06	107.0	1783	80.3	1338
4	0.12	213.4	1778	160.1	1334
5	0.23	421.1	1831	315.8	1373
6	0.44	807.5	1835	605.6	1376
7	0.70 / 0.68	1128.6	1612 / 1660	752.4	1075 / 1106
sum	1.61	2757.8	1713	1974.3	1226

The ceilings of the same pod, measured in situ by a standalone program over 2 GiB buffers, min of three reps: cudaMemset 1.53 TB/s written, cudaMemcpy device to device 1.47 TB/s read plus written, a grid-stride copy kernel 1.39 TB/s, and a grid-stride sum with one 8-byte load in flight per thread 1.25 TB/s, which is a latency-bound kernel and not a ceiling. No kernel reaches the part's 1.79 TB/s specification. The finder's counted rate at levels 3 to 6 exceeds every one of them, so counted bytes include cache hits.

Which bytes hit was settled by a probe: find_grid is {2 x n_selected, (n_nodes + 1) / 2} and find_block maps blockIdx so that the two siblings' warps for one feature are adjacent blocks; the probe kept the grid and the counted bytes and scheduled every small node's warp before every large node's, so a sibling pair no longer shares residency in L2. Same pod, same session, model_hash.py --grower cuda_depthwise equal on both arms (e1a5391a7beea349): find_kern 1.62 to 2.13 and 2.14 s (+32%), level 4 0.12 to 0.16, level 5 0.23 to 0.31, level 6 0.44 to 0.59, level 7 0.69 and 0.70 to 0.90 and 0.91, fit 5.641 and 5.753 to 6.100 and 5.957. Read as DRAM rates on the counted bytes the unpaired kernel runs at level 4 1334, level 5 1358, level 6 1369 and level 7 1254 GB/s: the same rates the paired kernel reaches on three quarters of its counted bytes. The derived node's read of its sibling's strip is therefore an L2 hit today, its cost is the extra 32%, and the pairing is a lever already taken by the grid shape. Levels 3 to 6 then run at 1.33 to 1.38 TB/s of DRAM, between 96% and 99% of the copy kernel and 90% of the memset, and level 7 at 1.08 to 1.11 TB/s, 78% of the copy kernel, not attributed.

The finder is a wide-only cost: tall 16M x 128 reads 0.02 s of finder over a 2.857 s fit (23 GB counted) and extreme 16M x 1024 reads 0.12 s of 13.881 (186 GB), because a strip is 4 KB per feature per node and only the feature count scales it.

Register and occupancy facts from cuobjdump --dump-resource-usage on the sm_120 module: both find_kernel instantiations and level_find_kernel compile to 128 registers with 24 and 32 bytes of stack; __launch_bounds__(32, 16) on the finder caps registers at 65536 / (32 x 16) = 128, so at most 16 one-warp blocks are resident per SM. The occupancy probe, same pod and session, interleaved with the base arm, wide fit seconds and find_kern per rep: base 5.633 and 5.534 with 1.63 and 1.61; __launch_bounds__(32, 24) compiles the finder to 80 registers with 168 bytes of stack and reads 6.470 and 6.237 with 2.48 and 2.34 (+45% on the finder, level 7 1.11 and 1.04 against 0.70 and 0.68); __launch_bounds__(32, 32) leaves the two instantiations uncapped at 138 and 142 registers with no stack, 14 resident warps per SM at that count (arithmetic), and reads 6.094 and 5.725 with 1.84 and 1.80 (+12%, level 7 0.82 and 0.80). Every arm prints the base hash e1a5391a7beea349 and r2 0.8597; tall reads 2.975, 2.990 and 2.985 and extreme 13.802, 13.960 and 13.825 (base, 24, 32), inside their spread. So the kernel does answer to warps in flight, two fewer cost 12%, but the spill that buys eight more costs 45%, and 128 registers is the working point of the live state this finder carries.

Rejected alternatives. Fusing the two siblings' warps into one that reads the parent and the small strip once and emits both: it cuts a quarter of the counted bytes and none of the DRAM bytes, since the read it removes is the L2 hit the probe priced, and it is not proposed. Not storing the large strip and deriving it from the grandparent when its children are found: a store and a read become two reads, the same strip count (arithmetic). Filling both children and deriving none: the finder's DRAM strips per pair go from three to two at levels 1 to 6, about 0.3 s at the measured rate, against a fill that would roughly double adv_hist from 1.51 s (arithmetic), a net loss. Narrower cells (int32 slots, half the bytes of every strip): a change to the fixed-point scale, so model bytes move at the float32 ulp on some inputs as decision 134's row-cap arms did; it is a wire change with its own decision, not a finder design.

Reopener. Level 7 runs at 78% of the copy kernel where levels 3 to 6 run at 96% and above; its 0.70 s holds about 0.2 s against the rate of the levels above it (arithmetic), and nothing here attributes the gap. The byte floor itself moves only with the cell width. A finder that keeps fewer live registers per warp without spilling would raise occupancy where the launch bound cannot; which live state the 128 registers pay for is not read here, and a SASS read of find_node is the step before that design. The remaining wide levers outside the finder are the single-chunk owner store priced at decision 137's 0.22 s of adv_memset, and adv_hist at 1.51 s.

139. The wide mapper fit moves to the device as a per-column radix sort over a raw matrix uploaded once, and the pageable ingest upload is host-memory-bound (adopted)¶

Decision. The bin mapper fit for the CUDA plane sorts on the device. When the raw feature matrix fits device memory it is uploaded once, column chunks are transposed, sorted by a hand-rolled per-column LSD radix sort, cut by a kernel that reproduces create_cuts bit for bit, and then the resident raw matrix is binned in place. When it does not fit, the host gathers the 200000-row sample as today and uploads only that. The host mapper fit stays the path of the CPU plane, the ColumnBatch input, and user-supplied edges. The change is priced at wide ingest (mapper-fit plus dbin) 1.93 s to 0.8 to 0.9 s, about -1.0 s of the 5.45 s fit, and is built after this entry with the same-pod A/B as its merge gate. Two side findings close levers on the way: the ingest upload of pageable input is bound by host memory bandwidth at about 28 GB/s pipelined, so neither cudaHostRegister nor a deeper pinned ring moves it, and the tall and extreme mapper fits are mostly the serial seeded sample draw, which no device sort touches.

Why. Decisions 135 to 138 took the wide fit to 5.45 s with the finder at its byte floor and the fill chunked per node. The ingest profile of the wide cell, read in the same class of pod session as decision 138 (not in that entry), shows mapper-fit 1.39 s and dbin 0.54 s, 35% of the fit, the largest buckets outside the round. Pricing a device fit needed three measurements no prior entry carried: what the host fit spends its time on, how fast the device sorts the same keys, and what an upload costs against the host's own copy rate.

Measurement. Host split (M2 laptop, 8 threads, a temporary probe hunk around BinMappers::fit that is not kept in the tree, two fits per cell, structure only, no laptop number is quoted as a pod number):

cell	sample draw s (serial)	gather thread-s	sort and cuts thread-s
wide 65536 x 8192	0.000	0.83 / 1.04	4.66 / 4.74
tall 4M x 128	0.064 / 0.066	0.11 / 0.27	0.25 / 0.26

On wide every row is used, there is no draw, and the sort with its cuts is 82 to 85% of the fit's thread time. On tall the draw is std::ranges::sample over iota(n) with one mt19937 per fit, O(n) and serial; at 16M rows it is about 0.26 s by arithmetic, most of the pod's 0.32 s tall mapper-fit and about half of extreme's 0.48 s.

Device sort ceiling: one RTX PRO 6000 Blackwell Server Edition (US-NC-1), a standalone program, cub::DeviceSegmentedRadixSort::SortKeys over float keys with every fourth column integer-valued, used as the ceiling only since decision 40 keeps cub out of the library, min of three reps, a descent count of 0 on every arm:

keys	count	sort ms	Gkeys/s	tmp
wide 16384 cols x 131072 rows, 512-column chunks	2.15 G	107.4	20.0	268 MB
same, 2048-column chunks	2.15 G	108.6	19.8	1.07 GB
same, 8192-column chunks	2.15 G	109.4	19.6	4.3 GB
tall sample 128 x 200000	25.6 M	1.6	16.2
extreme sample 1024 x 200000	205 M	10.4	19.7

cub's num_items is an int, so the whole wide matrix (2^31 keys) is sorted in column chunks; the chunk width does not move the rate.

Upload rates on the same pod for the 8.59 GB wide raw matrix: pinned cudaMemcpyAsync 152 ms, 56.4 GB/s; pageable cudaMemcpy 335 ms, 25.5 GB/s; host std::copy_n pageable to pinned 33.1 / 47.5 / 48.4 / 45.5 GB/s at 1 / 4 / 8 / 16 threads; cudaHostRegister of the pageable buffer 0.26 to 0.32 s (27 to 33 GB/s of registration), then DMA at 45.4 to 46.7 GB/s and unregister 0.11 to 0.14 s; a DMA from one pinned buffer concurrent with an 8-thread host copy into another ran at 28.4 to 30.2 GB/s and 28.0 to 33.4 GB/s, 0.30 s of wall for both. The two streams share one host memory budget, so the pipelined rate for pageable input is about 28 GB/s, and decision 133's ring (extreme dbin 2.78 s, 24.7 GB/s) is within 12% of it.

Price, arithmetic from the rows above. Wide today 1.39 + 0.54 = 1.93 s; the device path is one pageable upload 0.34 s, chunk transposes about 0.01 s, a hand-rolled sort at two to three times cub's time 0.22 to 0.33 s, a cuts kernel about 0.02 s, and binning from the resident matrix about 0.2 s (the dbin bucket less its copy), 0.8 to 0.9 s in all, so -1.0 to -1.1 s. Tall: the 1.6 ms sort replaces about 0.03 s of wall and the draw stays. Extreme: the 10 ms sort replaces about 0.2 s of host sort; the 0.8 GB sample gather and its upload stay. Device transient at wide: the 8.6 GB resident matrix plus two 268 MB chunk buffers, freed before the 2.1 GB BinStore is built.

Constraints the build holds. Edges bit-identical to the host's create_cuts, pinned by scripts/model_hash.py and a per-column edge comparison over adversarial columns: ties, -0.0 beside +0.0, fewer distinct values than the budget, the greedy path at max(counts) >= mean_bin, the stride path, NaN dropped at gather. The value of a tie run is its last element in sorted order (run_lengths writes run_val[m] = f on every element, so a -0.0/+0.0 run reads +0.0) and the stride path reads sorted values by index, so the device sort orders keys exactly as sortable_key does. The greedy path is serial over runs and runs one thread per column. The one mt19937(seed) draw over the whole matrix's rows stays on the host, so a device fit changes no sampled row.

Rejected alternatives. cudaHostRegister of the caller's array in place: registration runs at 27 to 33 GB/s, slower than the copy it would replace. Restreaming the raw matrix through the ring once for the fit and again for the bin: +2.8 s at extreme against a 0.2 s host sort. cub in the library: decision 40. Moving the sample draw to the device or to a parallel sampler: it changes which rows are sampled, so model bytes move on every CUDA cell; at about 0.26 s of tall's 2.86 s (9%) and 2% of extreme it is a separate model-changing entry. A device fit for the tall cell alone: 0.03 s.

Reopener. A user-pinned input path (a cudaHostAlloc array reaching fit unchanged) lifts the ingest bound from 28 to 56 GB/s; extreme's 2.78 s dbin would read about 1.3 s (arithmetic). The serial sample draw is the next tall and extreme mapper-fit lever, and a model-changing one. scripts/dag_model.py carries h2d 14e9 and h2d_pinned 24e9 from an older host and a mapper_fit note that predates the whole-matrix draw; both are refreshed with the build's constants.

140. The device mapper fit ships: the wide CUDA fit is 13.5% faster same-pod, the raw upload gets its own ingest bucket, and small host columns sort under the radix key order (adopted)¶

Status 2026-09-10: the reopener's 0.66 s was not the sort. Decision 141's decomposition splits it into sort 0.21 s and cuts 0.40 s, and the cuts kernel's serial walk was the lever; the sort's 0.21 s against the 0.11 s ceiling stays open there.

Decision. Decision 139's design is built and measured. The CUDA plane uploads a host feature matrix once (cuda_upload, an owning handle), sorts and cuts every column on the device (cuda_fit_mappers: a gather transpose into 32-bit sortable keys, one block per column running four LSD byte passes with a pass skipped when every key shares the byte, one warp per column walking the sorted runs and reproducing create_cuts), and bins from the same resident copy. The upload declines and the host path runs when there is no device, when max_bin exceeds the shared-memory ceiling the fill would refuse, or when the matrix would take more than half of free device memory counting the mempool's reserved bytes as free; extreme (16M x 1024, 68.7 GB against 96 GB) takes that branch and is unchanged. The DLPack path fits on the device too, where it gathered the sample to the host before; cuda_gather_rows is gone with its only caller. Three behaviour changes travel with it. mapper-fit in the ingest profile now measures the device fit on this plane, and a new upload= bucket carries the raw matrix copy, which nothing timed before and dbin no longer pays. The host's small-column sort (under 2048 rows, where sort_floats used an unkeyed std::sort) now sorts by the same key as the radix path, so -0.0 orders before +0.0 on both host paths and on the device; a serialized CPU-plane model whose column has under 2048 rows and both zero signs can differ from 2.2.0 by one sign bit in one cut, binning is unaffected because lower_bound treats the zeros equal, and scripts/model_hash.py holds at 55c6fe308852d9bb. The design vocabulary admits upload as the seam's name for a host-to-device copy that neither bins (ingest) nor reads a file (load), the word DeviceBuffer::upload already carries below the seam; gather survives the deletion as RowView::gather.

Measurement. One RTX PRO 6000 Blackwell Server Edition (US-NC-1), arms built as separate checkouts, cells interleaved, min over reps, BONSAI_INGEST_PROFILE=1 with the engagement line bonsai: mapper fit on the device asserted on every fit-arm run and absent on every base run:

cell	reps	base fit s	fit s	delta	base mapper-fit + dbin	device mapper-fit + dbin + upload	r2
wide 131072 x 16384	4 + 4	5.487	4.747	-0.74 s, -13.5%	1.41 + 0.53 = 1.94	0.66 + 0.15 + 0.34 = 1.15	0.8597 both
tall 16777216 x 128	3 + 3	2.926	2.910	-0.5%, inside the spread	0.31 + 0.39 = 0.70	0.30 + 0.04 + 0.37 = 0.71	0.8797 both
extreme 16777216 x 1024	3 + 3	13.794	13.780	-0.1%, inside the spread	0.47 + 3.53	0.44 + 3.55 (host path, upload 0.01 is the warm-up fit)	0.8788 both

Grow is flat on every cell (wide 3.81 to 3.83 s, tall 2.15, extreme 9.69 on both arms). The device model hash (model_hash.py --grower cuda_depthwise) is e1a5391a7beea349 on both arms and both repeats, so the device cuts are the host's bit for bit at 500000 x 100; the [cuda] suites pass on sm_120 (72 passed, 1 skipped) and on the Jetson Orin, whose device hash also matches its base. The wide reps on the base arm read 5.487 to 5.703 and on the fit arm 4.747 to 4.794, so the delta is four times the spread.

Against decision 139's price: the upload is the priced 0.335 s, the bin from the resident matrix 0.15 s against about 0.2, and the sort with its cuts 0.66 s against the priced 0.22 to 0.33, which put a hand-rolled sort at two to three times cub's 0.107 s ceiling. It is six times. The wide ingest lands at 1.15 s against the priced 0.8 to 0.9, and the fit at -0.74 s against the priced -1.0 to -1.1. At tall the copy moved buckets and no time moved: dbin 0.39 to 0.04 plus upload 0.37, and mapper-fit 0.31 to 0.30 because the seeded draw stays serial on the host.

Rejected alternatives. Loosening the device parity test to bin equivalence when the device's +0.0 disagreed with the host's -0.0 on a 256-row column: the host's two paths already disagreed with each other there (the radix path over 2048 rows orders -0.0 first, std::sort orders the zeros however it likes), so the fix is the host's key, and the test compares cut bytes. Folding the upload into dbin: it would hide inside a bucket named for the bin the one copy the device fit no longer pays there. Fitting extreme through chunked uploads: decision 139 priced restreaming the raw matrix at +2.8 s against a 0.2 s host sort. Bounding the upload by total device memory instead of half of free: the fill's own buffers follow.

Reopener. The per-column sort at 0.66 s against a 0.11 s ceiling is the next wide ingest lever, 0.5 s of a 4.75 s fit: one block per column serializes four passes over 131072 keys, and a column-parallel pass or a wider digit is the shape to price. A pinned input path (decision 139's reopener) halves the 0.34 s upload. The serial sample draw stays the tall and extreme mapper-fit lever and is model-changing. The dag_model.py constants h2d 25.5e9 and h2d_pinned 56.4e9 and the mapper_fit node now carry this session's numbers.

141. The cuts kernel finds runs a warp at a time: the wide CUDA fit is 7.3% faster same-pod and the device mapper fit is 0.27 s (adopted)¶

Decision. The device mapper fit's four stages are timed under BONSAI_CUDA_PROFILE (cuda-mapper-fit: gather= sort= cuts= d2h=), and the cuts kernel finds a column's runs of equal values in parallel. Before, one warp per column walked the sorted keys one element at a time through a 32-key cache: 131072 serial compare-and-branch steps per column on the wide cell while 31 lanes idled. Now every lane loads one key of a 32-key chunk, a shuffle brings its predecessor, and one ballot of prev < value gives the chunk's run-start mask; summarize_runs reads run lengths off the mask with popc and clz and reduces per-lane accumulators across the warp, push_every_run writes each run's closing value at n + rank in one step, and the greedy path's RunCursor walks the same masks. The compare is the float ordering the serial walk used, so -0.0 and +0.0 still share a run and the emitted cuts are the same bytes.

Measurement. The decomposition first, three reps at the instrumented base on one RTX PRO 6000 Blackwell Server Edition (US-NC-1), wide 131072 x 16384: gather 0.015 s, sort 0.206 s, cuts 0.404 to 0.406 s, d2h 0.001 s, against mapper-fit=0.66 in the ingest profile; the parts conserve (0.63 of 0.66, the rest is the host's from_cuts). The cuts kernel was 61% of the device fit and the sort 31%, so the lever moved from decision 140's sort to the walk. Then the lever, same pod, arms rebuilt in place and interleaved base, lever, base, lever, min over reps:

arm	reps	fit s	ingest s	cuts s	train s	r2
base	5	4.766	1.095	0.403 to 0.407	3.667 to 3.692	0.8597
lever	7	4.420	0.749	0.036	3.666 to 3.688	0.8597

Fit -0.346 s (-7.3%), ingest -31.6%, the cuts kernel -91%, train flat; the base reps read 4.766 to 4.828 and the lever reps 4.420 to 4.455, so the delta is five times the spread. Priced at -0.38 s from the decomposition, measured -0.35 s. The device model hash (model_hash.py --grower cuda_depthwise) is e1a5391a7beea349 on both arms, the [cuda] suites pass on sm_120 (72 passed, 1 skipped, 383401 assertions), and the CPU hash holds at 55c6fe308852d9bb. The device mapper fit now reads 0.27 to 0.28 s: sort 0.21, gather 0.02, cuts 0.04.

Rejected alternatives. Attacking the sort first, as decision 140's reopener said: the decomposition put it at 0.21 s of 0.66, and its whole distance to the cub ceiling is 0.1 s, 2% of the fit. A block-wide run scan (256 threads, shared-memory prefix) in place of the warp ballot: the walk was latency-bound on one lane, and a warp already reads a 32-key chunk in one coalesced load; the ballot form kept the kernel's one-warp-per-column shape, its eight-column blocks, and the stride path's SortedColumn::at untouched. Emitting cuts through a per-run atomic counter instead of n + rank: order would then depend on scheduling, and the cuts must be the host's bytes.

Reopener. The wide ingest is 0.75 s of a 4.42 s fit: upload 0.36 s (pageable at 24 GB/s, the pinned input path of decision 139's reopener halves it), the sort 0.21 s against cub's 0.11 s, dbin 0.18 s. Each is under 5% of the fit, so the next wide lever is priced against the train's 3.67 s before a pod is spent on ingest again. Tall and extreme are unchanged by construction: their mapper fit is the host's seeded sample draw.

142. The 2.3.0 refresh ships the tall CUDA cells 35 to 66% under the anchor and the cpu leafwise cell 5% under it; the cpu A/B leaves the GPU pod for a cpuset host (adopted)¶

Decision. The 2.3.0 standings supersede 2.2.0's on every axis, all eleven measured at the merged bump sha on one RTX PRO 6000 Blackwell Server Edition pod, with every moved cell of the release A/B shipping as measured. On the GPU plane the four gpu-tall cells sit 35.1 to 66.5% under the 1.15.0 anchor and the 1M x 100 cuda_levelwise cell 5.4% under 2.2.0, with peak host RSS 7.9% under 2.2.0 at 1M; the 16M cells read within 0.3% of 2.2.0. The anchor moves are the cumulative device work since 1.15.0 (decisions 126 and 127 carried most of it), and the move against 2.2.0 at the small cell is this release's device levers (decisions 135 to 141) reaching a fit short enough for its fixed costs to show. On the CPU plane the cpu-tall leafwise cell reads 4.5% under 2.2.0 and 5.1% under the anchor: at depth 8 the A/B's leafwise arm carries num_leaves of 256, a budget that cannot bind, and decision 132 now grows that configuration on the depthwise plane, so the move is the route landing on the host. The depthwise and levelwise cells read within 0.7% of both references.

The cpu A/B changes host. The 2.2.0 entry (decision 127) shipped a cpu-tall depthwise reading of +3.7% over the anchor as one repeat's spread on a bandwidth-metered GPU pod and set as its reopener a CPU-pod session. This refresh ran that session, and first ran the A/B once more on the GPU pod: 4 reps there read depthwise +14.4% over the anchor and +7.3% over 2.2.0, leafwise -5.3% and -2.1%, levelwise -0.9% and +1.1%, three growers moving in three directions by more than the band. Eighteen further interleaved reps of the depthwise cell on the same pod (six per arm) put the new build at -2.0% against the anchor and -2.9% against 2.2.0, and the per-rep spread on that host at the cell was 15 to 22%; no min over reps recovers a 2% band from a spread of that size. The same three arms on a cpu5g pod (16 vCPU, a cpuset with no bandwidth quota, OMP_WAIT_POLICY=passive) at 6 interleaved reps spread 0.7%, and that session is the one this entry ships. The host of record for the cpu axes stays the GPU pod's server CPU: the same cpu5g pod measured the cross-library ranking with LightGBM 3.0x slower than bonsai at the tall cell, where the GPU pod's server silicon had them near parity, so a desktop-class rental would rank a different machine class. A ranking wants the server silicon and an A/B wants a quiet host, so standings_refresh.py measure now rents both: the GPU pod for every axis and the GPU A/B, and a cpu5g pod for the cpu A/B alone, at 6 reps.

Measured (ab-gpu-2026-09.jsonl on the RTX PRO 6000 pod, 2 interleaved reps; ab-cpu-2026-09.jsonl on the cpu5g pod, 6 interleaved reps; min over reps; anchor 1.15.0 wheel, old 2.2.0 wheel, new 2.3.0 source at 0ea5b77). gpu-tall cuda_depthwise 1M x 100 anchor 0.46 s, old 0.25 s, new 0.24 s; cuda_levelwise 0.89, 0.32, 0.30 s; 16M x 100 cuda_depthwise 3.82, 2.47, 2.48 s; cuda_levelwise 3.87, 2.40, 2.39 s; peak host RSS 0.63 to 0.58 GB at 1M, 6.78 to 6.74 GB at 16M. cpu-tall 2M x 128 at 12 threads depthwise 13.27, 13.19, 13.18 s; leafwise 13.63, 13.54, 12.93 s; levelwise 11.64, 11.71, 11.72 s. Parity PASS at the anchor cell (fused 2.49 s against two-step 2.47 s, -0.5%). The A/B rows carry no host block, so the cpu A/B's host is recorded here and in the runbook's section 11 until the rows carry it.

Rejected. Shipping the GPU-pod cpu A/B under decision 127's reading rule: that rule covered one cell over the band by one repeat's spread, and this session's four-rep reading had three growers disagreeing with each other by more than the band, which is a host that cannot read the band rather than a cell to explain. Moving the cpu axes to the CPU pod along with the A/B: the ranking flip is measured on the same rental, and a standings row is a claim about a machine class (section 11 of the runbook); the option stays behind --cpu-plane-host cpupod. Widening the CPU band on GPU-pod hosts: the band would then hide a real 3% on the pod that can see it, decision 127's argument, unchanged. Reading the mean: unchanged from decision 127, noise on a fixed workload only adds time.

Reopener. A cpu5g session whose three arms spread more than 2% at the tall cell, at which point the cpuset claim is re-measured before the band is; a cpu-tall depthwise or levelwise reading more than 2% from the anchor on that host, which is the engine's and gets its own entry; or the A/B rows gaining a host block, at which point this entry's host attribution is redundant with the file.

Standings: gpu-tall, cpu-tall

Decisions¶

1. Binning: quantile, with low-cardinality fallback¶

2. BinMapper independent of Dataset. Two-stage API.¶

3. Trees store raw float thresholds, not bin indices¶

4. Dataset storage layout¶

5. (reserved)¶

6. Readers: free function per format, returning Dataset¶

7. Determinism contract: fixed thread count, not cross-thread¶

8. Two trees in Phase 1: depth-wise + oblivious¶

9. Tree is a concept; minimum surface = predict ×2 + diagnostics¶

10. Shrinkage is baked into leaf values at tree construction¶

11. ObliviousTree: per-level default_left, not per-node¶

12. Sampler is the booster's responsibility, not the grower's¶

13. Branchless NaN routing in predict¶

14. Splitter is a template parameter on the grower; one SplitFinder concept¶

15. Splitter returns one best candidate per call¶

16. Splitter tie-break: lowest fid, then lowest bin_idx¶

17. Partitioning: per-node row-index lists (strategy A)¶

18. Frontier holds histograms inline (no histogram pool)¶

19. Subtraction trick from day one, both growers¶

20. Phase 1 regularization knobs¶

21. Objective is a concept; static methods, no instance state¶

22. Objective does not own initial score or link inverse¶

23. Phase 1 objectives: MSE + binary logloss, single-output¶

24. compute writes raw-score grad/hess; output is overwritten, not accumulated¶

25. Sample weights are applied by the booster, not the objective¶

26. Dispatch: flat table over cartesian product, IBooster at boundary¶

27. Booster shape, training loop, and Sampler concept¶

28. Spine complete; insert Phase 2.5¶

29. Model file serializes the full Config via NLOHMANN macros, not the existing TOML codec¶

30. ObliviousGrower: fold-then-score was wrong, revert and re-spec¶

What landed (2026-05-21)¶

What went wrong¶

Verification against CatBoost (2026-05-22)¶

Resolution (2026-05-22)¶

What was kept¶

Outstanding¶

Lesson¶

31. LeafwiseGrower: best-first growth on a gain-keyed heap, max_leaves primary¶

32. Parallelism: OpenMP behind a one-function seam, determinism at any thread count¶

33. Hot-path perf: ordered gradients, stable scatter, node totals out of add¶

34. Feature-parity round: colsample, GOSS, early stopping, L1, and the OOB score bug¶

35. Remaining-gap round: objectives, monotone, interaction, DART¶

36. Python bindings: nanobind over the CLI's own seams, static libomp¶

37. Feature importance recorded at grow time; the guide series¶

38. Completing the non-categorical gap: rows 10-17 in one push¶

39. Categorical stage 1: measure before building¶

40. GPU-resident growing: widen the builder seam to an optional level backend¶

41. Grower data-plane as the LevelStep strategy; retire the copy-back ladder¶

42. GPU oblivious via a device level-find; no cuda_leafwise¶

43. CI: GitHub Actions gates every PR; sanitizers as a build option, not a preset¶

44. n_threads = 0 means capped auto, not all hardware threads¶

45. Ingest profiled, then one optimization kept: sort-based quantile cuts¶

46. Scaling suite: in-memory sweeps, synthetic Friedman target, frontier-as-data¶

47. Ingest and histogram-allocation rounds: measure on the platform that has the disease¶

48. The "5090 anomaly" was a defective host; benchmark pods now pass a sync-latency probe¶

49. Row-wise histogram fill over a row-major u8 mirror; determinism relaxes to fixed thread count¶

50. Float histogram cells; double reductions; the empty-split demotion guard¶

51. quantile_step ceiling stride: cuts never exceed the budget¶

52. Device residency: REFUTED by experiment; the lever is per-level staging latency¶

53. The level-transaction engine narrative (adopted)¶

54. Device binning at ingest (adopted)¶

55. The cut-quality gap is not sample size (study)¶

56. Quality-campaign fixes: the oblivious veto, exact duplicate cuts, the true softmax hessian (adopted)¶

57. Count-weighted cuts for heavy-value columns only (adopted)¶

58. Categoricals resolved by measurement: an encoder, not an engine feature (adopted)¶

59. Cross-architecture bit determinism: no fp contraction on the host plane (adopted)¶

60. Issue #72 resolved: the "cross-arch divergence" was OpenMP build variance (adopted)¶

61. The populate round: software prefetch closes the 16M CPU gap (adopted)¶

62. The GPU find-kernel round, abandoned by instrumentation; the 16M GPU frontier belongs to catboost (adopted)¶

63. GPU oblivious lost accuracy at scale to a missing issue-#60 port, not to catboost's algorithm (adopted)¶

64. One shared row sample for binning, not one reservoir pass per feature (adopted)¶

65. Reusable pre-binned Dataset with a sealed bin config (adopted)¶

66. Prebuilt wheels from native runners, not manylinux containers (adopted)¶

67. Automatic per-feature bin budgets: declined by measurement; explicit user edges: open design question (partial)¶

68. The Grinsztajn benchmark becomes the standings suite; bonsai has the best mean rank (adopted)¶

69. The benchmark charter: bonsai.bench in the wheel, two divisions, one row schema (adopted)¶

70. CUDA wheels: fold-in, static cudart, and a rented-GPU release gate (adopted)¶

71. The post-fix 16M GPU frontier: split by budget, ceiling recovered (adopted)¶

72. The marginal-round campaign: 155 to 104 ms, the frontier taken whole (adopted)¶

2. `BinMapper` independent of `Dataset`. Two-stage API.¶

4. `Dataset` storage layout¶

6. Readers: free function per format, returning `Dataset`¶

9. `Tree` is a concept; minimum surface = predict ×2 + diagnostics¶

11. `ObliviousTree`: per-level `default_left`, not per-node¶

14. Splitter is a template parameter on the grower; one `SplitFinder` concept¶

16. Splitter tie-break: lowest `fid`, then lowest `bin_idx`¶

21. `Objective` is a concept; static methods, no instance state¶

22. `Objective` does not own initial score or link inverse¶

24. `compute` writes raw-score grad/hess; output is overwritten, not accumulated¶

26. Dispatch: flat table over cartesian product, `IBooster` at boundary¶

29. Model file serializes the full `Config` via NLOHMANN macros, not the existing TOML codec¶

30. `ObliviousGrower`: fold-then-score was wrong, revert and re-spec¶

31. `LeafwiseGrower`: best-first growth on a gain-keyed heap, `max_leaves` primary¶

33. Hot-path perf: ordered gradients, stable scatter, node totals out of `add`¶

41. Grower data-plane as the `LevelStep` strategy; retire the copy-back ladder¶

42. GPU oblivious via a device level-find; no `cuda_leafwise`¶

44. `n_threads = 0` means capped auto, not all hardware threads¶

97. `cuda_leafwise` admitted by measurement (adopted)¶

99. `Dataset` takes a device hint, so the two-step form keeps device binning (adopted)¶

118. A config file states a key as strongly as `--set` does (adopted)¶