Skip to content

OptunaSharp 1.1.0 - Performance & Parity Release - #6

Merged
Nucs merged 145 commits into
mainfrom
release/1.1.0
Nov 14, 2026
Merged

Nucs merged 145 commits into
mainfrom
release/1.1.0

Conversation

@Nucs

@Nucs Nucs commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

OptunaSharp 1.1.0: the 145 commits on top of main. Release notes: docs/releases/RELEASE_1.1.0.md. Each change carries the numbers its development session measured.

  • Performance - three passes (plans 13-15), none of which changed a seeded result; against the published 1.0.0, on NumSharp 0.81.0: random x5.3->x8.1, CMA-ES x5.3->x10.3, QMC x4.1->x6.0, NSGA-II x2.3->x3.2, TPE x1.5->x3.4, GP x1.5->x1.6
    • CMA-ES restores its optimizer once per generation: 100,000 trials 988 -> 27.5 s and 519 -> 17.4 GB allocated; 10,000 trials 23 -> 2.6 s
    • In-memory storage writes a running trial in place and shares identical distributions: NSGA-II 100,000 trials 73 -> 22 s, 63 -> 12 GB allocated
    • TPE: fused scoring kernel, exact kernel widths, observation and split caches, lighter Parzen arrays: 1,000 trials 9.4 -> 4.1 s (plan 13) and 3.9 -> 2.5 s (plan 14); two objectives 47 -> 18 s; 10 int parameters 2.9 -> 1.4 s (plan 15)
    • NumSharp draws disposed instead of left to the finalizer: random search 10,000 trials 3.5 -> 0.5 s; NSGA-II 100,000 trials 17.2 -> 8.6 s (Rustuna 0.2.1: 19.4 s)
    • QMC: sample-id key computed once (10,000 trials 1.0 -> 0.44 s); scrambled Sobol and Halton engines kept across trials (Sobol, 4,000 trials: 2.53 -> 0.14 s, 962 -> 58 MB)
    • GP: solves over all right-hand sides, fused Cholesky solves, reused fit scratch (TuRBO's fit too): 150 trials 15.6 -> 13.2 s, 5.5 -> 1.4 GB allocated, full GCs 795 -> 143
    • Other samplers: BruteForce (4,320 points) 12.3 -> 8.7 s and 14.0 -> 5.4 GB; SMS-EMOA and MOEA/D 25-35 % faster, 43 % less allocated; SNES, xNES and OpenAI-ES 23-34 % faster, 57-69 % less; DE, PSO, LM-CMA and VkD-CMA up to 28 % faster, 28-37 % less
    • Storages: SQLite refreshes through UNION ALL (8,000 trials with one left running: 13.0 -> 9.6 s); SQLite and EF Core parse each distribution once per load (a 10,000-trial study: 2.1 -> 1.4-1.5 s, 1.22 GB -> 479 MB); journal files keep sparse line offsets (4.3 million lines: 160 -> 0.1 MB)
    • CLI starts faster (TieredPGO off): an ask + tell pair 1,794 -> 1,578 ms
  • Parity QA run 2 against Optuna 5.0.0 - an 8-hour run found 28 differences (5 medium, 16 low, 7 nit), pinned in 37 failing tests; 23 fixed, 3 kept as documented divergences, 2 deferred
    • NSGA-III's NaN niching and GridSampler's unvisited grids follow CPython 3.12's own list.sort and set order (624 cases recorded from Python)
    • Batched NSGA-II / NSGA-III asks on one thread follow Optuna
    • CMA-ES fails where Optuna and cmaes fail (3 findings)
    • OptunaHub bridges keep a persistent shadow study: 12 samplers and 6 pruners match native runs trial for trial; successive halving and Hyperband prune 36 and 31 of 40 trials, as natively (0 before)
    • SQLite falls back past its variable limit; attributes in key order (SQLite, MySQL); studies in creation order
    • Optuna's error types and messages (7 findings): KeyError, StorageInternalError, LightGBM's repr, and more
    • CLI: PyYAML's YAML (763 recorded cases), tell without --values, structured --sampler-kwargs, empty --log-file
  • NumPy's arithmetic through NumSharp
    • Hypervolume, HSSP and UNDX bit-exact (BLAS dot products): 37 of 240 volumes and 3 of 240 HSSP picks had differed, now none; a 24-parameter UNDX study now matches
    • NSGA-III matches from eight objectives on: 8- and 10-objective runs had left Optuna's trials at trials 80 and 100
    • Reduction orders: cross-validation variance, multi-objective GP standardization (17 of 45 statistics had matched; now 54 of 54), percentile pruner, EDF grid
    • GP grids and rounding (about 1 value in 800 had rounded to the neighbouring grid point), NaN-aware argmax / max, isclose, clip
    • TPE: NumSharp's expm1 (truncated-normal quantiles bit-identical to Optuna's: 44.2 % -> 58.6 %) and the pairwise continuous sum (QA-84; 27-66 % slower than the left-to-right sum, TPE still faster than 1.0.0)
  • NumSharp 0.81.0 - the release this PR waited for
    • Its OpenBLAS is NumPy 2.5.4's, so the bit-for-bit target is Optuna on NumPy 2.5.4: the 57 BLAS-backed live cases pass (14 of them fail against NumPy 2.4.2); 5 of 36 seeded probe studies moved (CMA-ES over mixed parameters, IPOP CMA-ES, NSGA-II with UNDX, xNES, JANUS), the 8 GP studies did not
    • QA-85 fixed upstream (the median of tied signed zeros is NumPy's -0.0); run 1's QA-17 and QA-34, already fixed with QA-80 and QA-83, lose their open marks
    • Python's OverflowError, ZeroDivisionError, FileExistsError and FileNotFoundError at 14 sites where the port threw .NET stand-ins
    • NumSharp's public ConcurrentOrderedDictionary replaces the vendored concurrent collections (3,550 lines deleted)
    • Float np.prod is sequential now: the hand-written hypervolume and fANOVA products stay (no array per call) and are pinned to it over 800 random cases
    • NumPy 2.5.4 pinned in CI and the package-parity container; pymoo 0.6.1.3 stays the AGE-MOEA-2 oracle through a test-only np.row_stack alias (NumPy 2.5 removed it)
    • Still open in NumSharp 0.81.0: the np.sum(axis=-1) short-row and out= costs, so QA-84's 27-66 % stays; NSGA-III keeps its term-by-term einsum port and the GP its own triangular solves
  • New opt-in APIs
    • Compact finished trials in InMemoryStorage: 100,000 CMA-ES trials retain 257 MB instead of 1,287 MB, 100,000 NSGA-II trials 88 MB instead of 671 MB; same trials, same run time
    • Batch attribute writes (SetTrialUserAttrs and siblings)
    • CmaEsSampler(reuseToldOptimizer: true): 100,000 trials 27.5 -> 18.2 s; seeded results leave Optuna's, by design
  • Behavior changes
    • Optuna's exception types in the storages, CMA-ES and multi-objective TPE; Python's exception classes from NumSharp 0.81.0 where the port threw .NET's (OverflowException, DivideByZeroException and FileNotFoundException are not their bases)
    • Seeded results move toward Optuna's: TPE, CMA-ES with margin, NSGA-III (infinite values, 8+ objectives), UNDX crossover, GridSampler (shifted numbering), multi-objective GP, stepped GP grids, one-thread NSGA batches; AGE-MOEA-2 at 8+ objectives toward pymoo's; a hypervolume by an ulp. Random, QMC, NSGA-II with its default crossover and single-objective GP keep 1.0.0's trials
    • BLAS-backed seeded results (CMA-ES's eigendecomposition, xNES, UNDX, JANUS) follow NumPy 2.5.4's OpenBLAS: they can differ from 1.0.0 in the last bits, and match Optuna on NumPy 2.5.4
  • Fixes, other
    • Windows journal lock waits out a delete-pending lock file, up to 1 s (it had failed a journal test in 2 of about 8 runs on a loaded machine)
    • Two regressions found and fixed within the cycle: a reused TPE sampler read a deleted study's values when its id came back; a step had slowed the multi-objective split 10-20 %
  • Docs and examples
    • Performance and Compatibility guide; NativeAOT note (Getting Started, FAQ, package README)
    • OptunaHub page; MOEA/D, DE, PSO and TuRBO method pages; six real-world examples
    • Parity QA journal and its repro tests; the records of plans 13-15
  • Release preparation
    • CmaEsSampler's 1.0.0 constructor restored: binary-compatible with 1.0.0 (ApiCompat)
    • Docs cite the rebased hashes
    • Release matrix rehearsed on this branch (3 OSes, both frameworks, the Slow category): green but macOS, where four bit-exact gates written since 1.0.0 had never run; they now go Inconclusive off x64 like their siblings (NumPy's macOS wheel uses Accelerate, not OpenBLAS: 54 of 240 hypervolumes a last bit off)
    • Windows test-host crash traced and fixed: NumSharp 0.81.0's OpenBLAS 0.3.34 faults (access violation) when CI forces its Haswell kernels on AMD Zen 4/5 runners, in JANUS-DE's quadratic chart; OpenBLAS's own kernel choice and 0.3.31 both pass there (3 diagnostic rounds on 16 fresh VMs). CI now lets OpenBLAS choose on those CPUs on Windows (Linux on Zen 4 runs the pinned kernels fine), the 6 Haswell-captured oracles report Inconclusive there, and Windows test hosts leave the blame sequence on a crash
    • REMBO's 16-worker no-repeat check, which failed 2 of about 45 CI runs, is [OpenBugs] beside a one-worker reproduction: REMBO's optimizer evaluates a point twice one trial at a time too (82 of 120 seeds of that objective), through the projection's clipping and its GP's incumbent; a trace of 13,200 concurrent proposals found no lost read
    • The release job publishes docs/releases/RELEASE_<version>.md as the GitHub Release; the 1.1.0 notes in NumSharp's format, benchmarked again on NumSharp 0.81.0

Verified locally: 20 packages packed, signed and verified as publishable; ApiCompat finds no break against 1.0.0; doc gate clean; examples 59/59; unit tests 2,309 on net10.0 and net8.0; live-Optuna interop 751 passed, 0 failed against NumPy 2.5.4 (6 skipped: 5 need a Redis server, which CI starts; 1 needs optunahub < 0.5.0); DocFX 0 errors.

Merge: fast-forward main to this branch's head, so the cited hashes stay valid; not GitHub's "Rebase and merge" or "Squash and merge".

Nucs and others added 30 commits November 12, 2026 19:03
…a logs nan

Found by diffing the INFO/WARNING log lines of a real-world-shaped run (completed,
pruned, failed-by-exception, NaN and multi-objective trials) against Optuna 5.0.0.
Every line matched except two: the exception type name (InvalidOperationException
vs RuntimeError - language, expected) and the NaN failure line.

optuna/study/_optimize.py::_log_failed_trial warns
f"Trial {trial_number} failed with value {repr(value_or_values)}." with the
objective's own return value - a float for a single-objective study, so
"Trial 0 failed with value nan.". The single-objective Study.Optimize overload
wraps the scalar in a one-element array before the loop, and
Internal/Optimize.cs::LogFailedTrial prints it with ListRepr: "[nan]". Same
scalar-wrapping root cause as QA-12 (the skipped-tell message), different message.

New [OpenBugs] test Optimize_NaNScalarObjective_LogsScalarValueLikePython in
test/OptunaSharp.Tests/QA/RealWorldExampleBugTests.cs (fails today).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…_exists) + stop() at a provable bound

A tunable bin-packing heuristic (size^p ordering with a seeded tie-break jitter,
first/best/worst fit, repair passes that try to empty the least-full trailer)
loads Falkenauer-style "triplet" shipments into 13.6 m trailers. The load
planners' rules of thumb (first-fit and best-fit decreasing) are enqueued twice
each with skip_if_exists=True (the duplicates are skipped), and a callback calls
study.Stop() once a plan reaches the lower bound ceil(sum/capacity). Here FFD and
BFD need 46 trailers, the tuned heuristic 45, the bound is 40 - so the budget is
used up and the output says so.

Checked while writing it (not bugs, faithful to Optuna): when stop() fires right
after the first trial, the second enqueued trial stays WAITING with no params and
study.trials includes it, as Python's does; the example summarizes COMPLETE
trials only and explains why.

Also checked in this round (no defect): sampler-fallback and enqueue warnings
match Optuna verbatim (ExperimentalWarning aside - removed on purpose earlier);
multi-objective trials_dataframe columns (flat and multi_index) match pandas;
3,000 TPE trials take 11.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…2 on one journal file

Optuna's shared-file-system recipe: the parent creates the study in a
JournalFileBackend file and starts three worker processes (the same program with
--worker), each joining via load_study and running 16 trials on two threads with
HyperbandPruner; then it audits the file. Objective: mini-batch SGD ridge
regression (optional pairwise-product features) on a deterministic house-price
set, validation RMSE reported every epoch so Hyperband's brackets compare trials
from different processes at the same epoch.

Run here: 48 trials, numbers 0..47 each exactly once, 34 pruned / 14 complete,
546 of 1,296 epochs trained (58 % saved), best RMSE 0.098 at the 0.1 noise floor
(mean predictor 0.676). No defect found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ted default recipe

Every trial of an L2 logistic churn scorecard (optional quadratic features,
gradient descent) reports its 5 CV fold accuracies with
TerminatorFunctions.ReportCrossValidationScores, and the default
TerminatorCallback (GP regret bound vs CrossValidationErrorEvaluator) ends the
study. The same seeded search is also run without the terminator for
comparison: it stopped at trial 20 of 120, and the remaining 100 trials would
have added +0.05 percentage points - inside the +-2 point fold noise it reads
from terminator:cv_scores. The per-trial `optuna.terminator` FutureWarning is
Optuna's own behavior (deprecated in v4.9). No defect found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ython) through the gRPC storage proxy

The database host runs GrpcProxyServer.MakeServer over an RDBStorage SQLite file
on a free loopback port (the hosted, stoppable form of run_grpc_proxy_server);
the study is created through a GrpcStorageProxy like any remote client; four
.NET workers (own proxy connection each, TPE seeds 1-4) and - when python with
optuna is on PATH - a Python Optuna worker using optuna.storages.GrpcStorageProxy
optimize it together. Objective: a quarter's gross profit under constant-
elasticity demand with a decaying promo lift and a post-promo dip.

Run here: 70 trials (4 x 15 .NET + 10 Python) in one study, best $222.5k found by
a .NET worker; the C# server and the Python client interoperate over the wire.
No defect found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…the managed random forest

The scikit-learn-style integration as a real job: OptunaSearchCV cross-validates
Core's byte-exact RandomForestRegressor port (SklearnEstimator, KFold(5, shuffle))
over n_estimators / max_depth / min_samples_leaf / max_features to predict a
building's heating load from geometry and glazing (a deterministic 768-design
table shaped like the classic energy-efficiency study); the refitted best forest
is scored on 200 held-out designs: CV R^2 0.9972, held-out R^2 0.9972, MAE 0.33.
The per-trial terminator FutureWarning is Optuna's own behavior. No defect found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…int/categorical space with constraints

Concrete mix design as Bayesian optimization: cement content (5 kg steps),
water-cement ratio (0.01 steps), slag and fly-ash replacement (int, step 5),
superplasticizer dose (continuous) and the aggregate type (categorical),
maximizing 28-day strength from Abrams' law with SCM and aggregate factors,
under SetConstraint cost <= $95/m^3 and slump >= 75 mm. 12 random pours then 28
GP-planned: 57.1 MPa (both constraints nearly active) vs 44.7 MPa best random.

Live parity probe (40 trials): every stepped, integer and categorical choice is
identical to Optuna 5.0.0's GPSampler; the continuous dose differs by <= 2.1e-4
relative from trial 24 on - the documented GP / L-BFGS-B tolerance class. No
defect found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s as the only path, opt-in switches where results would change

The owner asked for a plan to integrate Rustuna's optimizations, with an evaluation of where a dual implementation
is warranted (a user choosing Optuna's path or Rustuna's). Rustuna 0.2.1, the Optuna team's Rust port, runs its
announcement's workloads 3-60x faster than OptunaSharp (TPE 7x, CMA-ES 100k 60x, NSGA-II 3-4x), while OptunaSharp
is 3-5x faster than Optuna 5.0 and byte-identical to it.

Decisions recorded (owner, 2026-11-08, through two rounds of design questions):
- Rule: "prefer rustuna unless it affects results". A Rustuna technique that keeps every observable result becomes
  the only implementation; one that changes results is measured and decided per item.
- CMA-ES reusing the optimizer it just told (Rustuna's live object): opt-in, Optuna's path the default.
- TPE fast numerics: decided after the byte-identical fused kernel exists and can be measured.
- Separate trial queue and trial discarding: deferred. The owner asked whether Rustuna has a TrialQueue (it does:
  an in-memory default plus SQLite3 and directory backends, Redis planned). For discarding the owner asked for
  "what's best + don't break Optuna compatibility", so the plan proposes compact storage of finished trials first.

Evidence in the plan, all measured in session 5417a82e on the owner's i9-13900K (Release, throttling opt-out):
- A scratch prototype of the CMA-ES part (a patched copy of d0558f5) took 10k trials from 18 s to 2.6 s and 100k
  from 812 s to 26 s (single-threaded OpenBLAS), byte-identical: five seeded studies hashed over every trial and its
  stored pickle, 86 unit and 29 live-Python interop CMA tests, mixed C#/Python fleet included. Its three changes:
  an optimizer cache dropped after each tell, a folded history, zero-copy reads.
- How different the result-changing paths are. Keeping the told CMA-ES optimizer diverges from generation 3
  (eigenvector sign flips make the first difference large), with the same quality over 10 seeds and about 15 % more
  speed. TPE scoring noise up to 1e-6 changed no decision in three 1,000-trial runs (smallest gap 3.4e-4); MOTPE
  stayed identical up to 1e-9. The queue is identical in one process; discarding changes study.Trials.
- Profiles behind each item: copy-on-write storage clones 66 % of NSGA-II and 23 % of MOTPE; the MOTPE split,
  recomputed per parameter, 24 %; TPE scoring 58 %.

The plan: an inventory of 12 same-result items (S1-S12), 4 two-path items (D1-D4) and 6 rejected ones, each with
Rustuna's code reference, our current code and the evidence. Then the rule for one path or two, the switch design
(on the component, named for the behavior), phases 0-8, gates (Python as the oracle on every CI OS, concurrency
suites, Release benchmarks, both paths tested), targets and risks.

docs/plans/README.md lists plan 13 and describes the performance workstream.
…istory incrementally, same results as Optuna; opt-in reuseToldOptimizer

Plan 13 (docs/plans/13-rustuna-derived-performance.md), phase 1 (items S1-S3) and item D1. Optuna's
CmaEsSampler restores its optimizer from the pickled state and rescans every trial on every ask, which makes a run
quadratic in its trial count (40-D, seed 0: 18.3 s for 10k trials, 812 s for 100k; Rustuna 0.2.1 takes 1.6 s and
13.6 s). The scratch prototype of session 5417a82e took 100k trials to about 26 s with byte-identical results; this
is that prototype, rewritten for production.

What changed, none of it observable:
- Samplers/CmaEs/CmaTrialHistory.cs (new): Optuna's per-ask _get_trials, _restore_optimizer backward scan and
  _get_solution_trials, kept up to date incrementally. Finished trials cannot change, so the leading run of finished
  trials is folded once, in trial order, into a completed count, a by-generation index and the latest state holder.
  Every refresh classifies only the trials after the first unfinished one, which keeps Optuna's scan order when
  trials finish out of order. A history that is not the one folded (another storage, study, or trials at the folded
  positions) starts over and tells the owner, so the sampler drops anything keyed by trial id.
- CmaEsSampler: the optimizer deserialized from the latest stored state is cached under the id of the trial holding
  that state and reused by the generation's remaining asks, eigendecomposition included. It is dropped after every
  tell: Optuna's pickle stores only C's upper triangle and a restore mirrors it, while the told C keeps the update's
  floating-point asymmetry. Measured in the prototype: keeping it moved seeded 40-D results from generation 3.
  GetTrials/GetSolutionTrials are replaced by the history; the per-trial filter is AsSamplerTrial, so a pruned
  trial's value-rewritten copy is made once when folded. SampleIndependent's warning check reads the history too.
- BaseStorage.HasZeroCopyTrialsView (internal, true for InMemoryStorage) and Study.GetTrialsForSampler: samplers
  that read the history every ask use the zero-copy view where it is one, and Optuna's read (including the per-trial
  cache) elsewhere, so a database-backed storage makes no extra round trip.
- IntersectionSearchSpace.Calculate reads through GetTrialsForSampler: its walk already stopped at the cached trial
  number; the full copy of the history was what grew with the study.

D1, opt-in (owner's decision: Optuna's path stays the default): CmaEsSampler(..., reuseToldOptimizer: false). When
true, the optimizer that just told a generation keeps answering the asks after its trial completes, as Rustuna keeps
its live optimizer: about 15 % faster on a 40-D study, and seeded runs stop matching Python Optuna from about the
third generation (same quality: 40-D sphere, 10 seeds, best-value ratio median 0.79, range 0.51-1.48). The state is
still stored every generation exactly as Optuna stores it, so resuming and Python workers keep working. The
parameter comes after every Optuna parameter.

Tests:
- CmaTrialHistoryTests: randomly evolving studies (600 steps, 5 seeds; out-of-order completion, failures, pruned
  trials with and without intermediate values, attributes written while running) compared at every step with a naive
  re-implementation of Optuna's scans; a stuck running trial; the pruned copy made once; reset on another storage,
  study, folded trial or a shorter history; a holder in the tail; null arguments.
- CmaEsSamplerCacheTests: one restore per tell in a sequential run (RestoreCount, internal); a failed teller's
  generation told again from the previous state, the stored state identical byte for byte; a second study on
  another in-memory storage (colliding trial ids) never reuses the first study's optimizer; GetTrialsForSampler is
  the zero-copy view in memory and Optuna's read on a journal file; 8 parallel workers, shared and per-worker. D1:
  off by default (identical trials, one restore per tell), one restore per run when on, a failed teller restores
  the previous state, a default sampler resumes a study the reusing sampler wrote, and both paths reach 1e-6 on a
  10-D sphere for three seeds.
- Interop, live Python: long runs (10-D 400 trials, 6-D 300), 40-D 1,200 trials ([Slow]), failing tellers (every
  other teller fails, both languages failing the same trials), consider_pruned_trials, SepCMA and lr-adapt over
  30+ generations: every trial and every stored pickle equal Optuna's. The mixed C#/Python fleet with the C# worker
  reusing its told optimizer completes with the reference generations.

Results: the CI unit filter (OpenBugs/HighMemory/RequiresPython/Slow excluded) 1998/1998 on net8.0 and on net10.0;
unit CMA-ES/pickle/search-space 103 pass with no filter (the 3 failures are known [OpenBugs]); concurrency suites
243/243; live-Python CMA-ES suite 41/41 (engine required); documentation gate 0 issues.

Phase 2 of the plan (in-place writes to a running trial nobody holds, S4; shared distributions and parameter names,
S7) follows in its own commit; InMemoryStorage.cs carries only the HasZeroCopyTrialsView override here.
…t, share identical distributions and names; same results as Optuna

Plan 13 (docs/plans/13-rustuna-derived-performance.md), phase 2: items S4 and S7.

S4. Optuna's _InMemoryStorage copies a trial's record and the dictionary it writes on every write
(set_trial_param, set_trial_user_attr, set_trial_system_attr, set_trial_intermediate_value), so a FrozenTrial a
caller holds never changes. The port did the same, which made a trial's suggestions O(d^2): the profile of an
NSGA-II study with 40 parameters spent ~66 % in SetTrialParam's dictionary clones (plan 13 section 2.2). Rustuna
inserts in place and clones only for readers. Now:
- FrozenTrial.StorageExclusive (internal): true while the storage alone holds a record. CreateNewTrial sets it
  (both branches build dictionaries nobody else holds: a new running trial or a deep copy of the template).
- InMemoryStorage.WritableTrialUnlocked: a write changes the stored record in place while it is exclusive;
  otherwise it first replaces it with a copy that owns all five dictionaries (any later write may touch any of
  them), so the reader's record keeps its contents.
- HandOut: every read that returns the stored record itself clears the flag under the lock: GetTrial,
  GetBestTrial, GetAllTrials(deepcopy: false) (unfinished records only; finished ones are never written again),
  and through them the attribute reads and GetNTrials. Deep copies and GetTrialParam hand nothing out.
- GetTrialsPrefixView passes TrialsPrefixView a read for unfinished records (new internal constructor): taken
  under the storage lock with HandOut. Finished records stay lock-free, as before: a state change still publishes a
  new record whole (SetTrialStateValues keeps Optuna's copy), so a record that reads as finished is final.
- InMemoryStorage.WriteCopyCount (internal diagnostic, like CmaEsSampler.RestoreCount): writes that had to copy.
In an ask/suggest/tell loop the trial's own read at Ask is the only hand-out between its suggestions: about one
copy per trial instead of one per write.

S7. SetTrialParam stores one distribution instance per parameter across the study's trials when the new one is
identical to the bit (Distribution.IsBitwiseIdentical: the same runtime type, bounds/step compared by bits, log
flag), and one string instance per parameter name (StudyInfo.ParamNames), so names built per call ($"x{i}") are
not kept once per trial. Never shared: categoricals (equal choices may be different objects, 1 and 1.0 under
Python equality, and a trial must report its own), a -0.0 bound (equal to 0.0, printed differently), a deprecated
subclass against its base. Compatibility is still checked first, with Optuna's messages. Distributions are never
mutated after construction (only the constructor sets FloatDistribution.Step), so sharing is unobservable.

Docs: TrialsPrefixView remarks (liveness, the in-memory hand-out), THREADING.md section 6.1 (the snapshot
guarantee as it is now kept) and the lock inventory (the view takes the in-memory storage monitor per unfinished
record; it is still the last lock a sampler takes, so the order of section 5.2 holds).

Tests:
- InMemoryStorageInPlaceWritesTests (13 cases): every read path (GetTrial, GetAllTrials with no filter, RUNNING,
  WAITING, the view's indexer and enumerator, the three attribute reads, GetBestTrial of another trial) keeps its
  snapshot through two writes of each kind, and every write lands; deep copies leave the record writable in place;
  writes nobody reads between go in place (WriteCopyCount 0 over 20 parameters and 3 attributes) and each hand-out,
  GetNTrials included, costs exactly one copy; a state change makes the next write copy once and a templated trial
  never writes its template; the view returns the current record on every read and finished records as stored; a
  finished trial refuses writes; an ask/suggest/tell loop copies at most once per trial. S7: identical float,
  stepped float and int distributions shared, compatible-but-different ones keep the caller's instance; -0.0 keeps
  its instance and repr; categoricals never shared (a [1.0, 2.0] trial after a [1, 2] one reports 1.0); names stored
  once per study; IsBitwiseIdentical case by case; incompatible distributions still refused with Optuna's messages.
- StorageConcurrencyCoverageTests: new ConcurrencyGun case, the view and GetTrial read while 7 shooters write their
  own trials in place (30 rounds): no fingerprint changes under the reader, and the writers did copy held records.
  The existing GetAllTrials(deepcopy: false) case's summary now describes how the in-memory storage keeps it.
- Mutation check of the new tests (each reverted): the view handing out without marking fails 4 tests (including the
  view race case); a write copy that shares IntermediateValues fails 10; GetAllTrials(deepcopy: false) not marking
  fails 5 (including the existing race case).

Results: the CI unit filter (OpenBugs/HighMemory/RequiresPython/Slow excluded) 2022/2022 on net8.0 and on net10.0;
live-Python interop on net10.0 with the engine required 682 passed, 0 failed (24 not run here for services and Python
packages this machine lacks: Redis, and the pycma bridge and OptunaHub constraint fixtures); documentation gate 0
issues. Seeded 40-parameter studies hash identically to de495f8 (every trial's parameters and values, by bits):
random 2,000, NSGA-II 2,000, CMA-ES 1,500, TPE 150 and MOTPE 120 trials.
… history reads; same results as before

Plan 13 (docs/plans/13-rustuna-derived-performance.md), phase 3: items S6, S5 and S3 (TPE's constant-liar read,
QMC, the group-decomposed search space). Every change keeps OptunaSharp's results to the bit: seeded 40-parameter
studies hash identically to de495f8 (TPE 1,000 trials, MOTPE 1,000, QMC), and the live-Python TPE suite, now with
1,000- and 300-trial long runs, stays within its erf tolerance.

Measured (i9-13900K, Release, power throttling opted out, one rep each unless noted; 40 float parameters, seed 0):
- TPE, 1,000 trials: 10.08 s -> 4.76 s (first rep, JIT included), 3.93 s steady; allocations 4.4 GB -> 1.6 GB, gen-2
  collections 254 -> 2.
- MOTPE (two objectives), 1,000 trials: 41.07 s -> 15.77 s; allocations 28.7 GB -> 3.4 GB, gen-2 713 -> 1.
- MOTPE, 300 trials: 3.75 s -> 2.53 s.

S6, the TPE scoring kernel (Samplers/Tpe/ProbabilityDistributions.cs, ParzenEstimator.cs, Internal/TruncNorm.cs,
Internal/Erf.cs):
- AddContinuousLogPdf computed log(sigma), the standardized bounds and their log Gaussian mass once per candidate for
  every (kernel, dimension) pair: 936k Math.Log calls per trial at 1,000 trials and 40 dimensions, more than half
  the run. They are now computed once per pair, in grids laid out by dimension, and AccumulateDimension adds one
  dimension's term to every kernel's sum on Vector<double> lanes. Each kernel's sum still runs over the dimensions
  in order from +0.0, lanes hold different kernels and never mix, and .NET never contracts a multiply and an add
  into an FMA, so every lane computes what the scalar loop computed; the remainder takes the scalar path.
  LogGaussMass is elementwise and the vectorized-erf switch depends only on how many elements it gets, so laying the
  grid out by dimension changes no element.
- LogPdf takes each kernel's log weight once instead of once per candidate, and keeps its (candidates x kernels)
  matrix in a rented flat buffer: at 1,000 kernels it was a 192 KB large-object allocation per call, and the
  sampler waited on background GCs for them.
- TruncNorm.LogGaussMass and Ndtr, and Erf.ComputeArray, gained span overloads over rented scratch (the array
  overloads now wrap them). The central elements are gathered into spans of exactly the central count, so the erf
  path is chosen from the same length as before; a test pins that a span of 1,999 inside a 5,000-element buffer
  takes the exact path.
- ParzenEstimator.StableArgsort replaces Enumerable.Range(0, n).OrderBy(i => mus[i]): Array.Sort on a copy of the
  keys, then an index sort inside each run of equal keys (by CompareTo: NaNs together, -0.0 with 0.0), which is the
  same permutation the stable LINQ sort returned.
- TPESampler.GetInternalRepr fills pre-sized columns by position, without the per-trial LINQ closure, the
  dictionary lookup per value and the list regrowth; same values in the same order.

S5, the split cache (Internal/TpeTrialSplit.cs: TpeSplitCache; TPESampler): independent TPE (the default for several
objectives) split the history once per parameter, and every parameter of a trial sees the same records, so a
40-parameter trial computed the same non-domination ranks and hypervolume tie-break 40 times. The cache returns the
last split again only for the very same records (compared by reference, in order) of the same study and storage with
the same n_below: finished records never change, and a record changes reference on any write after a reader got it
(state changes publish a new record; InMemoryStorage copies a handed-out record before writing it). Returned lists are
copies. One cache per sampler instance, so a PerWorker worker has its own (TPESampler.SplitCache, internal, for tests).

S3:
- TPE with constant liar (Optuna 5's default) read the whole history including running trials for every parameter and
  then dropped the current trial; that read handed out the current trial's record, so every suggestion copied it
  again under item S4. On a storage with a zero-copy view it now reads through the view and skips the current trial's
  position without reading it (TPESampler.ReadOtherTrials): the same records in the same order, minus the current one.
- QMCSampler copied the COMPLETE/PRUNED history and sorted it by number on every ask to take the first one;
  FirstSuggestedTrial scans the history in number order through Study.GetTrialsForSampler and stops at the first
  match (O(1) on an in-memory storage, Optuna's read elsewhere).
- GroupDecomposedSearchSpace filters the states while scanning Study.GetTrialsForSampler instead of a copied list.

Tests:
- TpeExactKernelTests (unit): the continuous log-pdf against a verbatim scalar copy of the replaced code, bit for bit,
  for every kernel count from 1 to three vectors and more (every remainder), 1-40 dimensions, log dimensions,
  candidates outside the bounds; 1,000-kernel grids that take the vectorized erf; an empty interval (NaN, as before);
  LogGaussMass over rented scratch against the replaced allocation-per-call code for 1 to 6,000 intervals around the
  2,000 switch, NaN bounds and all three cases; the span erf choosing its path by the span's length; StableArgsort
  equal to the LINQ permutation over 300 arrays with ties, NaNs, both zeros and infinities.
- TpeSplitCacheTests (unit): the cached split equals SplitTrials for every n_below over a history with completed,
  pruned (with and without reports), infeasible and running trials; reuse only for the same records (one fewer,
  another order, a rewritten running record, another study all miss); returned lists are copies; null arguments; an
  independent multi-objective study splits once per trial, not per parameter, with constant liar on; constant liar
  still sees another worker's running trial and leaves the current one out; a PerWorker instance has its own cache.
- ZeroCopyHistoryReadTests (unit): QMC's search space comes from the lowest-numbered COMPLETE or PRUNED trial with
  trials finished out of order and running/failed trials before it; the running-trial union before anything finished;
  the group-decomposed space equals Optuna's decomposition of the copied, filtered history (same groups, order and
  distribution objects) with and without pruned trials and the cache.
- Interop, live Python, [Slow]: TPE multivariate 40-D 1,000 trials and MOTPE 40-D 300 trials stay in lock-step with
  Optuna within the erf tolerance; NSGA-II 40 parameters 2,000 trials is bit-identical in full state (plan 13 phase 0
  gates for S4-S6).
- The 8 [OpenBugs] TPE/PED-ANOVA QA cases fail with byte-identical messages before and after (NaN acquisition values
  and NaN prior weights included), so the edge behavior they pin is unchanged.

Also: OptunaHubBridgeParityTests.BenchmarkBridge_ToyProblem_ExposesSpaceAndOptimizes now seeds its TPE sampler. It ran
an unseeded study (OS-entropy seed) and asserted that the best of 40 trials is below 1.0, which fails in a few percent
of runs: it failed once in this phase's interop run (best 1.018) and passed eight reruns. Seeded studies are
bit-identical before and after this phase, so every OS seed maps to the same outcome in both trees and the failure rate
was unchanged; the test is about the bridge, so the seed is fixed (seed 0 passes).

Results: the CI unit filter (OpenBugs/HighMemory/RequiresPython/Slow excluded) 2044/2044 on net8.0 and on net10.0 with
S5/S6 and TPE's constant-liar read; the QMC and group-decomposition change and its 7 tests afterwards, with the QMC,
group and TPE suites: 51/51. Live-Python interop on net10.0 with the engine required, the three new long runs included:
684 passed, 24 not run here (Redis, the pycma bridge and OptunaHub constraint fixtures), 1 failed: the stochastic
OptunaHub toy test above. Long runs: TPE 1,000 trials 1 min 26 s, MOTPE 300 trials 58 s, NSGA-II 2,000 trials 16.5 s,
all passing. Documentation gate 0 issues (5,609 declarations).
…s it as a view; database loads parse each distribution once

Plan 13 (docs/plans/13-rustuna-derived-performance.md), phase 4: item S10, and item S9 as its measurement turned out.
Same records, same order, same values as before.

S10, CachedStorage (the wrapper Optuna puts around every RDB storage):
- GetAllTrials sorted the cached dictionary by trial number on every read, and GetTrialsPrefixView fell back to that
  copy, so the genetic samplers, which take the view on every ask, paid O(n log n) per ask on a database storage.
- StudyInfo.ByNumber mirrors the cache by trial number (AddTrialsToCache keeps it in step). A number not cached yet
  holds a null placeholder counted in MissingNumbers: a local create can cache trial n + 1 before a sync brings in a
  trial n another worker created.
- With no placeholder, GetAllTrials walks the mirror (the same records in the same order as the sort) and
  GetTrialsPrefixView wraps it in a TrialsPrefixView after the same synchronization GetAllTrials does. Cached records
  are never changed in place (writes go to the backend; a sync swaps in fresh records), so the view needs no lock.
  With a placeholder, both use the sorted copy, as before.
- HasZeroCopyTrialsView stays false here: samplers that share the per-trial cached list (GetTrialsForSampler) keep it,
  since a view read means a synchronization, which is a database round trip.

S9, measured first as the plan asked (10,000-trial, 40-parameter CMA-ES study on SQLite, cold load with a new
RDBStorage, i9-13900K, Release):
- GetAllTrials took 2.1 s and allocated 1.22 GB. Decoding every attribute value (26k values, 30 MB of JSON, the CMA-ES
  pickles included) took 0.06-0.08 s of it: lazy attribute decoding would save under 4 % and move malformed-JSON
  errors from load to access. Not built.
- The profile showed where the JSON time really goes: Distribution.FromJson for every parameter row, 400,000 parses
  of 40 distinct texts, about 40 % of the load.
- Both RDB storages (Storage.Sqlite and Storage.EFCore) now parse each distinct distribution text once per load
  through the journal storage's JournalDistributionCache (plan 10, item 2.7): the shared object equals a fresh parse,
  only the three current distribution types are shared (the deprecated ones are built per use, so their warnings
  fire as before), and the cache is local to the call. Cold load: 2.1 s -> 1.4-1.5 s, 1.22 GB -> 479 MB.

Tests:
- CachedStorageOrderedReadsTests (unit, over a fake database: an in-memory store returning fresh records per read
  that can hide another worker's trial from a sync): every state filter with and without deep copies returns the
  backend's trials in number order over a history finished out of order; the view is a TrialsPrefixView holding
  the very records of the uncopied read and sees an unfinished trial a later sync refreshed; a gap falls back to the
  sorted copy (the view to a List) until a sync fills it; seeded NSGA-II (which reads the view every ask) and CMA-ES
  studies on the cached fake database equal the in-memory runs trial by trial.
- SqliteRdbStorageTests.Reload_ParsesEachDistributionTextOnce_SharingOnlyTheCurrentTypes: a reloaded study shares one
  object per distinct text, each equal to a fresh parse of its stored text; a -0.0 bound is its own object and prints
  -0.0; a deprecated UniformDistribution is never shared; parameter values unchanged.

Results: the CI unit filter 2056/2056 on net8.0 and on net10.0; the storage suite on net10.0 452 passed, 144 not run
here (Redis, PostgreSQL and MySQL services), 0 failed; live-Python interop for the SQLite, EF Core, storage-URL and
storage-recipe cases 33/33 (Python writes and reads databases this loader parses); documentation gate 0 issues.
…ce and Compatibility guide; README rerun

Plan 13 (docs/plans/13-rustuna-derived-performance.md), phase 8: the documentation, and gate C's measurements.

Before/after proof (gate C): one program (session 5417a82e, scratchpad/proof) built in Release against the de495f8
snapshot and against 656ab94; 40 float parameters, seed 0, sampler defaults; each workload before then after, back to
back, on a machine other sessions loaded to 30-90 % CPU. Every pair hashed identically (every trial's parameters and
values by bits):
- CMA-ES 100,000 trials: 987.7 s -> 27.5 s (35.9x), allocations 519 GB -> 17.4 GB, peak working set 2.68 -> 1.88 GB;
- CMA-ES 10,000: 25.4 -> 4.5 s (rep 0), 23.0 -> 2.6 s (rep 1);
- NSGA-II 100,000: 73.2 -> 22.2 s, peak working set 1.53 -> 0.75 GB; NSGA-II 10,000: 3.8 -> 2.9 s / 2.7 -> 1.8 s;
- TPE 1,000: 10.5 -> 5.0 s / 9.4 -> 4.1 s, gen-2 collections 254 -> 2; MOTPE 1,000: 47.4 -> 18.1 s;
- QMC 10,000: 4.8 -> 2.5 s / 3.4 -> 1.05 s; Random 10,000: 5.8 -> 4.7 s / 4.9 -> 4.0 s.
- The opt-in reuseToldOptimizer (D1): CMA-ES 10,000 2.0 s steady, 100,000 18.2 s; results differ, by design.

docs/plans/13-rustuna-derived-performance.md: status BUILT except S8/S12; section 10 records each item, its commit,
how it was gated and what was measured, the findings (the pickle's upper triangle behind D1, GetNTrials' hand-outs,
dotnet-trace's suspensions showing up as lock and PollGC time), and the proof table. S9 is recorded as measured
(attribute JSON 3-4 % of a cold load; the distribution parse cache built instead), S11 as measured and unchanged
(the OpenBLAS thread count changes 100-D CMA-ES results, as it does NumPy's), S8 and S12 as left for the owner with
their numbers, D2 as asked with both of its numbers (scoring is 41 % of TPE 1k and 49 % of MOTPE 1k after S6).
docs/plans/README.md: plan 13 marked built; the workstream paragraph updated.

docs/website-src/docs/performance.md (new, in the Guides TOC): what is fast and why results stay the same, the
before/after table, the switches that change results (CmaEsSampler reuseToldOptimizer; TrialExecutorOptions.BlasThreads
= 1 at nJobs 1) with whether seeded runs still match Python and whether Python workers can share the storage, and
the thread-safety guarantee.

docs/THREADING.md section 6.3: the plan 13 caches (CMA-ES restored optimizer and folded history, TPE split cache) are
sampler state under the sampler monitor, one per PerWorker instance, keyed by what they were computed from.

README.md, Performance: benchmark 14's C# probes rebuilt against 656ab94, three fresh processes each. Rows that moved
beyond the busy machine's noise are updated (random over 1,000 parameters 13.5 -> 3.2 s, now faster than Python's
8.4 s, so the "one case where OptunaSharp is slower" sentence is gone; CMA-ES 200-D 2.8 -> 1.2 s; CMA-ES 6-D 1.75 ->
0.91 s; QMC 0.64 -> 0.39 s; random 6-D 1.34 -> 0.98 s; TPE 100-D 2.4 -> 2.1 s; the 50k blocks 0.6-0.8 s); the rest
keep their numbers, and GP was checked interleaved against a de495f8 build (same within noise). A new "Large studies"
table puts OptunaSharp next to Python Optuna 5.0 and Rustuna 0.2.1 on the announcement's workloads, with the
measurement conditions stated (OptunaSharp's column ran on the busier machine).

DocFX 2.78 builds the site with 0 errors; the new page adds no warning (the 1,160 warnings are the existing duplicate-<param> notes and one old link in our-methods/nsga-iii-cpt.md).
…5885 made ambiguous

dce5885 (plan 13, item S6: the fused TPE kernel) added span overloads
TruncNorm.LogGaussMass(ReadOnlySpan<double>, ReadOnlySpan<double>, Span<double>) and
Erf.ComputeArray(ReadOnlySpan<double>, Span<double>) next to the array ones. Three doc comments
still named the methods without a signature:

- src/OptunaSharp.Core/Internal/TruncNorm.cs:21 (class remarks) -> Erf.ComputeArray(double[]),
  the array overload the sentence is about (Optuna's >= 2000-element vectorized erf);
- src/OptunaSharp.Core/Samplers/Tpe/ProbabilityDistributions.cs:491-492 (the grid precompute's
  remarks) -> the span overloads, which is what that code calls.

A plain build does not show this: the projects keep NoWarn for the doc warnings, and only the
documentation workflow (.github/workflows/documentation.yml) rebuilds with -p:NoWarn= and
CS0419 among WarningsAsErrors. Found by running that gate with --no-incremental; an incremental
rebuild skips CoreCompile and reports 0 errors, so the gate has to run on a fresh compile.

No code change.
…), with Optuna-named overloads

Rustuna's storages take a batch of attributes (set_trial_user_attrs and friends) and write it under
one lock / in one transaction. Optuna has no such call: a caller loops over set_trial_user_attr,
and every call takes the storage lock, checks the trial, and (in a database) commits on its own.
The owner chose to build it (2026-11-09: "Build it", "Also have optuna's api overload"), so the
batch exists under Rustuna's plural names and under Optuna's singular names as overloads.

The result never differs from the loop: every override leaves exactly what calling the single-key
method once per entry, in the dictionary's enumeration order, would leave. What changes is the
cost, and that a rejected batch writes nothing.

API (all documented with params, exceptions and remarks):
- BaseStorage: virtual SetStudyUserAttrs / SetStudySystemAttrs / SetTrialUserAttrs /
  SetTrialSystemAttrs(id, IReadOnlyDictionary<string, object?>). The default body is the loop, so
  every third-party storage supports the call unchanged (atomic only as far as each single call).
  Non-virtual Optuna-name overloads SetStudyUserAttr(id, dict), SetStudySystemAttr(id, dict),
  SetTrialUserAttr(id, dict), SetTrialSystemAttr(id, dict) forward to them.
- BaseTrial.SetUserAttrs(dict) (virtual, default loop) + SetUserAttr(dict) overload;
  Trial.SetUserAttrs overrides it: one storage call, then the trial's own frozen view and the
  thread-cache mirror take every entry, exactly as SetUserAttr(key, value) updates them per key.
- Study.SetUserAttrs(dict) + SetUserAttr(dict) overload -> storage.SetStudyUserAttrs.
- Empty dictionary: writes nothing and checks nothing (no KeyError for an unknown id), in every
  override; null throws ArgumentNullException.

Per storage:
- InMemoryStorage: one lock, one existence/updatable check, and at most one record copy for the
  whole batch (WritableTrialUnlocked from S4: in place while no reader holds the record). Study
  attributes: one ShallowClone of the dictionary, every entry written into it, published once.
- JournalStorage: the batch is N ordinary SINGLE-KEY records appended with one AppendLogs. Not one
  record with N keys: Python's replay asserts `len(log["user_attr"]) == 1`
  (optuna/storages/journal/_storage.py, _apply_set_trial_user_attr and siblings), so a multi-key
  record would make the file unreadable to Optuna. Under the thread lock: sync, check the trial
  (new JournalStorageReplayResult.ThrowIfTrialNotUpdatable: KeyError NotFoundMsg /
  UpdateFinishedTrialError UnupdatableMsgFormat, the replay's own messages) or the study
  (ThrowIfStudyMissing), stamp the worker id, append, then SyncAfterBatch. SyncAfterBatch exists
  because the replay throws for a rejected record OF THIS WORKER after advancing past it, leaving
  the rest of the batch unread: it syncs up to batch-size times, keeps the first exception and
  rethrows it after the batch is drained, so one rejected record (another process finished the
  trial between our check and our append) does not poison the next call.
  Test hooks: AppendBatchAndSyncForTests, WorkerIdForTests (internal).
- CachedStorage: forwards each batch whole to the backend (its own cache is read-through for
  attributes, as for the single calls).
- SQLite RDBStorage: one transaction per batch: the study or trial (FindTrialOrRaise +
  CheckTrialIsUpdatableOrRaise) is checked once, every entry upserted with the same SQL as the
  single call, one commit. A failure rolls back the whole batch.
- EF Core RDBStorage: the same through Mutate(ctx => transaction; check; add/update each entry;
  SaveChanges; Commit).

Docs: three crefs that named BaseTrial.SetUserAttr / Study.SetUserAttr without a signature became
ambiguous with the new overload (ComponentLosses.cs, FrozenStudy.cs, StudySummary.cs, and one in
JanusMooCmixSamplerTests.cs); they now name (string, object?). The documentation gate
(check_documentation.cs over src: 0 issues; Core rebuilt --no-incremental with the workflow's
WarningsAsErrors: 0 errors) passes.

Tests:
- test/OptunaSharp.Tests/Storages/BulkAttributeWritesTests.cs (new, 14 cases): batch == loop for
  in-memory, journal and cached storage (trial user/system, study user/system, overwrite order);
  the Optuna-name overloads; empty and null dictionaries; finished/unknown trial and unknown study
  write nothing and throw the single call's error; Trial.SetUserAttrs updates the trial's own view
  and a later frozen copy; a storage that does not override gets the loop (DelegatingStorage test
  double); the journal writes N single-key records with one worker id; and the drain test (a
  rejected own record followed by valid ones: the call throws the first error and the next call
  works). The drain test was checked against a mutation (SyncAfterBatch reduced to one sync): it
  fails there.
- test/OptunaSharp.Tests.Storage/SqliteRdbStorageTests.cs (+3): batch leaves what single writes
  leave (read back through a fresh storage), all-or-nothing on a finished trial, and the batch
  through CachedStorage reaches the database.
- test/OptunaSharp.Tests.Interop/JournalInteropTests.cs (+1, live Python):
  CSharp_WritesAttributeBatches_Python_ReadsEveryOne: C# writes a trial user batch (int, float,
  string, bool, None, list), a trial system batch, a study user batch and a study system batch to
  a journal file; live Optuna 5.0 loads the study, reads every attribute with its value
  (json.dumps compared byte-for-byte), continues the study with two trials, and C# reloads all
  three.

Runs: full CI unit filter (TestCategory!=OpenBugs&!=HighMemory&!=RequiresPython&!=Slow) on both
TFMs: 2070 passed, 1 skipped, 0 failed (net8.0 and net10.0). Storage suite cases and the interop
case pass.
…n 13, item S12); same trials, a fifth of the memory

Owner, 2026-11-09: S12 "Build it", "If it changes optuna's/rustuna's behavior then make the feature optin". It changes
no result (seeded studies give the same trials, bit for bit, and live-Optuna parity holds on it), but finished trials
stop being dictionaries: their read-only accessors return views, numbers and long strings are rebuilt per read, and a
finished record no longer shares its dictionaries with the running record it was copied from. So it is a switch, off
by default:

    new InMemoryStorage(compactFinishedTrials: true)    // also: storage.CompactFinishedTrials

What a finished trial becomes (src/OptunaSharp.Core/Internal/CompactTrialParts.cs, new):
- Parameters: CompactParams = one internal-representation double per name against a ParamSchema (names +
  distribution instances) shared by the study's trials through a per-study ParamSchemaCache (at most 256 shapes;
  further shapes keep their dictionaries). A trial is accepted only when every stored value comes back
  indistinguishable from d.ToExternalRepr(d.ToInternalRepr(stored)) (DistributionIdentity.IsSameValue: same type and
  bits for numbers, -0.0 stays -0.0; same characters for strings; the same object otherwise), so reads return what
  was stored. Fallbacks that keep the dictionary, each pinned by a test: an int under a float distribution (Optuna
  keeps it an int), an integer beyond 2^53, a categorical choice Python-equal to an earlier one (1 then 1.0), a value
  its distribution cannot convert, names listed in another order than the distributions.
- Distributions: the schema's one read-only DistributionsView, shared by every trial of the schema. Schemas match
  distributions by DistributionIdentity.AreInterchangeable: bitwise for float/int (S7's rule), identical choices for
  a plain CategoricalDistribution (every SuggestCategorical call builds a new distribution, so identity matching
  would give every categorical trial its own schema), the same instance for anything else.
- Attributes: CompactAttrs = key and value arrays (one shared Empty), long ASCII strings as CompactString bytes:
  ASCII one byte a char, lowercase hex two digits a byte with an odd trailing digit kept aside (CMA-ES splits its
  pickle hex into 2,045-char chunks), uppercase hex and non-ASCII stay strings. Nested containers stay the same objects.
- Intermediate values: an empty dictionary becomes a shared sentinel never handed out; a non-empty one a copy sized
  to its entries.

FrozenTrial (src/OptunaSharp.Core/FrozenTrial.cs):
- The four dictionary fields are typed IReadOnlyDictionary and hold either the OrderedDict or a compact form;
  Params/Distributions/UserAttrs/SystemAttrs return the field as is (unchanged objects on the default path).
- ParamsDict/DistributionsDict/UserAttrsDict/SystemAttrsDict and IntermediateValues return real dictionaries: on a
  compact record the first read materializes one and installs it with Interlocked.CompareExchange (Decompact), so
  however many threads ask at once they all get the one dictionary the record keeps, and a write through it stays.
- New internal: CompactFinished(schemas), IsCompact, IntermediateValuesForRead (the non-pinning read).
- Equals, ToString, DeepCopy, LastStep, Constraints, Validate read through the views (comparing and printing never
  decompact); SetUserAttr, Suggest's distribution write and SetConstraint write through the mutable accessors;
  ShallowCopy gives a compact record its own dictionaries first and shares them, as Python's copy.copy shares dicts.

InMemoryStorage: the switch; compaction in SetTrialStateValues on the new record before it is published (under the
lock, only reading the dictionaries the old running record may still share) and in CreateNewTrial for a template
that is already finished (Study.AddTrial). GetTrialParam returns a compact trial's stored internal value directly.

Read paths kept compact: about 70 internal reads of finished trials moved from the mutable accessors to the
read-only ones (no change without the switch: both return the same object): CMA-ES/DE/ES/LM-CMA/VkD-CMA/PSO/JANUS
Transform(t.Params); GP, SAASBO, TuRBO, the terminators; TPE's GetParams (now IReadOnlyDictionary); NSGA-II/MOEA-D/
NSGA-III-CPT crossovers and conditional params; QMC, intersection and group-decomposed search spaces
(SearchSpaceGroup.AddDistributions takes IReadOnlyDictionary); BaseGASampler generations; SuccessiveHalving,
Percentile and Wilcoxon pruners; importance evaluators; dataframe; retry callbacks; Study.ShouldSkipEnqueue;
visualization; the CLI records; the BoTorch and pymoo bridges.

TPE fast path: GetInternalRepr copies a compact trial's stored internal value instead of boxing and converting the
external one, for parameters whose search-space distribution is interchangeable with the trial's
(ParamSchema.PositionsFor). Exact by construction: compaction stored internal = d.ToInternalRepr(stored) only when
d.ToExternalRepr(internal) is indistinguishable from stored, and an interchangeable distribution converts as d does.

Tests:
- test/OptunaSharp.Tests/Storages/CompactFinishedTrialsTests.cs (new, 35 cases): every kind of value reads back
  identical (types, bits, order, reprs, constraints, LastStep, GetTrialParam); Equals/ToString match Optuna's
  representation both ways; views read like dictionaries (generic and non-generic, CopyTo, KeyNotFound message) and
  refuse writes; mutable accessors keep writes and stay single under 8-thread contention; the five fallbacks; schema
  sharing per shape incl. per-trial categorical distributions and the 256 cap; identity rules; compact strings;
  15 sampler families run seeded on and off to identical trials with every finished record still compact; CMA-ES
  chunks stored as hex bytes and restored by a second sampler; 4 workers + 3 readers; retained memory below a
  quarter. Mutation checks: TPE reading through ParamsDict fails exactly the 5 TPE rows; a dropped trailing hex digit
  fails 19 cases; a plain assignment instead of the compare-exchange fails the contention test 3 runs of 3.
- test/OptunaSharp.Tests.Interop/CompactStorageParityTests.cs (new, 9 cases, live Optuna 5.0): Random with long ASCII,
  odd hex, non-ASCII and nested attributes, NSGA-II mixed space, QMC and CMA-ES bit-identical full state (CMA-ES chunk
  strings equal to an uncompacted C# run's), TPE multivariate within the erf tolerance; records stay compact.

Proof (scratchpad proof-s12, the plan 13 workloads: 40 floats, seed 0, plain then compact back to back, every pair
hash-identical; retained = heap the finished study keeps after a full collection):
  workload            retained plain -> compact     peak WS         time (steady)      allocated
  random 10k          64.2 -> 6.7 MB (6732->704 B)  200 -> 91 MB    3.52 -> 3.53 s     2.31 -> 2.32 GB
  qmc 10k             64.2 -> 6.7 MB                198 -> 93 MB    1.02 -> 0.85 s     1.58 -> 1.60 GB
  nsga2 10k           67.7 -> 9.7 MB (7099->1012 B) 205 -> 103 MB   1.57 -> 1.44 s     1.23 -> 1.29 GB
  tpe 1k              13.4 -> 6.9 MB                109 -> 92 MB    3.84 -> 3.37 s     1.57 -> 1.60 GB
  motpe 1k            7.2 -> 1.4 MB                 76 -> 77 MB     12.7 -> 11.6 s     3.42 -> 3.45 GB
  cmaes 10k           129.5 -> 26.5 MB              372 -> 137 MB   2.15 -> 2.46 s *   1.74 -> 1.85 GB
  nsga2 100k          670.9 -> 87.5 MB (7035->917 B) 751 -> 167 MB  15.7 -> 14.6 s     12.4 -> 13.0 GB
  cmaes 100k          1287.3 -> 256.8 MB            1846 -> 458 MB  23.2 -> 22.1 s     17.4 -> 18.4 GB
  * repeated alternately 3x3: steady reps average 2.28 s plain vs 2.31 s compact, i.e. noise.
  Before the two allocation fixes found by gc-verbose profiling (TPE boxing its history read; the constraints scan
  rebuilding every attribute string), compact TPE allocated 3.06 GB vs 1.57 and MOTPE 3.90 vs 3.42; now equal.

Also fixed on the way (same results, both modes): GetConstraintsFromSystemAttrs scans keys and reads only matching
values, so a compact trial rebuilds no attribute string it does not need.

Gates on the final code: unit CI filter 2105 passed / 1 skipped / 0 failed on net8.0 and net10.0 (2070 before + 35);
live-Python interop (net10.0, engine required, minus Slow) 692 passed / 24 skipped / 0 failed; Storage 455 passed /
144 skipped (no Redis/PG/MySQL services) / 0 failed; CLI 91 passed + its 4 known [OpenBugs] QA reproductions;
check_documentation.cs 0 issues; Core and the touched satellites rebuilt --no-incremental with the doc gate's
WarningsAsErrors: 0 errors.

Docs: performance.md (Memory: compact finished trials; Writing several attributes at once), storage.md, README,
study-and-trials.md (SetUserAttrs), THREADING.md §6.1, plan 13 status, §10 rows (S8, S12, D2) and new §10.2.
…§10 row

17716c7 wrote "this commit" in the plan's §10 S12 row because a commit cannot know its own hash. This fills in the
hash there and in the status line, and the row's peak working set now quotes the final proof run that §10.2's table
reports (1,846 → 458 MB; the earlier figure, 1,845 → 460 MB, came from the second of three runs).

No code change.
…parameter paths (plan 14, step A); random search 5x, NSGA-II 1.9x, same trials

The second Rustuna comparison (catalog note optunasharp/notes/rustuna-comparison-2026-11-08.md, "Second pass") found the
biggest remaining cost was not a Rustuna technique at all: every NumSharp NDArray has a finalizer, and the hot sampler
paths created arrays per suggestion and dropped them. Each one survived its first collection on the finalization
queue and was promoted, so almost every gen-0 collection became a gen-1 (random 10k: gen-0/1 154/153; NSGA-II 10k:
87/86). GC pauses were 47 % of a 40-parameter random search and 41 % of NSGA-II.

What changed (no result changes; every draw is the same call on the same generator, in the same order):

- New Internal/NumPyDraws: Rand(), Rand(n), RandomSample(n) and Uniform(low, high) read the generator's bit generator
  directly (BitGenerator.NextDouble under BitGenerator.@lock, exactly what NumSharp's scalar and fill paths do: one
  next_double per value, MT19937's 27 + 26 bits of two consecutive words), so they create no array at all.
  Uniform keeps NumPy's order: range = high - low, the finiteness check before any draw (OverflowError text "Range
  exceeds valid bounds"), then low + range * U. RandInt (masked rejection) and Normal (cached polar Gaussian) still call
  NumSharp and dispose the 0-d/vector result.
- SearchSpaceTransform: TransformValues (the managed array Transform wrapped) and UntransformValues(ReadOnlySpan<double>)
  (the untransform without a NumSharp copy). Untransform(NDArray) now disposes its astype(float64) copy and delegates.
  The untransform result dictionary is sized up front.
- RandomSampler.SampleIndependent draws the transform's bound rows one at a time through NumPyDraws.Uniform instead of
  rng.uniform(bounds[:, 0], bounds[:, 1]). Bit-identical: NumPy computes the same low + range * U per element from the
  same uniforms; the array form's only other difference (raising before any draw when a range is infinite) cannot
  arise, since a numerical parameter has one row and every categorical row is [0, 1]. It used to build five arrays per
  suggestion (bounds, two column views, the draws, the untransform copy).
- NSGA-II family: NSGAIIChildGenerationStrategy (crossover decision, copy-parent pick, one decision per gene),
  NsgaiiCrossover (tournament randints, categorical uniform crossover, parents transformed straight from their params
  with TransformValues, child untransformed with UntransformValues, no population copy before the first pick), every
  crossover (Uniform, BLX-alpha, SBX/VSBX and their shared tail, SPX, UNDX) and PolynomialMutation go through
  NumPyDraws; result dictionaries are sized up front; NSGAIISampler.InferRelativeSearchSpace sizes its result.
- CMA-ES: the per-ask seed draw, the told solutions' transform and the candidate's untransform use the helpers; the
  dimension comes from RawBounds instead of building the bounds array for its shape. QMC untransforms its point
  without a NumSharp copy. TPE's candidate sampling (mixture choice, categorical quantiles, truncated-normal
  quantiles) uses the helpers.

Measured (proof harness of plan 13 built against the b3baef9 snapshot and this tree, 40 floats in [-10, 10], seed 0,
warm rep, interleaved runs, machine at 25-48 % load from other sessions):

  random 10,000:  3.93-3.95 s -> 0.79-0.80 s (5.0x); allocated 2,305 -> 889 MB; gen-0/1/2 154/153/27 -> 56/16/7;
                  GC pauses 1.67-1.69 -> 0.17-0.18 s; hash 5DC310069B28C5D7 both
  NSGA-II 10,000: 1.80-1.81 s -> 0.95-0.96 s (1.9x; Rustuna 0.2.1: 0.82 s); allocated 1,234 -> 962 MB;
                  gen-0/1/2 87/86/20 -> 59/16/7; pauses 0.67 -> 0.20 s; hash 0B1FA7FBED2E9676 both
  CMA-ES 10,000 2.33 -> 2.20 s, TPE 1,000 3.80 -> 3.78 s, QMC 10,000 1.15 -> 1.21 s: unchanged within noise, hashes equal
  (their time is elsewhere; later plan 14 steps).

Parity coverage (each affected area, byte for byte):
- New unit NumPyDrawsTests: every helper against the NumSharp call it replaces, value bits and next stream state, on
  generators seeded alike (scalar/vector uniforms, TPE's sizes, every bound shape RandomSampler meets, row-by-row vs
  NumSharp's array uniform over a bounds column, non-finite ranges, randint seed ranges, Gaussians with the cached
  polar value, and NSGA-II's mixed draw sequence).
- SearchSpaceTransformTests: TransformValues equals Transform and UntransformValues equals Untransform bit for bit
  (mixed space, with and without the unit cube, clamped and tied coordinates); the NDArray overload still reads
  float32/int64 input through a float64 copy and leaves the input usable.
- RandomSamplerTests: 60 trials x 3 seeds over every distribution kind replay the pre-plan-14 array draw and match.
- New live parity RandomSamplerParityTests.LongRun_AllDistributionKinds_IsBitIdentical_ToLiveOptuna: 300 trials x 2
  seeds against live Optuna 5.0, one-hot categoricals with two and five mixed-type choices, log int to 2^40, stepped
  negative int, a float spanning +-1e300, every value compared through Python's repr.
- Live parity suites rerun, all green against Optuna 5.0 (142/142, Slow long runs included): RandomSampler, NSGA-II
  (all crossovers, polynomial mutation, constraints, 2,000-trial 40-parameter run), NSGA-III, CMA-ES (+ mixed fleet),
  QMC, TPE (+ 1,000-trial and MOTPE 300-trial runs), compact storage, battle-tested usage, edge cases, partial fixed,
  brute force.
- Hash probe (scratchpad p14/hashprobe, 35 seeded studies over every sampler family the change reaches directly or
  through RandomSampler and SearchSpaceTransform: random, TPE independent/multivariate/group/pruned/constrained, MOTPE,
  NSGA-II with every crossover, polynomial mutation and constraints, NSGA-III, SMS-EMOA, AGE-MOEA-2, MOEA/D, RVEA,
  CMA-ES plain/with margin/separable/IPOP, QMC Sobol/Halton, DE, PSO, SNES, xNES, OpenAI-ES, LM-CMA, VkD-CMA, JANUS,
  brute force): every hash equal before and after.
- Unit suite, CI filter: net8.0 2136/2136; net10.0 2135/2136 with JournalConcurrencyTests.Lock_WithoutAGracePeriod_
  WaitsForALeftBehindLockUntilItIsRemoved failing once under load and passing on 3 reruns (timing-sensitive journal lock
  test, untouched code). The OpenBugs subset of the touched areas fails the same 8 tests with the same messages before
  and after.
- Documentation gate: check_documentation.cs 0 issues; Core builds with the doc warnings as errors (--no-incremental).
… exp call where it is exactly zero (plan 14, step B); TPE 1.5x, MOTPE 1.2x, same trials

Stopwatch timing of TPE's stages on 1,000 trials over 40 floats (the profiler's sampled thread time had hidden it:
EventPipe suspensions inflate some frames) showed where a run went before this step: scoring the candidates 1.74 s
(40 %), building the Parzen estimators 1.34 s (31 %; the bandwidth argsort alone 1.05 s, 24 %), converting the
history to internal values 0.60 s (14 %). Three changes, each keeping every bit:

1. Kernel widths in O(n) when that is provably exact (ParzenEstimator.BinnedRawSigmas; Rustuna's binned_sigmas idea,
   rustuna_core/src/parzen_estimator/model.rs). A width is the larger distance to the two sorted neighbours, clipped up
   to minsigma, so a neighbour nearer than minsigma cannot change it. Binned at that width, a centre that is neither the
   smallest nor the largest of its bin clips to minsigma, and only each bin's extremes can have a farther neighbour (the
   adjacent non-empty bin's extreme, or the domain end). Rustuna trusts the binning and can differ in the last bits where
   rounding puts a centre in the neighbouring bin; this version proves its result instead: bin assignment
   floor((v - low) / minsigma) is monotone, so bins are contiguous runs of the sorted order and equal centres share a bin;
   a bin's smallest is taken at its first index and its largest at its last (the stable sort's positions); distances are
   the sort's subtractions of the same operands; and every bin is checked to span at most minsigma (computed), which by
   the monotonicity of IEEE subtraction bounds every in-bin neighbour distance, so dropping those changes no clipped
   width. Anything else (fewer than two centres, a centre outside the domain or NaN, a degenerate domain, magic clip
   off, a wider bin) falls back to the sort, now SortedRawSigmas. ComputeSigmas applies the same clip to either.
2. TpeObservationCache (Rustuna ddd9c56 / e7a22d7, "reuse completed observations across trials"): each finished trial's
   internal value per (parameter name, search-space distribution), by trial number, or the fact that it lacks the name.
   GetInternalRepr copies a known row from dense columns instead of chasing each trial's parameter dictionary; an
   unknown row is converted exactly as before (the subset test first, conversions in search-space order, so the same
   exceptions) and remembered. Unfinished trials (constant liar's running ones) are never cached. Columns are keyed by
   distribution value equality (equal distributions convert every value alike); a call for another study or storage
   starts over; past 256 columns (dynamic ranges) it starts over. Each instance (per-worker copies too) owns its cache
   under its own monitor. The non-cached path also reads each value with one TryGetValue instead of ContainsKey plus the
   indexer.
3. The log-sum-exp stores 0.0 for every term below MixtureOfProductDistribution.ExpIsZeroBelow (-746) instead of calling
   Math.Exp: exp there is exactly +0.0 (exp(-746) is 0.42 of half the smallest subnormal; NumPy's, ucrt's, glibc's and
   Apple's exp all return zero below the underflow point, about -745.1332), and the zero still enters the pairwise sum,
   so the summation tree is unchanged. Measured on this machine: Math.Exp costs 5 ns for normal results, 16 ns on its
   underflow-to-zero path and 63 ns where the result is subnormal (that band stays computed).

Measured (plan 13 proof harness built against the b3baef9 snapshot and this tree, 40 floats in [-10, 10], seed 0,
interleaved runs; hashes equal before and after):

  TPE 1,000 (warm rep):  3.81-3.95 s -> 2.51-2.55 s (1.5x); allocated 1,571 -> 1,028 MB; hash 77CB9F115180830A
  MOTPE 1,000 (rep 0):   12.67 s -> 10.68 s (1.19x); allocated 3,418 -> 2,832 MB; hash A14E3A50093C4DF4
  MOTPE 300 (rep 0):     2.31 s -> 1.83 s before change 3
  Stage timing after changes 1-2 (rep 0, scratch instrumented copy): internal values 0.60 -> 0.15 s, estimator
  construction 1.34 -> 0.54 s; scoring 1.75 s unchanged then (change 3 came after). MOTPE's scoring is dominated by exp
  calls whose results are subnormal (63 ns each), which an exact port cannot skip.

Parity coverage (byte for byte where OptunaSharp is compared with itself; TPE vs live Optuna is trajectory parity within
the documented erf tolerance, as before):
- New TpeBinnedSigmasTests: binned vs sorted widths bit for bit over uniform, tied (grids), clustered, on-edge,
  one-ulp-off-edge, domain-end and floor-spaced centres, 14 sizes from 2 to 1,500, ten domains (log, step-widened,
  tiny, huge, far from zero), both endpoint settings, three seeds; asserts the fast path answers on more than 90 % of
  uniform inputs so the equality checks are not vacuous; fallback cases; ComputeSigmas end to end with and without the
  magic clip. Scratch fuzz: 800,000 more generated cases, about 769,000 answered by the binned path, 0 mismatches.
- New TpeObservationCacheTests: warm cache vs the cache emptied before every suggestion, identical trials: dynamic
  ranges + conditional parameter + pruned trials in both modes on plain and compact storage, MOTPE, constant liar with
  running trials (ask three, tell three) in both modes, an ever-changing range (bounded at MaxColumns), and a sampler
  reused for a second study.
- New TpeExpUnderflowTests: Math.Exp returns +0.0 bits for the 100,000 doubles below the bound and 200,000 inputs down
  to -1e6 (and -inf, double.MinValue) on every platform CI runs; the bound sits below the underflow point.
- New live parity in TpeSamplerParityTests: DynamicRangesConditionalAndPrunedTrials (both modes, 4 cases, 80 trials)
  and ConstantLiarWithRunningTrials_AskTell (both modes, 3 cases, 15 rounds of 3 running trials). The whole TPE live
  suite passes, 103/103 with the importance, compact-storage, docs and recipe parity cases, Slow long runs included
  (TPE 1,000 trials x 40 parameters, MOTPE 300); importance parity (PED-ANOVA shares the estimator) 40/40.
- Hash probe (35 seeded studies across every sampler family): 0 differences before/after.
- Unit suite, CI filter: 2154/2154 on net8.0 and net10.0. Documentation gate: 0 issues; Core builds with the doc
  warnings as errors.
…step C); QMC 2.4x, same trials

QMCSampler finds its sample id in a study system attribute named "qmc:" + sha256(json.dumps(qmc_vars)), where qmc_vars
holds the engine kind, the search space as str(distribution) by sorted name, and the scramble flag and seed (Optuna's
_find_sample_id). OptunaSharp rebuilt that key on every trial: 40 distribution reprs (each an AsDict, a sort and a
StringBuilder), a JSON document and a SHA-256. An allocation trace of 5,000 40-parameter trials put most of QMC's
1.58 GB per 10,000 trials there.

The key is a pure function of the search space and the sampler's fixed settings (qmc type, scramble, seed are all
readonly). QmcIdKey now keeps it for the last search space and reuses it while the next call passes the same names in
the same order with the very same distribution objects, which a study's relative search space does from its first
trial on (QMC builds it from the first trial's distributions). Reference identity, not value equality, decides: two
categoricals can be equal by Python's rules (True == 1) and still print differently, so only the same objects are
known to print the same; anything else recomputes. The sample-id read and write still happen on every call. The memo
runs under the sampler's lock with the rest of sampling, and per-worker instances start without one.
Also sized up front: the relative search space and the untransformed result dictionaries.

Measured (plan 13 proof harness against the b3baef9 snapshot, 40 floats, seed 0, warm rep, interleaved):
  QMC 10,000: 1.24-1.30 s -> 0.52-0.53 s (2.4x); allocated 1,584 -> 472 MB; gen-0/1/2 98/22/10 -> 30/12/4;
  hash 1ADA76D940EDB34F both.

Parity coverage:
- New live gate QmcSamplerParityTests.LongRun_40Parameters_300Trials_FullState_IsBitIdentical: Sobol and Halton,
  scrambled and not, 37 floats + a log float + an int + a categorical over 300 trials; the whole study state, the
  qmc:<sha256> counter in the study system attrs included, bit-identical to live Optuna 5.0. The QMC live suite passes,
  16/16; with battle-tested usage and the recipes 47/47 before the new gate.
- New unit test QmcSamplerTests.QmcIdKey_Memoized_EqualsAFreshComputation: a memoized key equals a fresh sampler's for
  the same objects, for equal-by-value categoricals that print differently (True/False vs 1/0, which must and do give a
  different key), for a reordered space, and back again; scrambled and not.
- QMC unit tests 31/31.
… step D; Rustuna 82b13d8); 160 MB -> 0.1 MB on a 4.3M-line journal, same records

JournalFileBackend kept the byte offset of every log line it had read (Python's _log_number_offset dict), so a sync
can seek to the next unread line. A 100,000-trial, 40-parameter study writes about 4.3 million lines: the dictionary
alone held 160 MB for the life of the storage.

Now it keeps what a read actually needs (Rustuna's scheme): the furthest line start any read has established (a sync
asks for exactly the next unread line, so it still seeks straight there) and one checkpoint every
OffsetCheckpointInterval = 4,096 lines. A read for an earlier line (another storage over a shared backend that is
behind, a replay from the start) seeks to the nearest checkpoint at or before it and skips forward at most 4,095
lines, exactly as Python skips the lines before log_number_from when it reads from byte 0. The records each read
yields, their numbering and the size snapshot are unchanged:
- A line's successor start is recorded only after the line decodes (Python records it first and deletes it on a
  decode error), so an undecodable record never becomes a place a later read starts after; lines before the requested
  range are skipped undecoded and recorded, as Python records them.
- Readers sharing a backend record the same offsets (the file only grows): the furthest start only moves forward, a
  checkpoint is appended by whichever reader passes it first, and a reader can only start past a checkpoint some read
  passed, so checkpoints stay in order. All bookkeeping is under the existing _offsetLock.

Measured (journalmem probe, scratchpad p14/journalmem, built against the b3baef9 snapshot and this tree; a synthetic
4.3M-line, 1.09 GB journal of set_trial_param-shaped records):
  memory the backend retains after a full read: 160.3 MB -> 0.1 MB
  full read: 5.48-7.36 s -> 5.22-5.26 s; next sync (seek to the end): 0.28-0.35 ms -> 0.34-0.36 ms

Parity coverage:
- New unit JournalOffsetCheckpointTests: reads from 13 positions (both sides of each checkpoint, the end and past it)
  equal the matching slice of a full read on a cold and on a warm backend, and the warm backend kept 3 checkpoints for
  8,692 lines; a storage-like reader that stopped half way resumes exactly after the file grows and another reader
  reaches the end; 12 concurrent readers from scattered positions on one backend all read exactly; an undecodable
  record followed by another fails every read that reaches it (from before it, from it) instead of being skipped; a
  partial last line is left for the next read.
- New live gate JournalStateParityTests.LargeJournal_PythonWrites_CSharpReplaysAndContinues_SharedBackendCatchesUp_
  FullState: Python writes 250 trials x 40 parameters (more than two checkpoint intervals); a C# storage replays
  Python's full state; two C# storages share one backend while one writes 30 more trials, so the other's next sync
  starts behind the furthest offset; the lagging storage, a cold C# storage and Python's fresh replay all give the
  identical full state (floats and datetimes as instants).
- Journal suites: unit 95/95; live 30/30 (5 Redis cases skip without a Redis server; the Redis backend is untouched).
…an 14, step E; Rustuna d46ace5); 8,000 trials with a stale running trial 13.0 -> 9.6 s, same rows

CachedStorage refreshes a study by asking the RDB storage for trial_id > watermark (the last finished trial) plus the
unfinished trials it tracks at or below it: Optuna's query is `... WHERE study_id = ? AND (trial_id > ? OR trial_id IN
(...)) ORDER BY trial_id`. That second part is non-empty only while a trial below the watermark is still RUNNING: a
parallel worker's, or one a crashed worker left behind (then on every refresh for the rest of the study). SQLite answers
the OR by walking every trial of the study in ix_trials_study_id: 2.6 ms per refresh at 20,000 trials (plan 14
research, scratch p14/sqlplan), so such a study's per-trial cost grew with its whole history.

The SQLite RDBStorage now sends the two halves joined by UNION ALL, each indexable through the primary key (64 us there):
`<select> AND trial_id > ? UNION ALL <same select> AND trial_id IN (...) ORDER BY trial_id`. The halves are disjoint
(every included id is at or below the watermark, filtered as before), the final ORDER BY sorts the union, and both
halves repeat the same named parameters, which SQLite binds to the same values: the rows and their order are the OR's.
Rustuna also adds a (study_id, number) index; that changes Optuna's schema, so it is not taken. The EF Core storage
(PostgreSQL, MySQL, whose planners combine ORed index ranges) keeps Optuna's form.

Measured (sqlgrowth probe, scratchpad p14/sqlgrowth2, built against the b3baef9 snapshot and this tree: a seeded
10-parameter random search on a SQLite file in WAL mode with synchronous off, through CachedStorage, with one trial asked
and left RUNNING before the run):
  8,000 trials: 13.0 s -> 9.6 s; per trial by 2,000-trial chunk 1.77/1.38/1.55/1.78 ms -> 1.30/1.01/1.18/1.30 ms

Parity coverage:
- New live gate SqliteRdbInteropParityTests.RunningTrialBelowTheWatermark_CachedRefresh_MatchesPythonsRead: Python
  creates the study on a SQLite file, leaves its first trial running and completes 15; C# loads it through CachedStorage
  and optimizes 25 (every refresh takes the new query with the running trial included); Python then suggests a
  parameter, reports an intermediate value and tells the running trial; C#'s cached view equals Python's fresh read,
  field for field (41 trials, the formerly running one complete).
- New unit SqliteRdbStorageTests.CachedRefresh_WithRunningTrialsBelowTheWatermark_EqualsAFreshRead: two storages on one
  file; running trials below the watermark, completions around them, a running trial changed, then finished out of
  order (one completed, one failed); after every step the cached view equals a fresh storage's read (numbers, states,
  params, values, intermediate values, in order).
- SQLite suites: unit 52/52; live 21/21 (SQLite RDB, storage recipes, storage URLs). The schema-upgrade live cases
  (SqliteRdbUpgradeInteropParityTests) locate the repository root from the test binaries and so cannot run from the
  out-of-tree build used here; they exercise the migration chain, which this change does not touch.
- Documentation gate: check_documentation.cs 0 issues; Storage.Sqlite builds with the doc warnings as errors.
…ice (plan 14, step F; Rustuna 198e484); categorical-heavy TPE 2.5-2.7 -> 1.8 s, same trials

MixtureOfProductDistribution.LogPdf added, for every categorical dimension, log(weights[k, x]) for each candidate and
kernel: Optuna's `np.log(np.take_along_axis(weights[None], xi, axis=-1))[..., 0]`, one Math.Log per (candidate, kernel).
TPE scores 24 candidates against every kernel of both estimators, so a three-choice parameter over a thousand kernels
took 24,000 logs per estimator, of which only 3,000 are distinct. Rustuna's 198e484 also cuts these log calls (with its
own numerics, which OptunaSharp does not take).

The categorical branch now takes the log of a (kernel, choice) column the first time some candidate draws that choice
and reuses it for every other candidate with the same choice: one log per kernel and drawn choice. Math.Log is
deterministic, so every added term has the bits it had; the terms are added to the same buffer elements in the same
order (dimensions in order, then the log mixture weights, then the log-sum-exp), so every sum keeps its bits too. Out of
range or negative choices still throw IndexOutOfRangeException (from the column array instead of the weight matrix).

Measured (hash probe, scratchpad p14/hashprobe, built against the b3baef9 snapshot and this tree, interleaved):
  tpe-categorical-heavy (500 trials, 20 five-choice categoricals + 20 floats): 2.5-2.7 s -> 1.8 s, hash E015C07493B6830A
  on both sides.

Parity coverage:
- New unit TpeExactKernelTests.CategoricalLogPdf_OneLogPerKernelAndChoice_MatchesThePerCandidateLogsToTheBit (seeds 1
  and 2): mixtures of 1, 2 and 4 categorical dimensions with 2-7 choices, weights drawn with some exact zeros (so some
  terms are -inf), 1 to 40 kernels, 24 candidates each; LogPdf equals the per-candidate reference (the code this
  replaces, written out in the test) to the bit for every candidate.
- Hash probe: all 36 seeded scenarios (TPE independent / multivariate / group / pruned / constrained / MOTPE /
  categorical-heavy, NSGA-II with every crossover, NSGA-III, SMS-EMOA, AGE-MOEA-2, MOEA/D, RVEA, CMA-ES variants, QMC,
  DE, PSO, NES, LM-CMA, VkD-CMA, JANUS, BruteForce) hash the same before and after.
- Unit: TPE + importance filter 103/103 (net10.0); TpeExactKernelTests 15/15.
- Live Optuna (pythonnet, Python 3.12 + vendored optuna 5.0.0): TpeSamplerParityTests (non-Slow) + ImportanceParityTests
  70/70.
- Documentation gate: check_documentation.cs 0 issues; Core builds with the doc warnings as errors.
…d final proof; README and performance page numbers rerun after it

New docs/plans/14-rustuna-second-pass.md, the record of the six plan 14 steps on main (A cfe78ec NumSharp
temporaries, B 3050d81 TPE widths / observation cache / exp skip, C fe68085 QMC sample-id key, D 7537079 journal
sparse offsets, E 39fdd4b SQLite UNION ALL refresh, F 82b8a0b TPE categorical logs): the owner's request ("1. Each
affected area must have byte-for-byte parity test coverage 2. Improve performance - Implement."), the inventory and
verdicts (what was built, what was not taken because it changes bits or Optuna's schema, what stays open), what each
step does, the byte-for-byte gates per affected area (unit pins against the replaced code, the 36-study hash probe,
live Optuna 5.0), the final before/after proof, findings and open items (TPE ordered-array scoring and estimator
copies, NSGA-II per-trial precompute, CMA-ES in-memory restore, scrambled Sobol's O(n^2) point generation, NumSharp
temporaries in samplers that were not traced).

Final proof (plan 13 proof program built against the b3baef9 snapshot and 82b8a0b, 40 floats in [-10, 10], seed 0,
before/after back to back, 2026-11-09 14:23-14:27, ~30 % load from other sessions; every pair hash-identical):
  random 10k     rep 0 4.31-4.46 -> 0.81-0.82 s, rep 1 3.52-3.61 -> 0.50-0.52 s (6.8-7.2x); gen-0/1/2 130/129/2 -> 37/9/1
  QMC 10k        rep 1 0.99-1.01 -> 0.43-0.45 s (2.2-2.3x); allocated 1,584 -> 472 MB
  NSGA-II 10k    rep 1 1.51-1.57 -> 0.75 s (2.0-2.1x); 100k rep 0 17.19 -> 8.60 s (Rustuna 0.2.1: 19.4 s)
  TPE 1k         rep 1 3.86-3.91 -> 2.47 s (1.6x); compact storage 3.39 -> 2.43 s
  MOTPE 1k       12.70-12.82 -> 10.44-10.52 s (1.2x)
  CMA-ES 10k/100k unchanged within noise
  TPE, 20 categoricals + 20 floats, 500 trials: 2.5-2.7 -> 1.8 s
  full hash probe, 36 seeded studies over every sampler family: identical before and after

README: the sampler table's OptunaSharp column rerun (benchmark 14's C# probes built against b3baef9 and 82b8a0b,
three fresh processes each, interleaved; medians): TPE 6-D 1.12 -> 0.61 s, TPE multivariate 1.14 -> 0.66 s, TPE 100-D
2.1 -> 1.6 s, CMA-ES 6-D 0.91 -> 0.74 s, CMA-ES 200-D 1.2 -> 1.0 s, NSGA-II 0.60 -> 0.31 s, QMC 0.39 -> 0.13 s, random
6-D 0.98 -> 0.15 s, random 1000-D 3.2 -> 0.6 s, random 50k blocks 0.6-0.8 -> 0.08-0.25 s; GP moved less than the
noise and keeps 3.47 s. (The README's old numbers predate plan 14 and some were measured under heavier load; the plan
document has the interleaved before/after medians.) The large-studies table takes the final proof's rep 0 times: TPE
3.2 s, MOTPE 10.5 s, CMA-ES 10k 3.6 s and 100k 22.1 s, NSGA-II 10k 1.4 s and 100k 8.6 s.

docs/website-src/docs/performance.md: a "Second comparison" section with the plan 14 changes, their effect and the
before/after table. docs/THREADING.md: the two new sampler caches (TpeObservationCache, QMC's sample-id key) are sampler
state covered by the sampler monitor, one per PerWorker instance, like plan 13's. docs/plans/README.md: plan 14 row.

Final gates on 82b8a0b (out-of-tree Release build):
- Unit suite, CI filter: net8.0 2163 passed, 1 skipped; net10.0 2163 passed, 1 skipped (the skip is
  StopSignalsTests.ARealSigterm_CancelsTheToken_AndTheProcessLivesOn, which self-skips on Windows).
- Live Optuna 5.0 (engine required), every area plan 14 touched, non-Slow: RandomSampler, NSGA-II, NSGA-III, CMA-ES
  and the CMA-ES mixed fleet, QMC, TPE, importance, compact storage, battle-tested usage, edge cases, partial fixed,
  brute force, journal state / interop / mixed fleet / codec, SQLite RDB: 228 passed, 1 skipped
  (JournalMixedFleetInteropTests.PythonAndCSharpWorkers_OnOneRedisJournal_ProduceIdenticalPerfectStudies needs a Redis
  server; the Redis backends are untouched).
- Storage suite (SQLite RDB storage and options, journal backend contract): 95/95.
…ecreated under a reused id is not read from the deleted one (plan 14 concurrency review); new concurrency tests for plan 14's state

The owner asked whether plan 14 broke concurrency. The review (docs/plans/14-rustuna-second-pass.md section 8) went
through everything plan 14 added that more than one thread can reach:
- TpeObservationCache: reached only from GetInternalRepr <- BuildParzenEstimator <- Sample <- SampleRelative /
  SampleIndependent, all under the sampler monitor (BaseSampler.SyncRoot), which every study-driven call holds, also when
  several studies share the sampler. A PerWorker instance gets its own cache (the field initializer runs for the private
  copy constructor, which does not copy it).
- QMC's sample-id key memo: reached only from SampleRelative, under the monitor; QMC has no worker instances.
- JournalFileBackend's offsets: every access under _offsetLock, each read carries its own position, line starts are
  recorded after decode only, so the furthest start advances one line at a time and checkpoints are appended once, in
  order.
- NumPyDraws: no state; each draw or fill holds BitGenerator.@lock, the lock NumSharp's own primitives take
  (BoundedIntegers, Generator) and ask outside callers to hold.
- The rest (exact widths, exp skip, categorical logs, transform values, the SQLite query text) has no shared state.
No threading defect was found. New tests pin it (below), and every concurrency-focused suite was run three more times.

What the review did find is a correctness regression that is not about threads. TpeObservationCache started over only
when the storage object or the study id changed. SQLite reuses the highest deleted study and trial ids, and a gRPC proxy
or a custom storage passes them through as one storage object (PerStudyState's documentation describes exactly this,
pre-release review item H3), so a reused TPESampler on a study deleted and recreated under the same id read the deleted
study's internal values for every trial number it had cached. Optuna keeps no such cache and TPE before plan 14
re-read every trial, so results diverged from Optuna there. Reproduced end to end before the fix: the new storage-suite
test ran the same scenario twice over a gRPC proxy to SQLite, once with the first study deleted (its id reused) and
once without (a new id), and the reused sampler suggested different parameters from the first TPE trial on.

The fix applies PerStudyState's TrialAnchor rule per row: TpeObservationCache keeps each cached row's trial identity
(storage id and start time, by trial number) and ColumnsFor now takes the trials the caller is about to read and checks
every one with a cached row before anything is read; one mismatch (another storage id, or the same id started later,
since SQLite reuses trial ids too) and the cache starts over (IdentityRestarts counts it). An unknown start time on
either side is not a difference. GetInternalRepr passes its trials and records the identity wherever a finished row is
stored (present or absent). The cache also holds its storage through a WeakReference now, as PerStudyState does, so a
long-lived sampler no longer keeps the last storage it served alive.

Measured (plan 13 proof program, the 82b8a0b build against this tree, alternating): TPE 1k warm 2.68-2.90 s -> 2.58-3.04 s,
MOTPE 1k 10.80-11.01 s -> 11.01-11.20 s, compact TPE 2.61 -> 2.56 s, allocations equal: unchanged within noise. Hash
probe, 36 seeded studies over every sampler family: identical to 82b8a0b (and so to b3baef9).

New tests:
- Storage suite: GrpcStorageProxyTests.Tpe_ReusedOnARecreatedStudyUnderTheSameId_SamplesAsUnderANewId (failed before the
  fix, passes after; independent TPE, because Optuna's multivariate TPE refuses, as OptunaSharp's does, a sampler reused
  for a study under another id, which the reference run needs).
- TpeObservationCacheTests: LiveStudy_NeverStartsOverOnATrialsIdentity (both modes, plain and compact, constant liar with
  running trials: no false alarm, the cache stays in use); TrialNumbersCarryingOtherTrials_StartTheCacheOver (later start
  under the same ids, other ids, an unknown start); Cache_DoesNotKeepTheStorageAlive;
  WorkerInstances_OwnTheirCaches_AndThePrototypesStaysUnused (PerWorker, nJobs 8: worker caches distinct, the prototype's
  unused); SharedByEightWorkers_EveryCachedRowIsItsTrialsOwnValue (shared sampler, nJobs 8, constant liar, pruned and
  conditional parameters: every cached row equals its trial's conversion to the bit, no identity restart).
- JournalOffsetCheckpointTests.ReadersSyncingWhileOneWriterAppends_AcrossCheckpoints_AllReadExactly: one backend, one
  writer appending past three checkpoint intervals in batches, six readers syncing like storages from their own
  positions; each ends with exactly its slice of a full read, and the backend holds one checkpoint per interval.
- Mutation checks: with RowsMatch forced true, TrialNumbersCarryingOtherTrials_StartTheCacheOver fails; with the storage
  held strongly, Cache_DoesNotKeepTheStorageAlive fails.

Docs: plan 14 section 8 (the review and this finding); THREADING.md's cache paragraph now says how the TPE cache is
keyed (study and storage plus each row's trial) and that it holds the storage weakly.

Gates on this tree (out-of-tree Release build):
- Documentation gate: check_documentation.cs 0 issues (5,788 declarations); Core builds with the doc warnings as errors.
- Unit suite, CI filter: net8.0 2172 passed, 1 skipped; net10.0 2172 passed, 1 skipped (the Windows self-skip of
  StopSignalsTests.ARealSigterm_CancelsTheToken_AndTheProcessLivesOn).
- Concurrency-focused classes (OptunaSharp.Tests.Concurrency, OptimizeConcurrencyTests, SamplerNJobsSweepTests,
  JournalConcurrencyTests, JournalOffsetCheckpointTests, TpeObservationCacheTests, QmcSamplerTests), three more runs in
  a row on net10.0: 342/342 each time.
- Live Optuna 5.0 (engine required): TPE, compact storage, importance and battle-tested parity, 83/83.
- Storage suite: GrpcStorageProxyTests, SqliteRdbStorageTests, SqliteRdbOptionsTests, JournalBackendContractTests,
  114/114.
…d-ask child redraw, held running records (QA-56..58)

Second 8-hour parity QA run against Optuna 5.0.0 (journal docs/qa/PARITY_QA_2026-11-10.md, started 21:38 JST).
This commit holds the journal so far and the first three [OpenBugs] reproductions; nothing in src/ changes.

Method so far
- A/B oracle for the performance commits of master..main (plans 13 and 14): one probe, a JSON random-program
  interpreter (every parameter kind, conditional parameters, dynamic ranges, enqueued trials, constraints through
  set_constraint, pruners with reports, planned fail/prune/NaN/inf outcomes, batches told FIFO or LIFO, an
  ask-the-whole-batch-first mode), built against main and against de495f8 (the commit before the first plan 13
  change, already on NumSharp 0.80.0). 6,400 programs over Random, TPE, CMA-ES, NSGA-II/III and QMC on in-memory,
  compact, journal and SQLite storage: every output line identical. The "same trials" claims hold on these paths.
- A Python twin of the interpreter compares main with Optuna 5.0.0 on the same programs. Divergences are shrunk
  (greedy program shrinker), then traced: MT19937 states compared at every ask, Optuna instrumented by
  monkeypatching, the baseline copy of the port instrumented where needed.

Findings
- QA-56 (LOW) NSGA-III niche preservation: Optuna sorts the borderline front's (distance, index) pairs with
  list.sort(reverse=True); pairs holding a NaN distance are incomparable, so timsort keeps them in input order and
  pop() takes the last one first. The port's comparer orders NaN as equal and then by index, which reverses them.
  NaN distances arise whenever the front plus the elites has no finite value in one objective (_filter_inf).
  Population 2, a first generation of (inf, inf) trials, seed 0: Optuna's third trial has y=0.7151893663724195,
  the port's 0.5448831829968969. Found in 4 of ~120 NSGA-III fuzz programs, single- and multi-objective.
- QA-57 (LOW, owner question Q4) the plan 12 running-trial redraw (HandedOutParams.Claim, d0558f5) also fires in a
  single-threaded seeded batched ask/tell, Optuna's documented batch pattern, and compares partial children: a child
  whose mutated parameter was dropped is called a duplicate of another running trial's partial child although each
  trial samples that parameter on its own. NSGA-II and NSGA-III, population 6, seed 311031104, batches of 5: trial
  18's v5 is 0.00021556396515864647 (trial 1's value, from a fourth generated child) where Optuna samples
  0.02621137972316322. Every MT19937 state up to that trial's mutation decisions is identical.
- QA-58 (LOW) a FrozenTrial read with get_trials(deepcopy=False) while its trial runs keeps receiving that trial's
  later user attributes, intermediate values and system attributes in Optuna (in-memory and journal storage): Trial
  mutates the dictionaries of the record it was created from in place, and the storages' shallow copy-on-write
  shares every dictionary a write has not replaced. The port's record never changes.

Tests (all fail today for the reason stated, net10.0):
- test/OptunaSharp.Tests/QA/Run2SamplerParityBugTests.cs: Nsga3_AllInfiniteBorderlineFront_KeepsPythonsNichingOrder,
  Nsga_BatchedAskTell_PartialChildIsNotRedrawn
- test/OptunaSharp.Tests/QA/Run2StorageParityBugTests.cs:
  HeldRunningRecord_SeesTheTrialsLaterAttributeWrites_LikeOptuna
…, QA-60); Wilcoxon, GP, Grid, Hyperband sweeps clean

Parity QA run 2 (journal docs/qa/PARITY_QA_2026-11-10.md), second batch. Nothing in src/ changes.

Findings
- QA-59 (LOW) SQLite RDB storage: the cached refresh sends every unfinished trial below the last finished one as
  `trial_id IN (...)`, one bound variable each. Past SQLite's variable limit (32,766 in Microsoft.Data.Sqlite's
  e_sqlite3) the read throws SqliteException 'too many SQL variables' and the study can no longer be read. Optuna's
  RDBStorage._get_trials catches the OperationalError, logs "Caught an error from sqlalchemy: ... Falling back to a
  slower alternative." and reads every trial; its Anaconda SQLite 3.51.0 even allows 250,000 variables. Reproduced
  with 33,000 RUNNING trials (crashed workers) below a COMPLETE one: Optuna reads 33,003 trials twice, the port's
  second read throws. Found by a new technique: every raise/warn/log message literal of Optuna 5.0 (367) searched by
  its longest fragments in the port's sources; this fallback's warning was one of the two real misses.
- QA-60 (LOW) CmaEsSampler(with_margin=True) on a search space that loses a dimension before the first generation is
  told: Optuna's CMAwM.tell refuses the ragged x_for_tell list (ValueError: setting an array element with a sequence
  ... inhomogeneous shape) and the trial fails; the port's Cma.Tell reads the first n_dim entries of each vector and
  carries on with coordinates that stand for other parameters. Seed 0, x int [0, 10], y [0, 1] ([0, 1.5] on trial 2):
  Optuna raises at trial 3/4/5 for population 2/3/4, the port finishes 12 trials. Found by the new dynamic-range
  program fuzz (11 of 68 CMA-ES programs, all with margin).

Clean this batch (recorded in the journal's progress log)
- WilcoxonPruner after the QA-14 fix: 3,000 random cases (1-150 steps, ties, zero differences, out-of-order and
  missing steps, NaN/inf, both directions, p thresholds, startup steps), decisions and warnings identical.
- GP checked by acquisition value: the port's points are as good under Optuna's own LogEI (the 1e-4 L-BFGS-B
  tolerance class).
- Near-miss message templates (difflib over 349 messages): no wording drift that C# can reach.
- NSGA-II/III under HyperbandPruner (200 programs), Grid with batches (119), BruteForce sequential: identical.

Tests (fail today for the reason stated, net10.0)
- test/OptunaSharp.Tests/QA/Run2SamplerParityBugTests.cs:
  CmaWithMargin_SearchSpaceShrinksInFirstGeneration_RaisesLikeOptuna
- test/OptunaSharp.Tests.Storage/Run2SqliteParityBugTests.cs:
  CachedRefresh_TooManyRunningTrialsBelowTheWatermark_FallsBackLikeOptuna
…y steps (QA-61, QA-62); storage and fuzz-mode sweeps clean

Parity QA run 2 (journal docs/qa/PARITY_QA_2026-11-10.md), third batch. Nothing in src/ changes.

Findings
- QA-61 (LOW) CLI `-f yaml`: PyYAML's safe_dump writes every float with a decimal point and quotes the formatted
  datetimes; the port's YamlDotNet output writes 2.0 as `2`, 1e-07 as `1E-07`, 1e16 as `10000000000000000` and the
  datetimes bare. Read back with yaml.safe_load (YAML 1.1) the port's floats become the int 2, the string '1E-07'
  and the int 10000000000000000, and its datetimes become datetime objects where Optuna's output gives strings.
  OutputFormat.cs calls YAML byte parity best-effort; these change types, not bytes.
- QA-62 (LOW, same root cause as QA-3, owner question Q1) Python parses str(x) into an exact Decimal and computes
  with 28 significant digits; .NET's decimal keeps at most 28 digits after the point, so a value whose repr needs
  more places (a 17-digit double below ~1e-11) is rounded on the way in. After the QA-2/QA-11 fix, 1,439 of 100,000
  random stepped ranges still adjust `high` one ULP off and 184 of 5,000 BruteForce grids differ, every one among
  the inputs needing more than 28 places. FloatDistribution(4.1271208719049127e-14, 4.085121318968367e-13,
  step=1e-15).high: Optuna 4.0827120871904913e-13, the port one ULP below; _enumerate_candidates(2.942974008882182e-14,
  5.614297400888218e-13, 2e-15): 266 values in Optuna, 267 in the port.

Clean this batch
- E13 (the previous run's unfinished SQLite half): 270 random programs written to SQLite and journal storages by both
  languages (Random, CMA-ES, QMC, NSGA-II/III, categorical TPE with interleaved asks): identical.
- New fuzz modes: add_trial warm starts mid-run, Percentile/Threshold/Patient/Wilcoxon pruners, three objectives:
  600 programs identical.
- The journal's directions table now records the status of E1-E18 (the plan 13/14 commit review is complete: no
  regression in 6,400+ A/B programs).

Tests (fail today for the reason stated, net10.0)
- test/OptunaSharp.Tests.Cli/Run2CliParityBugTests.cs: TrialsYaml_FloatsAndDatetimes_KeepTheirTypesLikePyYaml
- test/OptunaSharp.Tests/QA/Run2DistributionParityBugTests.cs:
  SteppedHigh_ValuesBeyond28DecimalPlaces_FollowPythonDecimal
…, MEDIUM); parallel-coordinate fix re-check clean

Parity QA run 2 (journal docs/qa/PARITY_QA_2026-11-10.md), fourth batch. Nothing in src/ changes.

Finding
- QA-63 (MEDIUM) CLI `tell`: `optunasharp tell --study-name s --trial-number 0 --state fail` (or `--state pruned`)
  without `--values` fails. The `--values` option (Option<double[]?>, one-or-more) yields an empty array, not null,
  when it is omitted, so Core's tell sees values next to a PRUNED/FAIL state and raises "ValueError: Values were
  told. Values cannot be specified when state is TrialState.PRUNED or TrialState.FAIL."; the command prints an
  unhandled-exception stack, exits 1 and leaves the trial RUNNING. Optuna (argparse nargs="+", default None) logs
  "Told trial 0 with values None and state TrialState.FAIL." and exits 0. A bare `tell --trial-number n` also
  differs (warning "The number of the values 0 did not match..." and "with values []" instead of Optuna's "The value
  None could not be cast to float." and "with values None"). `--directions` already maps an empty array to null.
  CliParityTests' fixture runs these exact commands but ignores their exit codes, so it kept two RUNNING trials.
  Found by the new CLI differential (random ask/tell sequences with seeded samplers through both CLIs).

Clean
- The QA-26 fix: 600 random studies (None / bool / int / float / str choices including the string "None", missing
  parameters, pruned/failed trials, single and multi-objective) through _get_parallel_coordinate_info in both
  languages: identical.

Test (fails today for the reason stated, net10.0)
- test/OptunaSharp.Tests.Cli/Run2CliParityBugTests.cs: Tell_StateWithoutValues_FinishesTheTrialLikeOptuna
… scenario sweep

Parity QA run 2 (journal docs/qa/PARITY_QA_2026-11-10.md), fifth batch. Nothing in src/ changes.

Finding
- QA-64 (LOW) CLI `ask --sampler-kwargs`: Optuna builds the sampler with getattr(optuna.samplers, name)(**kwargs),
  so JSON objects and arrays reach it; the port's SamplerFactory.TryCoerce binds only scalars. GridSampler's
  search_space, CmaEsSampler's x0 and NSGAIIISampler's reference_points therefore fail with "Could not construct
  sampler 'GridSampler' from the given --sampler-kwargs ({search_space})." where Optuna asks normally.

Also recorded
- 14 fixed CLI scenarios (60 commands) through both CLIs: exit codes and stdout match except QA-63/QA-64 and the
  known QA-37; `tell --state complete` without values is QA-63's root cause too (added to its row).

Test (fails today for the reason stated, net10.0)
- test/OptunaSharp.Tests.Cli/Run2CliParityBugTests.cs: Ask_StructuredSamplerKwargs_ConstructTheSamplerLikeOptuna
Nucs added 10 commits November 12, 2026 19:03
…Optuna's error types and messages: KeyError, StorageInternalError, the full version, Python repr in LightGBM, FileExistsError on a directory, study.direction in MO TPE, n_jobs=0

Group 4 of the owner's "Fix now" list from the 2026-11-10 parity QA run (docs/qa/PARITY_QA_2026-11-10.md):
errors the port raised with another type or another message than Optuna 5.0.0 / optuna-integration 5.0.0.
Each finding's [OpenBugs] repro now passes and carries a "Fixed 2026-11-12" note.

QA-66 - storage-level errors (in-memory, journal, base implementation)
  * BaseStorage.GetTrialParam and InMemoryStorage.GetTrialParam raise KeyError(paramName) for a parameter the
    trial does not have (Python's `trial.distributions[param_name]`), not .NET's KeyNotFoundException; the base
    implementation serves the journal, cached and gRPC-proxy storages, and the compact-record path raises the
    same KeyError.
  * The in-memory and journal storages raise DuplicatedStudyError("") - Optuna's bare `raise
    DuplicatedStudyError` has an empty str(e) - instead of .NET's default text.
  * GridSampler.AfterTrial: an enqueued trial without a grid_id raises KeyError("grid_id"), as Optuna's
    `get_trial_system_attrs(...)["grid_id"]` does.
  * Kept on purpose: the gRPC proxy client still forwards the server's detail where Optuna raises bare
    KeyError() / ValueError() / ... `from` the RpcError. A .NET KeyError cannot carry the cause, and an empty
    message alone would drop the detail Python keeps in __cause__. Recorded in the journal row.

QA-74 - database errors become StorageInternalError (Optuna's _create_scoped_session)
  * EF Core: Mutate now catches DbUpdateException / DbException that is not an integrity error and raises
    StorageInternalError("An exception is raised during the commit. This typically happens due to invalid data
    in the commit, e.g. exceeding max length. ") wrapping the provider's error. Integrity errors propagate
    unchanged (Optuna re-raises IntegrityError, which callers turn into DuplicatedStudyError and friends):
    PostgreSQL SQLSTATE class 23 (psycopg2's mapping), MySQL exactly the seven numbers PyMySQL maps to
    IntegrityError (1048, 1062, 1215, 1216, 1217, 1451, 1452; every other number, a check-constraint failure
    included, is OperationalError there, which Optuna wraps). The translation sits outside the execution
    strategy, so EnableRetryOnFailure still sees and retries transient errors.
  * SQLite: under skip_table_creation the version-info step only reads version_info (Python creates no table
    in that mode), and a database error there becomes StorageInternalError in the constructor - Optuna's
    measured behavior on a database without its tables, with or without skip_compatibility_check. A constraint
    error (another process inserted the row) is ignored, as ignore_integrity_error=True does. Every upgrade-test
    database has version_info, so `storage upgrade` is unaffected (Storage and CLI suites green).
  * StorageInternalError gained a (message, innerException) constructor - the `raise ... from e` analogue.

QA-75 - OptunaSharpVersion.OptunaCompatibilityTarget is "5.0.0", Optuna's __version__, so the RDB
  compatibility error names the full version like Optuna's.

QA-76 - optuna-integration LightGBM messages in Python's forms, independent of the current culture
  * LightGBMPruningCallback's not-found error prints the evaluation list with PythonRepr.Repr per element:
    100.0, 1e-05 and 0.5 under de-DE (was 100, 1E-05, 0,5), quotes switched for a name holding a quote.
  * The tuner's duplicated-aliases warning prints the dict in repr and the chosen value in str
    ({'sub_row': 0.5, 'subsample': 1.0}, bagging_fraction=1.0); the metric type error names the Python type
    (<class 'int'>, <class 'float'>, <class 'NoneType'>, ...).
  * Kept: a cross-validation entry prints four elements; Python's fifth is the standard deviation, which the
    public LightGbmEvaluationResult record does not hold (adding it would change the record's API).

QA-78 - DownloadArtifact's existence check covers directories too (os.path.exists), so a directory at the
  destination raises the IOException (Python FileExistsError) before anything is downloaded.

QA-79 - TpeTrialSplit.GetPrunedTrialScore reads study.Direction, which raises Optuna's RuntimeError ("A single
  direction cannot be retrieved from a multi-objective study. ...") on a multi-objective study whose PRUNED
  trials carry numeric reports, as Optuna's _get_pruned_trial_score does; a NaN report still scores
  (-step, inf) before the direction is read, as in Python.
  * TpeSplitCacheTests asserted the old behavior (its two-objective history had pruned trials with numeric
    reports). The history builder now takes the number of objectives: one-objective histories keep numeric
    reports, two-objective ones report NaN (the only report a multi-objective split accepts), and
    TheCachedSplit_IsSplitTrials_AndTheSameRecordsReuseIt runs on both (6 rows). A new test,
    AMultiObjectivePrunedReport_RaisesLikeOptuna_AndIsNotCached, checks the RuntimeError propagates through
    the cache and that a throwing split is not stored.

QA-81 - OptunaSearchCV maps nJobs: 0 to 1 (optuna-integration's `n_jobs if n_jobs else 1`).

Also updated: CompactFinishedTrialsTests.EveryKindOfValue_ReadsBackIdentical expected KeyNotFoundException for
a missing parameter on both the plain and the compact record; it now expects KeyError with the name as message.

Gates (net10.0, out-of-tree artifacts, CI filter
TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=RequiresPython&TestCategory!=Slow):
  * OptunaSharp.Tests: 2,296 passed, 0 failed (first run: 6 failures, all the two tests above; fixed).
  * OptunaSharp.Tests.Storage: 463 passed, 0 failed; the QA-74 EF Core rows ran for real on MySQL 8.0 and
    PostgreSQL 16 through Testcontainers (OPTUNASHARP_EFCORE_REQUIRED=1); the 144 skips are the Redis suites
    (no Redis server; group 4 does not touch them).
  * OptunaSharp.Tests.Integration.LightGbm: 57 passed. OptunaSharp.Tests.Cli: 90 passed.
  * check_documentation.cs over src: clean; Core, Storage.Sqlite, Storage.EFCore and Integration.LightGbm build
    with the doc warnings as errors.

Journal: QA-66, 74, 75, 76, 78, 79 and 81 marked fixed 2026-11-12, with the two kept sub-items spelled out.
…ders (list.sort with NaN, set iteration), insertion-ordered study maps, SQLite's too-many-variables fallback, attributes in index key order; QA-77 kept on the logger, QA-58 deferred

Group 5 ("Orders + core edge cases") of the owner's "Fix now" list from the 2026-11-10 parity QA run
(docs/qa/PARITY_QA_2026-11-10.md). The owner answered the two open design questions on 2026-11-12: QA-58 "Defer",
QA-77 "Keep the logger".

QA-56 - NSGA-III niche preservation sorts NaN-bearing tuples as CPython does
  * New src/OptunaSharp.Core/Internal/PyListSort.cs: CPython 3.12's list.sort ported step for step - reverse as
    CPython does it (reverse, stable sort, reverse), count_run (3.12: ascending or strictly descending), binary
    insertion up to minrun, the powersort merge policy (powerloop / found_new_run), merge_at with its gallop trims,
    merge_lo / merge_hi with the adaptive min_gallop, merge_force_collapse. With a total order it is a plain stable
    sort; with an inconsistent one (NaN) it reproduces CPython's exact comparison sequence, which is the only way to
    land where CPython lands.
  * NSGAIIIElitePopulationSelectionStrategy.PreserveNicheIndividuals sorts the borderline front with it and the new
    DistanceIndexLessThan (Python's tuple <: equal distances by index, a NaN distance never less, never greater).
    The old comparer treated NaN as equal and reversed an all-NaN front, so pop() took the other pair.
  * CPython 3.13 rewrote count_run, so a 3.13 interpreter may order NaN-bearing lists differently; the port and the
    corpus pin 3.12, the interop suite's interpreter (local 3.12.12, CI 3.12).

QA-68 - GridSampler lists unvisited grids in CPython's set-iteration order
  * New src/OptunaSharp.Core/Internal/PyIntSet.cs: CPython 3.12's set table for non-negative ints (hash(k) == k):
    8-slot minimum, home slot + 9 linear probes then i = (i*5 + 1 + perturb) & mask with perturb >>= 5, growth
    when fill*5 >= mask*3 (to > used*4, used*2 past 50,000), dummies left by discards, the last passed dummy reused
    by an insert (measured on 3.12: set(range(20)) minus {3, 5} plus 131 lands in slot 5), the dummy purge after a
    difference update, set_merge's slot-for-slot copy, and set_difference's two strategies (copy-then-discard when
    len(self) >> 2 > len(other), otherwise a fresh set filled in slot order).
  * GridSampler.GetUnvisitedGridIds builds set(range(n)) - set(visited) - set(running) (and the unfinished
    fallback) with it, so rng.choice indexes the same list as Optuna: list(set(range(16)) - {0, 1, 2, 3, 5, 6, 7, 9,
    10, 12, 13, 14, 15}) is [8, 11, 4]. The old [Misaligned] note assumed only concurrency reaches that branch; a
    single-threaded optimize reaches it whenever the numbering is shifted (add_trial, enqueue_trial, resumed study).

Oracle for both: test/OptunaSharp.Tests/oracle/cpython_order.json (546 KB), recorded from CPython 3.12.12 by
oracle/gen_cpython_order_oracle.py (refuses another minor version): 271 set-difference histories (both strategies,
the purge, sizes to 70,000 incl. both sides of 50,000; bitmaps in, full lists or count/head/tail/SHA-256 out) and
353 (distance, index) sorts up to 1,000 elements (NaN rates 0 to 100 %, ties, infinities, signed zeros, runs and
plateaus; distances as two-character pool indices). Internal/CPythonOrderOracleTests replays it with no Python:
every case matches; plus a total-order stability check against LINQ and argument checks.

QA-67 - InMemoryStorage._studies and JournalStorageReplayResult._studies are insertion-ordered OrderedDict<int, ...>
  (Python dicts), so studies list in creation order across deletions; Dictionary reused a deleted study's slot
  (create a, b, c; delete b; create d; delete a; create e, f: Optuna [c, d, e, f], the port [e, d, c, f]). The
  journal snapshot codecs write and read studies in that order. Both maps are only touched under their locks.

QA-59 - SQLite cached refresh falls back like Optuna's _get_trials
  * The multi-trial read catches the SqliteExceptions Python's sqlite3 raises as OperationalError (primary codes
    ERROR - which "too many SQL variables" is - PERM, ABORT, BUSY, LOCKED, READONLY, INTERRUPT, IOERR, FULL,
    CANTOPEN, PROTOCOL, EMPTY, SCHEMA), logs Optuna's "Caught an error from sqlalchemy: {e}. Falling back to a
    slower alternative." and reads every trial of the base query (study and states), keeping the included ids and
    those above the watermark, in trial-id order. 33,000 RUNNING trials below a COMPLETE one now read back.
  * EF Core needs no fallback, measured with ToQueryString() on EF Core 8 and 9 (throwaway probe, not committed):
    Pomelo inlines the ids (`trial_id IN (3, 5, 8, 13, 21)`), Npgsql passes one array (`= ANY (@__included_2)`).

QA-71 - attributes come back in the order Optuna's index scans return them
  * SQLite: a study's attributes ORDER BY "key" and a trial's ORDER BY trial_id, "key" (the unique (..., key)
    autoindex's BINARY order Optuna's unordered where_study_id / selectinload queries come back in); get_all_studies
    keeps insertion order, as Optuna's table scan does.
  * EF Core: the same on MySQL (ORDER BY key: the column collation's order, InnoDB's unique-index order; new
    AttributesInKeyOrder, passed into the static GetTrialsInternal); PostgreSQL keeps insertion order, where
    Optuna's own order is unstable.
  * docs: the gRPC cross-language page no longer says every OptunaSharp storage keeps insertion order.

QA-77 - kept, [Misaligned] ("Keep the logger"): warnings stay on the optuna logger. Run2WarningParityBugTests'
  two repros became [Misaligned] pins (Warnings_FollowTheLoggersVerbosity_SoErrorHidesThem,
  RepeatedWarning_IsLoggedOnEveryOccurrence: 5 records for 5 trials) and OptunaWarnings documents the difference.

QA-58 - deferred by the owner: snapshots stay (copying Optuna's dictionary aliasing would undo the copy-before-write
  that keeps n_jobs readers safe); the repro stays [OpenBugs] with a "Deferred" note.

Repros un-OpenBugged with "Fixed 2026-11-12" notes: Nsga3_AllInfiniteBorderlineFront_KeepsPythonsNichingOrder,
GridSampler_ShiftedNumbering_PicksUnvisitedGridsInCPythonSetOrder, StudyListing_AfterDeletions_
KeepsCreationOrderLikePythonDicts, CachedRefresh_TooManyRunningTrialsBelowTheWatermark_FallsBackLikeOptuna,
AttributeReads_ComeBackInKeyOrderLikeOptunasIndexScan, MySqlAttributeReads_ComeBackInKeyOrderLikeOptuna.

Gates (out-of-tree artifacts; CI filter
TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=RequiresPython&TestCategory!=Slow):
  * OptunaSharp.Tests: 2,306 passed on net10.0 and on net8.0 (the vendored OrderedDictionary path), 0 failed.
  * OptunaSharp.Tests.Storage (net10.0, OPTUNASHARP_EFCORE_REQUIRED=1): 466 passed, 0 failed (the 25 upgrade
    tests first failed only for the missing repository-root shim in the fresh artifacts folder and passed, 27/27,
    once it was copied); the EF Core rows ran on MySQL 8.0 / PostgreSQL 16 through Testcontainers; 144 Redis skips.
  * OptunaSharp.Tests.Interop GridSamplerParityTests + NsgaiiiSamplerParityTests against live Optuna: 21 passed.
  * check_documentation.cs over src: 5,958 declarations, 0 issues; Core, Storage.Sqlite, Storage.EFCore build with
    the doc warnings as errors.

Journal: QA-56, 59, 67, 68, 71 fixed, QA-77 kept [Misaligned], QA-58 deferred.
… shadow study in lockstep with the C# study; 12 registry samplers and 6 pruners now match native Optuna trial for trial

Group 6 ("OptunaHub bridges") of the owner's "Fix now" list from the 2026-11-10 parity QA run
(docs/qa/PARITY_QA_2026-11-10.md). The sampler bridge rebuilt a Python study per trial from
create_trial(state, values, params, distributions) and never forwarded before_trial / after_trial; the pruner bridge
rebuilt one per prune call. So every system attribute a hub object wrote was lost: PSO never left its random phase
(it builds its swarm in after_trial), MO-CMA failed with KeyError: 'mocma:g', the NSGA-II variants lost their
generations, AutoSampler differed from trial 1, successive halving and Hyperband never pruned (36 and 31 of 40
natively, 0 through the bridge).

New src/OptunaSharp.Interop.pythonnet/OptunaHubShadow.cs - one persistent in-memory Optuna study per OptunaSharp
study (keyed by storage instance + study id), shared design for both bridges:
  * Sync (before every call into Python): every OptunaSharp trial, in number order and in any state, is mirrored with
    its values, params, distributions, user and system attributes and intermediate values; an unfinished mirror
    receives the params, reports and attributes written since, then its new state (WAITING -> RUNNING, or finished, so
    a trial finished without after_trial - failed as stale - finishes too); the study's user and system attributes
    follow. Finished mirrors are skipped. Reads go to the storage afresh, and the values last synchronized are kept
    per key, so what the bridge wrote back is not echoed.
  * The trial objects Python receives are native Optuna's: each running trial keeps its storage record from the
    trial's start (Trial._cached_frozen_trial), updated in place by mirrored params, reports and user attributes as
    Trial._suggest / report / set_user_attr do; LatestTrial is Trial._get_latest_trial() (a shallow copy with lazily
    read system attributes) for sample_independent and prune; StoredTrial is storage.get_trial for after_trial.
  * BeginTrial clears the shadow study's per-thread trial cache (study._thread_local.cached_all_trials) as Study.ask
    does "once every trial". Without it the shadow kept its first read for good: restart CMA-ES diverged from trial
    1, TPE from its first model-based trial, the NSGA-II variants from their first generation boundary (found by the
    new native comparison; only samplers read that cache).
  * The shadow carries the OptunaSharp study's name: HyperbandPruner assigns brackets by
    crc32(f"{study.study_name}_{trial.number}").
  * Call<T> snapshots the study's and the trial's user and system attributes around a Python call and writes every
    key it set to the OptunaSharp storage (MO-CMA's mocma:g, PSO's swarm, successive halving's completed_rung_N), so
    the C# study holds what a native run stores. Values with no JSON shape (NumPy arrays, arbitrary objects) stay in
    the shadow only. Deep JSON-shaped conversion both ways (dict / list / tuple / scalars, NumPy scalars) and a
    structural equality (numbers by value, NaN matching itself) live here; PyConvert's public surface is unchanged.

OptunaHubSampler: BeforeTrial (BeginTrial, cached record refreshed, before_trial), InferRelativeSearchSpace /
SampleRelative (cached record), SampleIndependent (latest trial), AfterTrial (stored record; the shadow trial is
finished in a finally, as _tell_with_warning writes the state even when after_trial raises), all through Call.
OptunaHubPruner: Prune syncs its own shadow and calls prune(study, latest trial) as Trial.should_prune does.
Both dispose every shadow in Dispose. Calls stay serialized on the Python thread under the bridge's gate.

Pythonic.cs: Python.optuna.samplers.BaseSampler.before_trial / after_trial; Python.optuna.create_study takes the
study name; new Python.optunasharp_shadow - a small helper module defined once in the embedded interpreter
(add_trial via create_trial + create_new_trial(template_trial=), begin_trial, get_trial, latest, set_param, report,
set_user_attr, set_system_attr, set_study_user_attr, set_study_system_attr, set_state, attrs, changed), Python source
so each helper reads as the Optuna code it mirrors.

Tests:
  * Run2OptunaHubBridgeParityBugTests: the three repros pass (PSO trial 11 is the native swarm candidate bit for bit;
    MO-CMA completes 30 trials; successive halving prunes 36 of 40, trial 3 after step 1); OpenBugs categories and
    the CI=true -> Inconclusive guard removed, "Fixed 2026-11-12" notes added.
  * New OptunaHubNativeParityTests: each registry object run natively and through the bridge in the same
    interpreter (study name hub-parity, seeded), every trial's state, params (float.hex) and reported steps compared.
    Samplers: pso, mocma, nsgaii_with_initial_trials, speaii, implicit_natural_gradient, differential_evolution,
    restart_cmaes, levy_flight_sampler, nelder_mead, tpe_union_multivariate, random_search, and AutoSampler (GP-based,
    ~20 s a side, [Slow]). Pruners: successive_halving, hyperband, median, percentile, wilcoxon, threshold. All 18
    match. A package whose imports or lazily imported optional modules are missing reports Inconclusive.
  * BridgeSamplerParallelReadTests.OptunaHub_BothVolleys: the hub DE's pin moves from 49 collisions / 0 history
    repeats to 0 / 4, in both volleys. The old numbers were artifacts of the rebuilt shadow (every read of a round
    got the same shadow trial number, and the history cache never refreshed); a throwaway probe ran the same lockstep
    pattern natively in Python: all 64 points identical, bit for bit. The threaded volley gives 0 / 4 as well (three
    runs; the bridge serializes every call on the Python thread). The test gained a historyRepeats DataRow column.

Docs: docs/website-src/integrations/python/optunahub.md ("Compatibility and state" rewritten for the persistent
shadow, with the measured list and the remaining limits: Python-side state not persisted, non-JSON values stay in
the shadow, calls serialized with one shared history cache, a Hyperband-filtered C# study mirrored whole; the pruner
paragraph; the validation section points at the native-parity tests), integrations/python/index.md, the package
README.

Journal: QA-72 and QA-73 fixed 2026-11-12; a "Fixes (2026-11-12)" section maps the six groups to their commits and
tallies the run: 23 of 28 fixed, 3 kept [Misaligned] (QA-62, 69, 77), 2 deferred (QA-58, QA-82).

Gates (net10.0, live Python 3.12 + Optuna 5.0.0 + OptunaHub + pinned refs/optunahub-registry):
  * OptunaSharp.Tests.Interop, everything but Slow: 727 passed, 0 failed, 30 skipped (pycma and Redis not
    installed here). The 13 SqliteRdbUpgradeInteropParityTests failed first only for the repository-root shim
    missing in the fresh artifacts folder, and passed (13/13) once it was copied.
  * The OptunaHub classes on net8.0 as well: 29 passed. AutoSampler ([Slow]) passed on net10.0 (23 s).
  * OptunaSharp.Tests.Hosting: green (it builds every package's DI host, the bridge included).
  * check_documentation.cs over src: clean; OptunaSharp.Interop.pythonnet builds with the doc warnings as errors.
…blies built against 1.0.0 still bind

ApiCompat (Microsoft.DotNet.ApiCompat.Tool 11.0.100, `apicompat package` of each 1.1.0 candidate package against the
nuget.org 1.0.0 package, 2026-11-12) found one binary break across the 20 packages: CmaEsSampler's 14-parameter public
constructor of 1.0.0 was gone. 99e9e73 (plan 13, item D1) appended the optional `reuseToldOptimizer` as a 15th
parameter. Every source call site compiles unchanged, but the constructor got a new metadata signature, so a call
compiled against 1.0.0 (a library built on OptunaSharp 1.0.0, used by an application on 1.1.0) throws
MissingMethodException. The other 19 packages showed no breaking change; a positive control (Storage.Sqlite checked
against the Storage.Redis 1.0.0 baseline) confirmed the tool reports breaks. The net8.0-vs-net10.0 OrderedDict findings
inside Core are identical in 1.0.0 (the in-box OrderedDictionary base on net9+), so they are not new.
Owner's decision, 2026-11-12: "Restore 1.0.0 overload".

Change (src/OptunaSharp.Core/Samplers/CmaEsSampler.cs):
- a public constructor with 1.0.0's exact 14 parameter types that chains to the public one with
  reuseToldOptimizer: false (Optuna's path), so a 1.0.0 caller gets exactly the sampler it got before;
- no default values on it: with defaults, every call that leaves out an argument (new CmaEsSampler(),
  new CmaEsSampler(seed: 42)) would match both constructors equally well and fail to compile as ambiguous (CS0121);
  without them those calls bind to the public constructor, and only a call passing all fourteen arguments binds here,
  with the same result;
- [EditorBrowsable(EditorBrowsableState.Never)], so IntelliSense offers only the public constructor. The CLI's
  SamplerFactory tries public constructors widest first and binds the 15-parameter one for any --sampler-kwargs
  subset, so it never reaches this one.
- Documented in full: every parameter, the ValueError it can raise (the public constructor's checks), and remarks on
  why it exists and why it has no defaults.

Test (test/OptunaSharp.Tests/Samplers/CmaEsSamplerCacheTests.cs):
TheVersion1_0_0Constructor_StillBinds_AndIsOptunasPath looks the constructor up by 1.0.0's parameter types (null
before this commit, as ApiCompat reported) and invokes it the way a 1.0.0-compiled call does, every argument explicit
with 1.0.0's defaults; its 120 trials equal those of new CmaEsSampler(seed: 21, popsize: 6, reuseToldOptimizer: false).

Gates: Core builds with the documentation warnings as errors (0 errors, only the known CS8632 nullable-context
warnings); the 53 CmaEs* tests pass on net10.0 and net8.0. ApiCompat and the full unit filter run again on the final
1.1.0 candidate.
…to SciSharp main, so 155 citations in seven docs named commits that no longer exist there

Owner's decision, 2026-11-12: rebase the local journey1 line onto origin/main (SciSharp main = f36573c, the owner's
README edit made on GitHub on 2026-11-09) and open a PR from it, instead of merging. The rebase replayed the 125
commits since v1.0.0 (journey1 e9c3c81) with no conflict, and the rebased tip's tree is byte-identical to the merge of
the two (c3f1376). Every commit got a new hash, so documents written before the rebase cited hashes the public history
never has. This commit rewrites those citations; nothing else changes.

How each citation was mapped (scratchpad script find_old_refs.py):
- the 125 old/new pairs come from rev-list v1.0.0..e9c3c81 and f36573c..(rebased tip), in order, and were checked
  pair by pair: same author, author date and full message, and the same added/removed lines (0 mismatches);
- a citation is a 7-40 character hex token, bounded by non-word characters, that is a prefix of exactly one old hash
  AND that git resolves to that commit alone (git rev-parse --disambiguate, commits only), so a token that only looks
  like a journey1 hash, or is ambiguous with another commit, is left alone;
- the replacement keeps the token's length (a 7-character citation becomes the new commit's first 7 characters), and
  every new token resolves to exactly one commit. A re-scan finds no old citation left.

Rewritten: docs/PORTING_PLAN.md (6), docs/plans/13-rustuna-derived-performance.md (24), 14-rustuna-second-pass.md (27),
15-performance-pass-3.md (43), plans/README.md (4), docs/qa/PARITY_QA_2026-11-02.md (11), PARITY_QA_2026-11-10.md (40):
155 citations of 49 commits, on 125 lines. Line endings are untouched (the files were rewritten as bytes).

Not rewritten: the messages of the rebased commits themselves. 45 of them cite an earlier journey1 hash (126
citations); changing them would mean rewriting those commits again. A cited old hash maps to the commit with the same
subject and author date among the 125 commits after f36573c.
…ASE_<version>.md, NumSharp's notes file, notes first and the install table last

Owner, 2026-11-12, asked where the 1.1.0 release notes should live: "See other release notes and produce same format".
The other release notes are NumSharp's (OptunaSharp's infrastructure template) and OptunaSharp's own 1.0.0:
- NumSharp commits its notes as docs/releases/RELEASE_<version>.md, written to its changelog skill's house style
  (refs/NumSharp/.claude/skills/changelog/references/CHANGELOG_STYLE.md). Its create-release job appends that file to
  the GitHub Release body after the install lines and the package table, and turns GitHub's generated notes off when
  the file exists (refs/NumSharp/.github/workflows/build-and-release.yml, "Compose release notes").
- The published releases put the notes FIRST and the install table last, under a titled name: NumSharp 0.80.0
  ("NumSharp 0.80.0 - Strings Stable Release": intro, Overview, Detailed Breakdown, then Install via NuGet and
  Packages) and OptunaSharp 1.0.0 ("OptunaSharp 1.0.0 - Stable": Features, then the install lines and the package
  table). Both were rearranged by hand after their release runs.
OptunaSharp's create-release composed only the install lines, the package table and GitHub's generated notes, which
here are a bare compare link, because nothing merges through pull requests.

Change (.github/workflows/build-and-release.yml, create-release):
- A sparse checkout of docs/releases. The job had no checkout; the notes are read from the tagged commit.
- Compose release notes (now id: notes): when docs/releases/RELEASE_<version>.md exists, its H1
  ("# OptunaSharp <version> - <title>") becomes the release's name and the rest of the file opens the body, followed
  by "---", the install lines and the package table, as before. The H1 must name the tagged version (exactly, or
  followed by a space and a title); otherwise the job fails before anything is created, which catches notes copied
  from the previous release and not renamed. Without the file the body is today's (an "## OptunaSharp v<version>"
  header and the install table) and the name "OptunaSharp v<version>".
  The name leaves the step through a delimited GITHUB_OUTPUT block, so a title with characters a key=value line would
  misread passes through whole; HAS_NOTES is a plain key.
- Create Release takes name: from the step, and generate_release_notes only when there is no notes file.

Checked:
- The step's script, extracted from the YAML and run three ways: a stand-in notes file for 1.1.0 (the name taken
  from the H1, the notes first, the install table after "---"), a version without a file (today's body and the
  "OptunaSharp v<version>" name), and a file whose H1 names another version (exit 1 with the error, nothing written
  to GITHUB_OUTPUT).
- actionlint 1.7.12 on the workflow (-shellcheck= -pyflakes=, neither tool is installed here): no findings.
…8 commit bodies since 1.0.0 read and bucketed, performance measured against the published 1.0.0

docs/releases/RELEASE_1.1.0.md, the file the create-release job now turns into the GitHub Release (previous commit).
Owner, 2026-11-12: "See other release notes and produce same format". Format and process are NumSharp's changelog
skill (refs/NumSharp/.claude/skills/changelog: CHANGELOG_STYLE.md for the shape, triage-taxonomy.md and
verify-accuracy.md for the process), laid out like NumSharp 0.80.0's published release: an H1 that names the release,
an intro, "Overview", then "Detailed Breakdown" in a collapsed <details> with Dashboards & Docs, New APIs & Modules,
Performance, Parity & Fixes, Testing & Tooling and Breaking Changes, last. NumSharp-only sections (New NuGet Packages,
ndarray surface) are omitted: 1.1.0 adds no package. Hashes are 7 characters, OptunaSharp's convention.

.gitignore: the Visual Studio build-output rule [Rr]eleases/ hid docs/releases/, so the file could not be added; a
!docs/releases/ exception follows the rule, as in NumSharp's .gitignore. Checked with git check-ignore: the notes file
is tracked, while bin/ outputs and any other releases/ folder stay ignored.

Title proposed: "OptunaSharp 1.1.0 - Performance & Parity Release". The two-sentence intro is a draft for the owner to
rewrite in their own voice (NumSharp's notes open with the owner's paragraph).

Process: every commit of v1.0.0..(this branch) was read, subject and body, oldest first, in batches of 16 (scratchpad
batches/b01..b08), and put in one of the five buckets:
- Cited, 45 commits: every perf commit (27), every fix but two, both feat(storage) commits, the expm1 refactor, the
  JIT-posture chore, and the QA run's closing journal commit (Testing & Tooling).
- Folded, 2: dfc026f (the TPE observation cache's reused-id regression) and d7aa9d9 (step F's split regression) fix
  changes made inside this range, so their substance is in the lines of the features they repaired (style 10.4).
- Excluded, 81: 46 docs commits (plans 13-15, the QA journal's progress entries, cref fixes, the README edit), 29
  test(qa) commits (the [OpenBugs] repros; their fixes are cited instead), and the 6 example commits, which surface
  without hashes in Dashboards & Docs (style 7B).
Two claims found in bodies were not carried over:
- QA-41 / QA-42 "fixed": c5d79f9 credits plan 14 step A, 8a7e19e credits NumSharp 0.80.0's legacy RandomState
  rewrite, and 0.80.0 already shipped with OptunaSharp 1.0.0. Which release fixed them is not established, so no line.
- 2189a12's "continuous sums kept in registers": 465f91c (QA-84) replaced that code with NumSharp's np.sum, so only
  the step's stepped-dimension work is in the TPE line.

Numbers: the Overview and the first nine Performance lines were measured for these notes, against the PUBLISHED 1.0.0,
because the passes measured against three different baselines and QA-84 then took part of TPE's gain back. A
file-based consumer outside the repository (its own nuget.config: OptunaSharp* from a local folder holding the
nuget.org OptunaSharp.1.0.0.nupkg and the candidate packed as 1.1.0-local.1 from e9c3c81, the pre-rebase tip, whose
library code is this branch's except the 1.0.0 CmaEsSampler constructor overload; NUGET_PACKAGES isolated), Release,
.NET 10, EcoQoS opted out, one process per (version, workload), rep 0 (JIT included) and rep 1, two alternating
rounds on an i9-13900K shared with other sessions; each cell is the better round. Seed 0, shifted sphere over 40
floats unless stated; two-objective spheres for NSGA-II and MOTPE.

  workload                1.0.0 rep0 / rep1   1.1.0 rep0 / rep1   ratio        allocated          same trials
  random, 10,000           6.91 / 5.50 s       1.11 / 0.77 s       x6.2->x7.1   3,608 -> 666 MB    yes
  QMC (default), 10,000    5.16 / 3.72 s       1.29 / 0.75 s       x4.0->x5.0   3,740 -> 485 MB    yes
  QMC scrambled Sobol,     4.12 / 3.39 s       0.36 / 0.17 s       x11.6->x20.4 1,255 -> 60 MB     yes
    4,000 over 10 floats
  NSGA-II, 10,000          4.72 / 3.29 s       1.96 / 0.98 s       x2.4->x3.4   2,912 -> 723 MB    yes
  TPE, 1,000               10.75 / 9.72 s      4.24 / 3.25 s       x2.5->x3.0   4,441 -> 840 MB    no (moves toward Optuna)
  TPE, 1,000 over 10       3.69 / 2.62 s       2.35 / 1.11 s       x1.6->x2.4   1,296 -> 259 MB    no
  MOTPE, 300               4.46 / 3.25 s       2.59 / 1.29 s       x1.7->x2.5   2,386 -> 340 MB    no
  CMA-ES, 10,000           27.60 / 25.45 s     4.46 / 3.66 s       x6.2->x7.0   10,896 -> 1,691 MB yes
  GP, 150 over 10          17.86 / 16.40 s     11.49 / 9.12 s      x1.6->x1.8   5,893 -> 1,513 MB  yes
  (round 1 ran 1.0.0 first, round 2 the candidate first; round 1's TPE ran under heavy load on both sides, e.g.
  1.0.0 TPE 25.9 s against round 2's 10.75 s. "Same trials" = an FNV hash of every trial's number, parameters and
  values by their bits, equal across versions and rounds.)
The equal hashes are the evidence behind the Breaking Changes line "Random, QMC, NSGA-II, plain CMA-ES and
single-objective GPSampler over floats give 1.0.0's trials bit for bit"; the TPE and MOTPE hashes differ, as the
expm1 change (0e18eba) and QA-84 (465f91c) predict.
Every other number is quoted from a commit body, using the final value where one moved: CMA-ES 100,000 trials
987.7 -> 27.5 s and the compact-storage retention (ca9b9fe, f0d62a0, measured against de495f8, whose library code is
1.0.0's); plan 15's final survey for BruteForce, SMS-EMOA / MOEA/D, the ES family and DE / PSO / LM-CMA / VkD-CMA
(283876b); TuRBO (9533366); SQLite (2dc35cf, 9043b43); the journal offsets (45fadd1); the CLI cycle (27269b1); the
parity counts from the fix commits and docs/qa/PARITY_QA_2026-11-10.md.

Checks (verify-accuracy.md): all 45 cited hashes resolve to commits of the release range; the Overview's counts are
the file's (43 parity fixes: 36 QA findings plus expm1, hypervolume / HSSP / UNDX, multi-objective GP
standardization, the percentile pruner's zero signs, AGE-MOEA-2's row norms, JANUS's component mean and the journal
lock; 18 measured speedups; 11 behavior changes; 130 commits since 1.0.0 including this one); no em or en dashes; the
"fixed" verbs match bodies that fixed (repro tests that lost [OpenBugs]), and the deferred / kept findings are named as
such.
…se note in Getting Started, the FAQ and the package README; the README's parallelism line no longer promises async-await

The two documentation follow-ups left open since the 1.0.0 release (catalog State, 2026-11-08). Owner, 2026-11-12:
"Fix them and after release commit them back in".

1. NativeAOT. The 1.0.0 smoke test found that a consumer written as a .NET 10 file-based app fails in trial 0: file-based
   apps default to PublishAot=true, which turns RuntimeFeature.IsDynamicCodeSupported off, and NumSharp compiles its
   kernels at run time (DirectILKernelGenerator). Only the root README's contributor section ("Scripting with dotnet
   run") showed #:property PublishAot=false. Now:
   - docs/website-src/docs/getting-started.md, Install: OptunaSharp needs dynamic code, does not run under NativeAOT, and
     a file-based app needs the property next to its package directive; the exact error it fails with otherwise.
   - docs/website-src/docs/faq.md: "Can I publish with NativeAOT, or run OptunaSharp from a file-based app?" with a
     complete file-based example; a regular project needs nothing extra.
   - src/OptunaSharp.Core/README.md (the README nuget.org shows): the same note in two sentences.
   Checked 2026-11-12 with nuget.org's OptunaSharp (isolated NUGET_PACKAGES, nuget.org as the only source): the
   Getting Started directives with a 5-trial TPE study print "ok 1.0.0 ..."; the same file without the property fails
   trial 0 with "PlatformNotSupportedException: Dynamic code generation is not supported on this platform." (the text
   quoted in all three places); the FAQ's code block, extracted from the page as written, compiles and runs.
   #:package OptunaSharp@* floats to the newest release, so the pages do not go stale with each version.

2. README.md, line 27 (the owner's line of 2026-11-08): "reused dedicated threads and ThreadPool async-await" becomes
   "reused dedicated threads or the .NET ThreadPool". OptunaSharp has no async API (the 2026-11-08 async evaluation
   built none); the link already points at the three executors the page documents (nJobs: 1, TrialWorkerScheduler,
   TrialExecutorKind.ThreadPool).
…n commit (6fa30e7), now part of the release

Owner, 2026-11-12, in the PR round: "Include in this PR" for the two documentation fixes. 6fa30e7 was written for
after the release (its message quotes the owner's earlier "Fix them and after release commit them back in"); it is now
on the release branch, so the notes account for it.

docs/releases/RELEASE_1.1.0.md:
- Dashboards & Docs gains one line: Getting Started, the FAQ and the package README say that OptunaSharp needs dynamic
  code, does not run under NativeAOT, and that a .NET 10 file-based app needs #:property PublishAot=false, with the
  error it fails with otherwise. Like the examples line, it carries no hash (CHANGELOG_STYLE 7B); 6fa30e7 is a docs
  commit that documents a user-facing contract, the one kind of docs commit the style lets into the notes.
- The Overview's count: "of the release's 132" (v1.0.0..this commit). The cited set is unchanged at 45 commits; the
  README parallelism line in the same commit is cosmetic and gets no line.
…passes moved - UNDXCrossover, AGE-MOEA-2 at 8+ objectives, the hypervolume

Found while reading the development sessions' transcripts (Agentmaster, --minimal) for the numbers the 1.1.0 PR
description now carries, at the owner's request in the PR round: "Read the transcripts (minimal) via agentmaster cli
to find the numbers of each development".

The Breaking Changes bullet "Seeded results move toward Optuna's" listed TPE, CMA-ES with margin, NSGA-III,
GridSampler, multi-objective / stepped GP and one-thread NSGA batches, then said "Random, QMC, NSGA-II, plain CMA-ES and
single-objective GPSampler over floats give 1.0.0's trials bit for bit". Two NumSharp commits move results it did not
name:

- c600736 (Optuna's 1-D np.dot / np.linalg.norm through NumSharp's OpenBLAS ddot / dgemv): UNDXCrossover's products
  and norms. Its body: NsgaiiSamplerParityTests.UndxCrossover_ManyParameters_FullState_IsBitIdentical (24 float
  parameters, 48 trials, seeds 0 and 7) "failed on both seeds before, bit-identical full state after"; the
  3-parameter UNDX test passed before and after. So a seeded NSGA-II / NSGA-III study with UNDXCrossover over many
  parameters changes. The same commit moved 37 of 240 hypervolumes by an ulp or two (HypervolumeParityTests, live
  against Optuna 5.0.0), so a hypervolume can differ from 1.0.0's by an ulp.
- 156ce8a (reductions in NumPy's order): AgeMoeaSurvival's per-row norms now follow np.linalg.norm(front, p, axis=1),
  "equal below eight objectives; NumPy's pairwise rows from eight on". AGE-MOEA-2 is matched to pymoo, so it moves
  toward pymoo's, not Optuna's.

docs/releases/RELEASE_1.1.0.md:
- The seeded-results bullet adds "`UNDXCrossover` over many parameters" to the moved list, then "AGE-MOEA-2 at 8 or
  more objectives moves toward pymoo's, and a hypervolume can differ from 1.0.0's by an ulp"; the unchanged list says
  "NSGA-II with its default crossover"; the bullet cites c600736 as well. c600736 was already cited by the Parity and
  Testing lines, so the notes still cite 45 distinct commits.
- The Overview's count: "of the release's 133" (v1.0.0..this commit).
@Nucs

Nucs commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

On hold: waiting for NumSharp 0.81.0 for its fixes and performance boosts. The NumSharpVersion bump comes to this branch before the merge.

What 1.1.0 waits for:

  • Performance: np.sum(axis=-1) over short rows and into out=, both pinned as OpenBugs in NumSharp. Until they are fixed, TPE's pairwise continuous sum (QA-84) costs 27-66 %.
  • Fixes: tied signed zeros in np.percentile / np.quantile (OptunaSharp's QA-85, NumSharp's QB43), and float np.prod in NumPy's sequential order; the hypervolume keeps its own product until then.
  • At the bump: NumPyDrawsTests expects System.OverflowException where 0.81.0 throws NumSharp.OverflowError.

Nucs added 7 commits November 14, 2026 03:34
…get; QA-85 fixed upstream

Owner, 2026-11-14: "NumSharp 0.81.0 was released, using am cli find all the minimal transcripts about numsharp
needing to do some fixing or changes, well it did all of them and now we need to align, fix, migrate to NumSharp.
Commit and push and prepare cicd to release". PR #6 had been on hold for it ("waiting for NumSharp 0.81.0 for fixes
and performance boosts").

The bump
- Directory.Build.props: $(NumSharpVersion) 0.80.0 -> 0.81.0 (NumSharp and NumSharp.Interop.OpenBLAS, both on
  nuget.org). The whole solution compiled unchanged against 0.81.0's 20 breaking changes (NumPy's signatures, Python
  exception classes, read-only scalars); refs/NumSharp moves to the v0.81.0 tag (65ef305).

What 0.81.0 changes for OptunaSharp, measured
- NumSharp.Interop.OpenBLAS 0.81.0 bundles numpy 2.5.4's scipy-openblas 0.3.34.106.0 (0.80.0 bundled numpy 2.4.2's).
  The 36-study seeded hash probe (plan 14/15 harness), base 38735c2 vs the bump: 31 studies identical, 5 moved in
  their last bits, all through BLAS / LAPACK: cmaes-mixed and cmaes-ipop (eigh), xnes (eigh), nsga2-undx (ddot,
  norm) and janus. The 8-study GP probe is identical. So the byte-exact target moves to NumPy 2.5.4:
  - CI pins numpy==2.5.4 where it pinned 2.4.2 (build-and-release.yml: the satellite job's vendored Optuna, the
    interop job's install, its re-pin after torch/numba and the assert), and so does the package-parity
    Dockerfile; numba 0.68 (numpy < 2.6) and scipy 1.18 (numpy < 2.8) accept it, and the sdk:10.0 image is Ubuntu
    24.04 with Python 3.12, which numpy 2.5.4 (>= 3.12) needs.
  - The CMA-ES pickle oracle is regenerated under numpy 2.5.4: CmaPickleOracleTests' 30-dimensional case compared
    a told state whose eigendecomposition came from numpy 2.4.2's OpenBLAS; 27/27 pass on the regenerated file.
    Its generator could not run: 73dd305 had written its newline="\n" argument as a literal line break (a syntax
    error), repaired byte-exactly; its docstring now names numpy 2.5.4 and says why two runs never write the same
    file (cmaes' np.empty _funhist_values, gzip timestamps) while the tests still compare.
- NumPy's exception classes: 0.81.0's np.random uniform raises NumSharp.OverflowError (Python's class, under
  ArithmeticException) where 0.80.0 raised .NET's OverflowException. NumPyDraws.Uniform mirrors that call, so it
  raises OverflowError("Range exceeds valid bounds") too, as NumPy does; NumPyDrawsTests and the QA-41 test
  (RandomSampler over an infinite range) expect the class exactly. A trial failed this way now logs Optuna's own
  line, OverflowError('Range exceeds valid bounds').
- QA-85 is fixed upstream: 0.81.0's several-kth partition runs NumPy's introselect_ statement for statement (its
  QB43 fix), so PercentilePruner.NanPercentile's median of [0.0, -0.0, -0.0, -1.0] is -0.0 as in NumPy 2.4.2 and
  2.5.4 (both checked live). PrunerTests.NanPercentile_TiedSignedZeros_KeepNumpysZeroSign passes and loses
  [OpenBugs]; the pruner's remarks, the test docs and PARITY_QA_2026-11-10.md say so.
- Doc alignment: both GP search spaces document NumSharp's np.arange ZeroDivisionError for a zero step (0.81.0's
  class; it was DivideByZeroException); TPE's np.sum remark records that NumSharp's two pinned costs behind QA-84
  (SumOverShortRows_CostsAFewLongRowSums, SumIntoOut_CostsAboutAFreshSum) are still [OpenBugs] in 0.81.0, so
  QA-84's 27-66 % TPE cost stays; CLAUDE.md names 0.81.0, the bundled OpenBLAS and the pin that follows it.

Not in 0.81.0 (checked in its sources and its OptunaSharp consumer pins), so OptunaSharp keeps its code
- The two np.sum(axis=-1) performance pins: still [OpenBugs] in OpenBugs.Consumers.OptunaSharp.cs.
- np.einsum at optimize=False: NumSharp's owner kept the pairwise BLAS contraction ("Keep the BLAS route"), so
  NSGA-III's 4-operand einsum stays a hand port.
- The default argsort tie order (QB-52): still [OpenBugs] in NumSharp.
- A public triangular solve: OpenBlasEngine's Trsm is still behind internal interfaces.

The premise, live (interop, net10.0, the vendored Optuna 5.0.0): the 57 BLAS-backed parity cases
(CmaEsSamplerParityTests, LinAlgParityTests, NsgaiiSamplerParityTests, HypervolumeParityTests) fail 14 times against
numpy 2.4.2 with NumSharp 0.81.0 (CMA-ES long runs, lr_adapt, warm start, UNDX) and pass 57/57 against numpy 2.5.4.
LinAlg's remark records it, with the footgun that the pin must follow the bundled OpenBLAS.

Verified on the tree with this commit and the four after it (none of them moves a seeded result: the 36- and 8-study
probes are identical to this commit's alone): whole solution builds; OptunaSharp.Tests on net10.0 and net8.0 2,310
passed, 1 skipped, 41 failed, all 41 [OpenBugs] (none unexpected); CmaPickleOracleTests 27/27.
…or and FileNotFoundError from NumSharp 0.81.0 where the port raised .NET stand-ins

NumSharp 0.81.0 ships every exception class of CPython 3.12 and NumPy 2.5.4, each under its .NET twin's base
(OverflowError / ZeroDivisionError under ArithmeticException, OSError and its subclasses under IOException). Until
now the port raised .NET types where Optuna, CPython or NumPy raise these, because NumSharp had no such class; the
code comments said so ("DivideByZeroException is this port's ZeroDivisionError"). The owner asked to "align, fix,
migrate to NumSharp" once 0.81.0 shipped, so every such stand-in now raises the Python class with the same text.

Raised as Python's class now (the message is unchanged in every case)
- OverflowError: PyStrFormat's "%c arg not in range(0x110000)" and "int too large to convert to float"
  (str.format / the SQLAlchemy and storage URL templates); SqlAlchemyUrl's int() of an infinity ("cannot convert
  float infinity to integer"); the EF Core MySQL URL's C int arguments ("Python int too large to convert to C long",
  "signed integer is greater than maximum" / "... less than minimum"); the Redis URL's TLS version and timeouts
  ("Python int too large to convert to C long", "timestamp out of range for platform time_t"); the SQLite URL's C int
  arguments ("Python int too large to convert to C int").
- ZeroDivisionError: separable CMA-ES with popsize 1 ("division by zero", cmaes' int division; QA-80's fix raised
  DivideByZeroException).
- FileExistsError: DownloadArtifact onto an existing file or directory ("File already exists: <path>", QA-78's
  fix raised IOException). FileExistsError IS an IOException, so a catch (IOException) still sees it.
- FileNotFoundError(2, "No such file or directory"): the MySQL URL's ssl_ca / ssl_cert files and the Redis URL's
  ssl_ca_certs (they raised FileNotFoundException with the same text). The Redis client certificate and key now
  check their files first too: Python's load_cert_chain raises that FileNotFoundError for a missing one, where
  .NET's PEM loader threw FileNotFoundException with .NET's own text.

Why it matters beyond the type
- A trial that fails on one of these logs Optuna's exact line, "... because of the following error:
  ZeroDivisionError('division by zero')", because the warning prints the exception's class name.
- The QA-5 repro tests (IntDistribution.ToExternalRepr(inf), suggest_int of an enqueued inf; still [OpenBugs])
  now expect Python's OverflowError("cannot convert float infinity to integer") exactly, the fix they pin.

Kept as .NET types (not Python errors): RDBStorage's 32-bit step column limit, journal and pickle size limits,
Int32/Int64 conversions of stored values, and every other OverflowException that has no Python counterpart.

Breaking for C# callers that caught the old types: OverflowException, DivideByZeroException and
FileNotFoundException are not the new classes' bases (catch OverflowError / ZeroDivisionError / FileNotFoundError,
or ArithmeticException / IOException). Listed under the 1.1.0 notes' Breaking Changes.

Tests: EFCoreDriverParityTests, RedisUrlTests and RedisTlsTests expect the classes by exact type; the separable
CMA-ES and artifact QA tests use ThrowExactly; the interop StorageUrlStandardParityTests maps OverflowError to
Python's "OverflowError|..." outcome.
…replaces the vendored concurrent collections

The owner's TODO of 2026-10-27 (834a368), above both vendored types: "once NumSharp 0.80.0 is released, use its
NumSharp.Collections.ConcurrentOrderedDictionary<TKey,TValue> instead of this type ... then delete this file.
ConcurrentOrderedDictTests pins this copy's contract: point the tests that still apply at the NumSharp type and
drop the rest." 0.80.0 shipped neither type publicly; 0.81.0 ships ConcurrentOrderedDictionary (public sealed,
NumSharp.Collections) and keeps its ConcurrentDictionary internal.

- Deleted src/OptunaSharp.Core/Collections/Concurrent/ConcurrentOrderedDict.cs (internal), ConcurrentDictionary.cs
  and ConcurrentDictionaryInternals.cs (the vendored dotnet ConcurrentDictionary, internal, whose only consumer was
  ConcurrentOrderedDict): about 3,550 lines no library code used (THREADING.md §5.1 said so). Nothing public
  changes, so ApiCompat against 1.0.0 is unaffected.
- ConcurrentOrderedDictTests -> NumSharpConcurrentOrderedDictionaryTests: all 22 contract tests still apply
  (NumSharp's type is a superset of the vendored API: the same constructors, key and index paths, GetOrAdd /
  AddOrUpdate / TryUpdate, RemoveAt, snapshot enumeration) and now construct NumSharp's type; the two smoke tests
  of the vendored ConcurrentDictionary clone are dropped with it.
- docs/THREADING.md: §5.1 and the test table name NumSharp's type.
…0.81.0's np.prod, bit for bit

NumSharp's fix brief (session ab93c03a, 2026-11-12) named the consumer follow-up: "After items 2 and 3 land,
OptunaSharp can drop its hand-ported sequential products (Hypervolume, HypervolumeHistory, FanovaTree)". Item 2
landed: 0.81.0's float np.prod multiplies in NumPy's sequential multiply.reduce order on every route (0.80.0's SIMD
lanes did not, which is why the port wrote loops).

The loops stay, by the rule the NumSharp migration of 2026-11-11 set (owner-approved): where a hand-written helper
already gives NumPy's exact result, it stays if it runs per point or per trial, because a NumSharp call costs about
2 µs there (MathUtil's sums, TPE's argmax). ProdRefMinus runs once per point inside the hypervolume / HSSP /
box-decomposition loops, HypervolumeHistory.Prod once per trial, FanovaTree.GetCardinality once per tree node.

What changes instead
- SequentialProductTests (new): the three private helpers, reached by reflection as NumPyReductionOrderTests reaches
  its own, equal NumSharp's np.prod(a - b) bit for bit: 400 random rows of 2-16 objectives (magnitudes 1e-3..1e3,
  both signs) for the two hypervolume helpers, 400 random bound pairs of 1-16 features for the cardinality, plus rows
  with an infinite distance, zero times infinity and a signed-zero distance. If a NumSharp release reorders np.prod
  again, these fail and name the helper.
- The three helpers' comments and Hypervolume's class remark say np.prod is now NumPy's order in NumSharp too, why
  the loop stays, and which test pins the equivalence.

Item 3 (np.einsum at optimize=False computing like c_einsum) did not land: NumSharp's owner kept the pairwise BLAS
contraction, so NSGA-III's 4-operand einsum stays a hand port, unchanged.
… 2's QA-83 and QA-80, still marked open

Found by the NumSharp 0.81.0 bump's full unit run, which includes the [OpenBugs] category: three known-failing tests
passed. One was QA-85, fixed by NumSharp. The other two passed on the 0.80.0 base as well (re-run on 38735c2):
- ErrorContractParityBugTests.CmaEs_X0MissingParameter_FailsWithPythonsMessage (QA-17, run 1): an x0 missing a
  parameter raises Python's "Parameter configuration must contain all distributions." since 1e8aba6 made
  SearchSpaceTransform.TransformValues check it (run 2's QA-83).
- SamplerParityBugTests.CmaEs_X0OutsideBounds_FailsTrialsLikeCmaes (QA-34, run 1): Cma / SepCma assert cmaes'
  _is_valid_bounds since 1e8aba6 (run 2's QA-80).
1e8aba6 marked QA-80 / QA-83 fixed in the run 2 journal but left run 1's repro tests marked [OpenBugs], so CI (which
excludes the category) never ran them. Both lose the attribute and gain a "fixed by 1e8aba6" note;
PARITY_QA_2026-11-02.md marks QA-17 and QA-34 fixed.

Also, a follow-up to 9f77939: the QA-80 (separable CMA-ES, popsize 1) and QA-78 (DownloadArtifact) tests' docs
name the classes they now expect, NumSharp.ZeroDivisionError and NumSharp.FileExistsError (an IOException).
…r NumPy 2.5.4

The NumSharp 0.81.0 bump (9d3285f) moved the live oracle to numpy 2.5.4, the NumPy whose OpenBLAS 0.81.0 bundles.
NumPy 2.5 removed np.row_stack, and pymoo 0.6.1.3 calls it in NSGA-II's crowding distance
(rank_and_crowding/metrics.py: np.row_stack([F, np.full(n_obj, np.inf)])). The full interop suite on net10.0 against
numpy 2.5.4 (2026-11-14): 675 passed, 30 failed, 52 skipped; 17 of the failures were every pymoo-bridge case
(PymooSamplerParityTests, BridgeSamplerParallelReadTests' two pymoo cases, PythonThreadTests' PymooSampler case), all
"module 'numpy' has no attribute 'row_stack'". The other 13 are SqliteRdbUpgradeInteropParityTests, which cannot find
the repository root from an out-of-tree build folder (the known out-of-tree limitation; CI runs them in-tree).

Why not pymoo 0.6.2 (the latest, which no longer calls row_stack): it changed AGE-MOEA-2's arithmetic. Diffing its
age2.py against 0.6.1.3 without docstrings, the Newton-Raphson curvature now evaluates point**x as
exp(x * log(point + 1e-10)) with overflow guards, and the distances sum with np.sum where 0.6.1.3 used Python's sum;
the native AgeMoea2Optimizer reproduces 0.6.1.3 generation by generation, so 0.6.1.3 stays the oracle.

What changes: PythonSession, after setting NPY_DISABLE_CPU_FEATURES and before Optuna's import, sets
numpy.row_stack = numpy.vstack when NumPy lacks it. NumPy 2.0-2.4 defined row_stack as exactly that alias, so no
result changes. Every other NumPy name pymoo 0.6.1.3 uses that 2.5.4 lacks (np.Inf, np.NaN, np.math, np.product) was
already gone in NumPy 2.0, so no test reaches it. The product's pythonnet runtime leaves the user's NumPy untouched:
with NumPy 2.5, a pymoo bridge user needs pymoo 0.6.2.

Also checked: NumPy 2.5.4 keeps the CPU dispatch names PythonSession disables on AVX-512 Linux hosts (X86_V4,
AVX512_ICL, AVX512_SPR), so that guard still applies.

After this change: the pymoo, AGE-MOEA-2, bridge-parallel-read and Python-thread suites 66 passed, 5 skipped, 0
failed against numpy 2.5.4.
…e target, Python's exception classes, QA-85 / QA-17 / QA-34, the benchmark re-run

The NumSharp 0.81.0 migration (9d3285f..8e7418e) changed what the 1.1.0 notes promise, and the benchmark re-run on
it changed their numbers. docs/releases/RELEASE_1.1.0.md now says so; the PR description follows.

Intro and Overview
- A sentence and an Overview bullet for NumSharp 0.81.0: its OpenBLAS is NumPy 2.5.4's (scipy-openblas
  0.3.34.106.0), so the bit-for-bit target is Optuna running on NumPy 2.5.4; QA-85 is fixed upstream; Python's
  OverflowError, ZeroDivisionError, FileExistsError and FileNotFoundError are raised where Optuna raises them.
- The binary-compatibility bullet names the new behavior changes (Python's exception classes, BLAS-backed results
  following NumPy 2.5.4).
- The count line, recounted from the notes by a script, by the rules that gave the earlier figures: 48 parity fixes
  (39 QA ids, 9 items without one; was 43 = 36 + 7), 18 measured speedups (unchanged), 14 behavior changes (was 11),
  50 cited commits (was 45) of the release's 140 (v1.0.0..this commit; was 133), 2,310 unit tests on each framework
  (was 2,307), 751 live-Optuna interop tests against NumPy 2.5.4 (was 727). The 751 are the complete non-Slow
  interop suite on net10.0, run 2026-11-14 with every backend installed (numpy 2.5.4, cma 4.4.4, the vendored
  optunahub and its registry, JANUS_REF_PATH): 751 passed, 0 failed, 6 skipped - five Redis cases without a local
  server (CI's Linux leg starts one) and PreV050_AppliesNoConstraints_MatchBaseProblemCall, which needs
  optunahub < 0.5.0.

Performance - re-measured 2026-11-14 on NumSharp 0.81.0 by the same method as e3a25cb: the published 1.0.0 package
against the 1.1.0-local.4 candidate, one process per version and workload, two reps (rep 0 cold, rep 1 warm), best
of two alternating rounds, an isolated NUGET_PACKAGES, on the shared i9-13900K.
- random x6.2->x7.1 -> x5.3->x8.1; CMA-ES x6.2->x7.0 -> x5.3->x10.3; QMC x4.0->x5.0 -> x4.1->x6.0; scrambled
  Sobol x11.6->x20.4 -> x12.1->x16.8; NSGA-II x2.4->x3.4 -> x2.3->x3.2; TPE (40 floats) x2.5->x3.0 -> x2.7->x3.4,
  4.4 GB -> 0.81 GB (was 0.84); TPE over 10 floats x1.6->x2.4 -> x1.5->x2.3; multi-objective TPE x1.7->x2.5 ->
  x1.7->x2.3; GP x1.6->x1.8 -> x1.5->x1.6, 5.9 GB -> 1.3 GB (was 1.5), gen-2 collections 7x to 9x fewer (was about
  4.5x). The Overview's span of reductions becomes 4.0x to 21x (was 3.9x).
- CMA-ES's 10,000-trial workload no longer hashes to 1.0.0's trials: its eigendecomposition follows NumPy 2.5.4's
  OpenBLAS now. The Overview and the CMA-ES line say so and cite 9d3285f. Random, QMC, scrambled Sobol, NSGA-II and
  GP still give 1.0.0's trials, hash for hash; the TPE workloads differed already (QA-84, expm1).

Parity & Fixes
- The CMA-ES line adds run 1's QA-17 and QA-34 (5b0e4fd: fixed by 1e8aba6, their repro tests run in CI again).
- New lines: NumSharp 0.81.0's OpenBLAS (the 57 BLAS-backed live cases pass, 14 of them fail against NumPy 2.4.2;
  9d3285f), QA-85 (the median of tied signed zeros is NumPy's -0.0, NumSharp's several-kth partition being NumPy's
  introselect_; 9d3285f), and Python's exception classes (14 throw sites in 8 files; 9f77939, 9d3285f).

Testing & Tooling
- NumPy 2.5.4 as the live oracle's NumPy, the regenerated CMA-ES pickle oracle and the np.row_stack alias for the
  pinned pymoo 0.6.1.3 (9d3285f, 8e7418e); SequentialProductTests, 800 random cases against np.prod (784f495).

Breaking Changes
- DownloadArtifact raises FileExistsError (an IOException), separable CMA-ES with popsize=1 ZeroDivisionError
  (9f77939 joins 3009c4a and 1e8aba6).
- "plain CMA-ES" leaves the list of samplers that give 1.0.0's trials bit for bit.
- New: BLAS- and LAPACK-backed seeded results follow NumPy 2.5.4's OpenBLAS (5 of 36 probe studies moved: CMA-ES
  over mixed parameters, IPOP CMA-ES, NSGA-II with UNDX, xNES and JANUS; CMA-ES with margin, separable CMA-ES and
  the 8 GP studies did not); Python's exception classes replace .NET stand-ins, and OverflowException,
  DivideByZeroException and FileNotFoundException are not their bases; the NumSharp 0.81.0 dependency, whose own
  breaking changes reach code that calls NumSharp directly.

Not cited: 1592d05, which deletes the vendored concurrent collections (internal, unused) for NumSharp's public
ConcurrentOrderedDictionary, changes nothing a user sees.
@Nucs

Nucs commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Hold lifted: NumSharp 0.81.0 is in. Six commits move the branch to it and a seventh updates the notes; the description above is updated.

What 0.81.0 fixed for 1.1.0:

  • OpenBLAS = NumPy 2.5.4's (scipy-openblas 0.3.34.106.0): the bit-for-bit target is now Optuna on NumPy 2.5.4, and CI and the package-parity container pin numpy==2.5.4. The 57 BLAS-backed live cases pass; against NumPy 2.4.2, 14 of them fail. 5 of 36 seeded probe studies moved (CMA-ES over mixed parameters, IPOP CMA-ES, NSGA-II with UNDX, xNES, JANUS); the notes list it under Breaking Changes.
  • QA-85 (tied signed zeros, NumSharp's QB43): fixed, and its repro test runs again.
  • Float np.prod in NumPy's sequential order: the hypervolume and fANOVA keep their own loops (no array per call), now pinned to np.prod over 800 random cases.
  • Python's exception classes: 14 sites, NumPyDraws' OverflowError (the at-the-bump item) among them, raise OverflowError, ZeroDivisionError, FileExistsError or FileNotFoundError as Optuna does.
  • Public ConcurrentOrderedDictionary: the vendored concurrent collections are gone (3,550 lines).

What it did not fix:

  • The np.sum(axis=-1) short-row and out= costs are still OpenBugs in NumSharp 0.81.0, so TPE's pairwise sum (QA-84) keeps its 27-66 %. Re-measured on 0.81.0, TPE is still x1.5->x3.4 faster than 1.0.0.

Found on the way: NumPy 2.5 removed np.row_stack, which pymoo 0.6.1.3, the AGE-MOEA-2 oracle, still calls. The interop harness restores the alias, test-only; pymoo 0.6.2 changed AGE-MOEA-2's arithmetic, so 0.6.1.3 stays the oracle.

Verified locally: unit tests 2,310 on net8.0 and net10.0; live-Optuna interop 751 passed, 0 failed, 6 skipped against NumPy 2.5.4 (5 Redis cases without a local server, which CI runs; 1 by design); 20 packages publishable; ApiCompat clean against 1.0.0. Next: this PR's checks and a full manual run of the release matrix (3 OSes, both frameworks, Slow) on release/1.1.0; then fast-forward main and tag v1.1.0.

…tes written since 1.0.0 pinned to x64 for macOS, crash dumps for Windows test hosts

The release-matrix rehearsal of 8c31fb6 (workflow_dispatch run 43545001631, 2026-11-14: 3 OSes, both TFMs, the Slow
category) passed everything but its two macOS jobs; PR #6's own run of the same commit (43544913859) passed
everything but one Windows test host, which crashed. This commit answers both.

1. macOS

The rehearsal ran macOS for the first time since the v1.0.0 tag run. Four bit-exact gates written after that run failed there; none of them had run on macOS
before, and all four compare last bits that differ on arm64 for reasons the suite already documents and pins to x64
elsewhere (PythonSession.BitExactHost; the unit suite's own x64 guards in AgeMoea2OperatorParityTests,
CmaPickleOracleTests, ShapleyImportanceEvaluatorTests, SklearnOracleParityTests and the ES digest tests).

Interop (macos-26, net8.0: 248 passed, 3 failed, 506 skipped):
- HypervolumeParityTests.ComputeHypervolume_MatchesOptunaBitForBit_TwoToFiveObjectives (c600736, 2026-11-11):
  54 of 240 volumes a last bit off, e.g. trial 9, m=3 n=37: 3ff02f8a9b00e63c against Optuna's 3ff02f8a9b00e63b.
- HypervolumeParityTests.SolveHssp_SelectsOptunasIndices_ThreeAndFourObjectives (c600736): 5 of 240 selections differ
  (trial 66, m=3 n=17 k=12: [...,13,12,16,14] against Optuna's [...,14,10,15,11]), the zero-contribution ties.
  Both compare NumSharp's OpenBLAS ddot/dgemv with NumPy's @ / np.dot. That is bit-exact only where NumPy's wheel
  bundles the same scipy-openblas, i.e. x64; the macOS runner installs numpy-2.5.4-cp312-cp312-macosx_14_0_arm64.whl,
  NumPy's macOS 14+ wheel, which is built against Apple's Accelerate.
- CmaEsSamplerParityTests.SeparableCma_LongRun_FullState_IsBitIdentical (99e9e73, 2026-11-08): trial 11's pickled
  optimizer state differs. Every other full-state CMA-ES test here was already EnsureBitExactHostOrInconclusive; the
  short SeparableCma_FullState_IsBitIdentical still runs everywhere (it passed on macOS), the 30-generation run meets
  the documented arm64 last-bit difference.
  All three now call PythonSession.EnsureBitExactHostOrInconclusive() instead of EnsureOrInconclusive().

Unit (macos-26, net8.0: 2,284 passed, 1 failed, 20 skipped):
- SamplerParityBugTests.GpSampler_SteppedFloat_SuggestsNumpyGridValues (QA-49; written 2026-11-03 as [OpenBugs],
  unmarked by the 2026-11-11 fix, so it never ran on macOS before): the fourth trial is 0.9999999999999998 where
  Optuna on x64 (and the port on Windows and Linux) gives 1.0; every other trial matches and picks the same grid
  point. The grid itself is NumPy's on macOS too (GpSearchSpace_SteppedFloatGrid_UsesNumpyArangeStep passed in the
  same run), so the GP's last-bit arithmetic on arm64 reached that point by another route: the value equals the
  rounding path's low + k*step (0.9999999999999999) after the normalize / unnormalize round trip. The test now goes
  Inconclusive off x64, with the reason.

Each test's docs say why it is pinned and what macOS showed. No product code changes: these gates hold on the hosts
the parity claims are made for (x64, where NumPy and NumSharp share one OpenBLAS and one libm family), and macOS keeps
running the tolerance-based parity cases. On x64 all four still run: the three interop gates pass locally against
numpy 2.5.4 (3/3, net10.0).

2. The Windows test-host crash

PR #6's test (windows-2025) job, net10.0, aborted after 1 min 8 s: "The active test run was aborted. Reason: Test
host process crashed", with nothing after "crashed" (no managed exception, no stack overflow message). Its TRX
(artifact test-results-windows-2025) holds 915 results; the last ones, GridSamplerTests at 02:14:48.47, come right
before the JANUS suites in discovery order, and the host died about 0.46 s later, so one of the next few tests was
running (VSTest batches results, so the last few can be lost). No earlier run in SciSharp's last 60 Build and
Release runs crashed a test host. It did not come back:
- the rehearsal's test (windows-2025) job ran both TFMs on the same commit, green;
- locally (i9-13900K, Windows 11, OPENBLAS_CORETYPE=Haswell as in CI, DOTNET_GCgen0size=0x40000 so collections and
  NumSharp finalizers run constantly, --blame-crash): the JANUS / GridSampler / MOEA/D / GP-primitives window 10 of
  10 rounds green (71 tests each), the whole CI-filtered net10.0 suite 3 of 3 rounds green (2,310 passed, 1 skipped).
So the Windows leg of the unit test step now runs with --blame-crash --blame-crash-dump-type mini, and the upload
keeps TestResults' *.dmp and Sequence_*.xml beside the TRX files: a recurrence leaves the running test's name and a
dump. Linux and macOS keep the plain run (the flag was only tried on Windows). The bash array form expands to nothing
off Windows, checked under CI's `bash -e -o pipefail`; the SDK's blame collector names its file Sequence_<id>.xml.
.claude/CLAUDE.md's test-job bullet says where the dump lands.

Also: the notes' "of the release's 140" becomes 141 (v1.0.0..this commit); the PR body's commit count follows.
@Nucs

Nucs commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Release matrix rehearsed - a manual run of Build and Release on release/1.1.0 at 8c31fb6 (3 OSes, both frameworks, the Slow category, no release jobs): green except macOS.

  • macOS: four bit-exact gates written after 1.0.0's tag run, the last run with macOS, failed on their first macOS run. None is a port error, and the new commit pins all four to x64 like their siblings (Inconclusive elsewhere):
    • hypervolume (54 of 240 volumes a last bit off) and HSSP (5 of 240 selections): NumPy's macOS 14+ arm64 wheel computes @ / np.dot with Apple's Accelerate, not the scipy-openblas NumSharp bundles;
    • a 30-generation separable CMA-ES state (the short run still passes on macOS);
    • QA-49's GP trial: 0.9999999999999998 where x64 Optuna gives 1.0, the same grid point.
  • Windows: this PR's run had one test-host crash (net10.0, near the JANUS suites, nothing logged). The rehearsal's Windows legs (both frameworks) and 13 local stress runs (constant GCs, Haswell OpenBLAS kernels as in CI) did not reproduce it. From now on a Windows test host that crashes leaves a mini dump and the test sequence in the test-results-windows-2025 artifact.
  • Slow: 43 BoTorch interop cases, and 13 slow unit tests on each framework, passed.

A second rehearsal runs on the new head.

Nucs added 2 commits November 14, 2026 05:28
… fault there was PR #6's Windows test-host crash

PR #6's test (windows-2025) job crashed its test host on 8c31fb6 and on 7d73f5a, both times in
JanusDeSamplerTests.Sampler_ScalarMode_Runs (VSTest blame, added by 7d73f5a), with nothing logged and no dump, while
both release-matrix rehearsals and every local run passed. Three diagnostic rounds on a throwaway branch
(ci/win-crash-diag, each leg a fresh windows-2025 VM) found it:

- Round 1, run 43548926143 (the unit suite as the PR runs it, four legs): crash on AMD EPYC 9V74 (Zen 4) and 9V45
  (Zen 5), also with the JANUS-DE suite run alone; pass on AMD EPYC 7763 (Zen 3) and an Intel Xeon 8370C. No WER
  dump, no Application Error event; the host's own log ends at "Executing test Sampler_ScalarMode_Runs".
- Round 2, run 43549856491 (the scenario as a console program, six legs, seven variants each, full exit codes): on
  the three Zen 4 legs, NumSharp 0.81.0's OpenBLAS 0.3.34.106.0 with OPENBLAS_CORETYPE=Haswell exits 0xC0000005
  (access violation) between trials 150 and 200, on net10.0 and net8.0, with OPENBLAS_NUM_THREADS=1, and with .NET's
  AVX-512 / wide vectors off. The same build PASSES with OpenBLAS's own kernel choice, and NumSharp 0.80.0's OpenBLAS
  0.3.31.22.0 (NUMSHARP_OPENBLAS_VERSION override) passes WITH the pin. Zen 3 x2 and an Intel Xeon 8573C pass every
  variant. So: OpenBLAS 0.3.34's Haswell kernels, forced on an AMD CPU with AVX-512, fault; CI forces them.
- Round 3, run 43550350376 (the whole unit suite, both TFMs, OPENBLAS_CORETYPE unset, six legs): no crash anywhere.
  Zen 3 x4 green. Zen 4 and Zen 5 fail the same six Haswell-captured oracles on both TFMs: CmaPickleOracleTests
  (cma, cma_lr_adapt, cma_30d), EvolutionStrategyExtraCandidateTests' xNES digest, AgeMoea2OperatorParityTests'
  Normalize_IsBitExact and Do_Age2_IsBitExact. (One Zen 3 leg also failed the 16-worker REMBO
  no-duplicate-evaluation test once on net8.0; it passed on every other leg and run, unrelated to the pin.)

The product is not affected in its default configuration: a user's OpenBLAS picks its own kernels, the variant that
passed on the same machines. Only a process that forces OPENBLAS_CORETYPE=Haswell on Zen 4/5 meets the fault.

What changes:
- .github/scripts/openblas_coretype.sh (new), sourced by all five x64 test steps that pinned Haswell inline (unit,
  slow unit, satellites, interop main, interop slow): it exports OPENBLAS_CORETYPE=Haswell as before, except on an AMD
  CPU with AVX-512 (Windows: PowerShell 7 asks .NET for the vendor and Avx512F; Linux: /proc/cpuinfo; Git Bash's
  emulated /proc/cpuinfo is deliberately not trusted for the flag), where it leaves OpenBLAS to choose, and prints
  which and why. Live NumPy parity still holds there: the interop suite's NumPy and NumSharp load the same OpenBLAS
  build, which makes the same choice. Checked locally under CI's `bash -e -o pipefail` and `-uo pipefail`, on x64
  Intel (pins Haswell) and with RUNNER_ARCH=ARM64 (does nothing).
- The three Haswell-captured oracle guards also require OpenBlasEngine.CoreName == "Haswell" (openblas_get_corename),
  and report Inconclusive with the kernel name otherwise: AgeMoea2OperatorParityTests.RequireBitExactLinalgHost,
  CmaPickleOracleTests.RequireOracleHost (every variant), EvolutionStrategyExtraCandidateTests' xNES row (OpenAI-ES
  and SNES never reach OpenBLAS and held). Locally: OPENBLAS_CORETYPE=Haswell 52/52 of those classes pass;
  OPENBLAS_CORETYPE=Sandybridge 37 pass and 15 report Inconclusive, none fail.
- The Windows --blame-crash comment and .claude/CLAUDE.md say what the blame data led to; CLAUDE.md gains the pin
  rule (a new Haswell-captured oracle needs the same guard).

Upstream: worth an OpenBLAS issue (0.3.34, DYNAMIC_ARCH, OPENBLAS_CORETYPE=Haswell on Zen 4/5 -> access violation),
with the console repro of the diagnostic branch; NumSharp 0.81.0 ships that build.
…d PR #6's Windows test-host crash (4cbbef1)

Testing & Tooling gains one line for 4cbbef1: CI no longer forces OpenBLAS's Haswell kernels on AMD CPUs with
AVX-512, because NumSharp 0.81.0's OpenBLAS 0.3.34 running them there faults (an access violation in JANUS-DE's
quadratic chart, found on Zen 4 and Zen 5 runners by the diagnostic runs 43548926143, 43549856491 and 43550350376);
OpenBLAS's own kernel choice and 0.3.31 pass, and the Haswell-captured oracles report Inconclusive on other kernels.
The line also tells a reader who sets OPENBLAS_CORETYPE=Haswell on such a CPU that they meet the same fault; a
default-configured process picks its own kernels and does not.

The Overview's count line, recounted by count_notes.py: 51 cited commits (was 50) of the release's 143
(v1.0.0..this commit; was 141, +4cbbef1 +this one). Parity fixes (48), measured speedups (18) and behavior changes
(14) are unchanged: Testing & Tooling is not counted in them. 7d73f5a (the macOS x64 pins, test-only) stays uncited,
as test-only commits do.
@Nucs

Nucs commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

The Windows test-host crash: traced and fixed (4cbbef1, notes 07f8232).

It is OpenBLAS 0.3.34 (NumSharp 0.81.0's bundle) running the Haswell kernels CI forces with OPENBLAS_CORETYPE=Haswell, on AMD Zen 4/5 CPUs: an access violation (0xC0000005) inside JANUS-DE's quadratic chart. It took three diagnostic rounds on a throwaway branch, each leg a fresh windows-2025 VM:

CPU Haswell forced, OpenBLAS 0.3.34 OpenBLAS's own kernels Haswell forced, OpenBLAS 0.3.31
AMD EPYC 9V74 (Zen 4) access violation, 4 of 4 VMs: net8.0 and net10.0, 1 or 4 threads, .NET's AVX-512 on or off pass pass
AMD EPYC 9V45 (Zen 5) crash pass -
AMD EPYC 7763 (Zen 3) pass pass pass
Intel Xeon 8370C / 8573C pass pass pass
  • Users are not affected in the default configuration: OpenBLAS picks its own kernels, the variant that passes on the same machines. A process that forces OPENBLAS_CORETYPE=Haswell on Zen 4/5 meets the fault; the notes say so.
  • CI: .github/scripts/openblas_coretype.sh keeps the Haswell pin except on AMD CPUs with AVX-512. Without the pin, the whole unit suite passes there except 6 Haswell-captured oracles (CMA-ES state, xNES digest, AGE-MOEA-2 linalg). Those now check OpenBlasEngine.CoreName == "Haswell" and report Inconclusive on other kernels.
  • Why the earlier rehearsals passed: their Windows jobs landed on Zen 3 or Intel machines. A tag run on a Zen 4/5 runner would have crashed.
  • Upstream: worth an OpenBLAS issue (0.3.34, DYNAMIC_ARCH, forced Haswell on Zen 4/5). The diagnostic branch's console repro reproduces it in seconds.

Runs: 43548926143, 43549856491, 43550350376. A third rehearsal of the release matrix runs on 07f8232.

Nucs added 3 commits November 14, 2026 05:52
…uns forced Haswell fine, and unpinned it broke a live CMA-ES gate

4cbbef1 left OPENBLAS_CORETYPE unset on every AMD CPU with AVX-512. The third release-matrix rehearsal (dispatch
43551908135 on 07f8232) then put its ubuntu Slow interop leg on such a runner and LongRun_40D_FullState_IsBitIdentical
failed: trial 31's stored 40-D CMA-ES state differed from live Optuna's in the same process. Meanwhile the fault
behind 4cbbef1 had only ever been seen on Windows.

A check on throwaway branch ci/linux-haswell-check (run 43553335934, six fresh ubuntu-24.04 VMs, the console repro of
the fault, Haswell forced and unpinned, net10.0 and net8.0) settled it: three AMD EPYC 9V74 (Zen 4) legs ran the
forced Haswell kernels to the end, exit 0 on both TFMs (OpenBLAS's own choice there is SkylakeX); so did EPYC 7763 x2
and a Xeon 8370C. The fault is in OpenBLAS 0.3.34's Windows build only.

So .github/scripts/openblas_coretype.sh applies the exception only when RUNNER_OS is Windows; Linux keeps the Haswell
pin on every CPU, as before 4cbbef1, which keeps the committed oracles and the live byte-for-byte gates (the 40-D
CMA-ES state among them) bit-exact on Linux Zen 4 too. The script's header gives both measurements; the workflow
comments, the three oracle guards' remarks, .claude/CLAUDE.md and the notes line say "Windows on AMD CPUs with
AVX-512". Checked locally under `bash -e -o pipefail`: RUNNER_OS=Windows on this Intel host and RUNNER_OS=Linux pin
Haswell, RUNNER_ARCH=ARM64 does nothing.

The notes' count line becomes "of the release's 144" (v1.0.0..this commit); the cited-commit count (51) is unchanged:
the notes line keeps citing 4cbbef1, which this commit narrows.
…s optimizer repeats evaluated points one trial at a time too, a model bug, not a lost read

The `rembo` row of SamplerParallelReadTests.Optimize_SixteenWorkers_SamplerThatNeverRepeats_EvaluatesNoPointTwice
failed 2 of about 45 CI runs of the 1.1.0 release branch (net8.0), for example:

  rembo evaluated these points more than once:
  x=9.999999999999998|y=-3.6688215204910746|z=0.9715361410124286 x4

The owner's decision (2026-11-14): investigate first, then open an OpenBug and proceed with the release.

Investigation (2026-11-14, scripts kept in the session scratchpad, not in the repo)
-------------------------------------------------------------------------------------

1. Reproduction. The test's exact scenario (RandomEmbeddingSampler(2, seed: 1), 60 trials, nJobs: 16, objective
   (x-2)^2 + (y+1)^2 + z over x, y in [-10, 10], z in [0, 1]) repeated a point in 3 of 80 runs on 32 CPUs and
   0 of 80 pinned to 4 CPUs (.NET 11, the scratchpad lies outside the repo's global.json). Each repeat had one
   latent stamp ("rembo:latent") on every copy, e.g. [-1.414205579096142, 1.4142055790961423]: the center of
   DIRECT's corner cell of the latent box [-sqrt2, sqrt2]^2 after 11 trisections (sqrt2 - (2*sqrt2/3^11)/2).

2. Not a lost read. A storage decorator (an InMemoryStorage subclass) numbered every read and write under one
   lock: ALL = the per-trial history snapshot, RUN = the running-trial read (the pending set), STAMP = a latent
   stamp, TELL = a final state. For every stamp, the proposing thread's last ALL and RUN reads give exactly the
   history and the pending set its proposal used. Over 220 runs and 13,200 proposals (.NET 10): no proposal
   missed an earlier proposal's latent stamp. GetOrComputeLatent reads the running trials first (fresh,
   useCache: false) and the history right after (the first useCache read of the trial), all inside `_gate`, and
   every stamp is written before `_gate` is released. Every repeated point was visible to the proposal that
   repeated it, as a finished observation (most) or as a running trial's pessimistic lie (one), and the
   acquisition's optimizers still returned the same DIRECT cell. The repeats seen under 16 workers were exact
   latent repeats at corners of the latent box at poor values (about 72 for the CI point; the minimum is 0).

   One concurrency imprecision showed on the way: in about 1.4 % of proposals (186 of 13,200) a trial told
   between the RUN and the ALL read is in both, so FitWithPendingLies holds its real value and its lie at one
   input. No repeat coincided with it. Left as is; noted in the OpenBugs test.

3. Not concurrency at all. The same objective and configuration with ONE worker (nJobs: 1), seeds 1..120:
   82 runs evaluated a parameter point twice; counting exact latent repeats only, 67 runs repeated 149 latent
   points (138 at values <= 2 near the best point found, 11 from 2.05 up, among them three corners of the
   latent box in seed 48 at 67 and 122). Seed 1 repeats nothing alone, which is why the 16-worker test passed
   about 95 % of the time: an interleaving acts as another seed (as the test's xNES remark already says).

4. Two routes, both with one worker (seed 95, 60 trials):
   - Clipping. x = p_X(A*y) clips onto the parameter box, so a region of the latent box lands on one corner:
     (10, 10, 0) at #7, #18, #34 and (-10, -10, 1) at #19, #45, each from a different latent point. The default
     low-dimensional kernel measures distances between latent points and sees them as far apart - the waste
     Wang et al. (2016, sec. 3.2) name as that kernel's weakness ("two points far apart in this region may be
     projected via p_X to nearby points on the boundary of X"), which their high-dimensional kernel avoids.
   - The incumbent. The GP interpolates (the Cholesky never needs more than its 1e-6 nugget; the posterior std
     at an evaluated point is 2e-4..5e-4 of the profiled amplitude from trial 29 on), so expected improvement at
     the incumbent is that std times phi(0): small but not zero. Once the GP is confident across the box,
     DIRECT and CMA-ES find nothing larger and return the incumbent's own coordinates: #35's latent point again
     at #36-#37, #40's at #41-#44, #46's at #47.
   (Ruled out on the way: jitter escalation - it stays at 1e-6; log-EI underflow - Acqf.StandardLogEi has the
   erfcx branch below z = -25; a spurious per-study state reset - TrialAnchor compares by id and start time.)

What changed
------------

- SamplerParallelReadTests: the `rembo` DataRow leaves the shared 16-worker test (the other 14 rows stay), and
  the new Optimize_SixteenWorkers_Rembo_EvaluatesNoPointTwice, marked [OpenBugs], runs that same scenario
  through the shared method (whose 60-trial condition for "rembo" stays live). Its remarks carry the CI
  failures, the local rates, the trace and the corner cells. The class remarks say REMBO is an open
  exception, kept out of RepeatsFinishedPoints because it is a bug, not a design choice; its volley rows
  (ParallelReads_SharedSampler_HandsNoNewPointToTwoTrials), which have never repeated, stay asserted.
- RandomEmbeddingSamplerTests: new [OpenBugs] Optimize_OneWorker_EvaluatesNoPointTwice, the deterministic
  reproduction (seed 95, one worker), documenting both routes and a fix direction (keep the acquisition's
  maximizer off evaluated and running points, compared in the projected parameters). It fails today on
  net8.0 and net10.0 with "x=9.999999999999998|y=9.999999999999998|z=0 at #7,#18,#34" first.
- docs/THREADING.md: the whole-run and history-repeat passages of sec. 6.3 and the test-map row name REMBO as the
  open exception, with the tests that pin it.

No source change: RandomEmbeddingSampler and RemboGpProposer are untouched, so every seeded REMBO result is
the same.

Verification
------------

- dotnet test test/OptunaSharp.Tests -c Release, filter RandomEmbeddingSampler|SamplerParallelReadTests:
  net8.0 92 of 93 pass, net10.0 91 of 93. Optimize_OneWorker_EvaluatesNoPointTwice fails on both (intended);
  Optimize_SixteenWorkers_Rembo_EvaluatesNoPointTwice failed on net10.0 in that run and passed on net8.0
  (flaky, intended - now outside CI). Every other test of the two classes passes on both.
- The CI filter (TestCategory!=OpenBugs&!=HighMemory&!=RequiresPython&!=Slow) lists neither new test.
…r no-repeat check is [OpenBugs] beside a one-worker reproduction (6bed3d7)

The owner's call (2026-11-14) on the REMBO flake that had failed 2 of about 45 CI runs: investigate first, then open
an OpenBug and proceed with the release. 6bed3d7 did both; this commit tells the release's readers.

- Testing & Tooling gains one line: REMBO's 16-worker no-repeat check is now [OpenBugs], beside a one-worker
  reproduction; RandomEmbeddingSampler evaluates a point twice one trial at a time too (82 of 120 seeds of the
  check's objective), from latent points the projection clips onto one box corner and by proposing a confident
  GP's incumbent again; a trace of 13,200 concurrent proposals found no lost read - 6bed3d7.
- The Overview's count line, recounted: 52 cited commits (was 51; 6bed3d7 is new) of the release's 146
  (v1.0.0..this commit, which counts itself; was 144), 2,309 unit tests on each framework (was 2,310: the rembo row left the CI set and both new tests are
  [OpenBugs]; measured with the CI filter on both TFMs, 2,309 passed and 1 skipped, ARealSigterm_..., a POSIX-signal
  test that self-skips on Windows, as in the earlier count). The other figures are untouched by this change:
  18 measured speedups and 14 behavior changes recount the same with count_notes.py; the 48 parity fixes keep the
  item rule 8c31fb6 used (39 QA ids + 9 items without one, where the script's bullet rule gives 5), since neither
  commit touches the Parity & Fixes section.
@Nucs
Nucs merged commit b3316c4 into main Nov 14, 2026
58 of 60 checks passed

This branch was successfully deployed

1 active and 1 inactive deployments
nuget — b3316c4f Deployed Nov 14, 2026 by Nucs via publish-nuget #35
github-pages — b3316c4f Deployed Nov 14, 2026 by Nucs via deploy #14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant