fix(cuda): persist and restore native model package profiles

[Qiiks]

Summary

Fix the owned-CUDA model.load path so persisted models retain their owned profile and package locators across restart. Previously, a dynamically loaded CUDA model could persist without the required owned family and prevent module startup on reload.

This branch is stacked on PR #18 (head 6ee5494); the catalog-specific commit is 65bdbdd. Please land after #18 or review that commit independently.

Verification

• Targeted catalog regression passes: actual load construction, persistence/restart normalization, and package assembly.
• Related owned-CUDA module tests: 2 passed, 0 failed.
• Module formatting check passed.
• Release module was exercised under SubC on Windows: native Qwen3 load completed, certification against 64 references reached mean cosine 0.99999785 and rank overlap 1.0.
• A subsequent normal native embedding request returned 1024 dimensions with owned-cuda provenance.

Consumer migration and OMP lifecycle verification are separate and are not claimed by this PR.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Fixes owned-CUDA model.load and restart handling so dynamically loaded models no longer lose the metadata needed to start after reload. CUDA certification evidence now probes the model's worker in a blocking task, keeping the async path responsive.

Persistence

• Rebuilds stored owned-CUDA rows with family, dtype, execution, and config/extra locators through the shared profile builder.
• Reuses owned model package assembly for CUDA rows and preserves config locators during catalog normalization.

Hardware and Packaging

• Probes driver API and compute capability through the selected CUDA worker when complete environment overrides are absent, caching results per worker path.
• Preserves the last 4096 bytes of probe stderr in refusal evidence without fabricating hardware readings.
• Packages and verifies Windows CUDA sidecars with component licenses and SHA-256 manifests.
• Replaces the off-PATH failure gate with sidecar hash, DLL resolution, and hardware-probe checks.
• Windows worker builds require CUDA 13, and missing CUDA libraries return a Rust error instead of a loader exception.

^{Written for commit 95f0a47. Summary will update on new commits.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[cubic-dev-ai]

All reported issues were addressed across 9 files

_{You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Qiiks]

Both P2s addressed in 95f0a47: certification evidence now resolves the model's own worker path from the catalog (spec.worker_bin) instead of the default probe, and the probe runs in tokio::task::spawn_blocking so a hung worker cannot block an executor thread. Added regression cuda_certification_evidence_uses_model_worker_without_blocking asserting (1) the model-specific worker is the probe source (its driver_api value lands in evidence.observed) and (2) the probe stays pending on a current-thread executor while a slow probe runs, proving the async path is not blocked. Catalog regression and clippy -D warnings pass locally.

[synapse-alfonso]

Confirmed the defect at source on master before reading the fix, and it is worse than the description says: this is boot-blocking for the whole module, not for the one model. execute_model_load_job builds the owned profile only for owned-metal and persists a dynamically loaded owned-CUDA row with owned_family = None; on the next boot refresh_catalog walks every catalog row through normalize_catalog_model, owned_cuda_catalog_config refuses family=None, and the ? at lib.rs:2289 propagates — so one model.load on a CUDA engine followed by a restart leaves the daemon unable to start until someone edits the store by hand. Any operator on the Windows or Linux owned-CUDA lane who has used model.load is one restart away from it. Thank you for finding it.

The fix reads correctly: the owned profile is built through one helper for both owned engines, the CUDA arm carries family, dtype, execution, and the config/extra locators, and normalization preserves the locators instead of dropping them. Moving owned_cuda_evidence onto spawn_blocking is right too — the floor probe spawns and waits on a subprocess, which was doing that inside the async path.

Two things before merge:

1. The base predates the fix(engine-cuda): bound Qwen3 shape-plan cache with a 2-entry LRU #16/floor: probe driver API and compute capability via a short-lived worker subprocess #17/Windows owned-CUDA worker: delay-loaded cuBLASLt + sidecar packaging gate #18 landing (your git diff against master currently shows the verification report and my preflight change as removed, which is base staleness, not your edits). A rebase on current master, or leaving it to me to merge — I am fine either way, but say which.
2. I am running the actual failure mode on a rented 4090 rather than trusting the unit test alone: model.load on master → restart → capture the exact boot refusal, then the same on this branch → boot, catalog row present, embed succeeds with no new load, certification probe completes through the blocking path with wall time recorded. The unit test proves normalization; the restart proves the bug. Results go into docs/audits/pr15-cuda-verification.md and the merge follows them.

Cargo.lock is untouched in this branch, which is exactly right.

[Qiiks]

Please handle the current-master integration on your merge branch; I will not rebase this head while your hardware verification is in progress. Thank you for testing the actual restart failure. The native AFT acceptance check also found a separate discovery omission: models.list omits recommended_batch for owned-cuda, which AFT correctly rejects. A targeted regression reproduced it, and extending the existing module batch-policy arm to CUDA passes locally. I am keeping that follow-up separate from your restart verification.

[Qiiks]

The separate batch-advice fix is now commit Qiiks/synapse@920d27f on fix/owned-cuda-batch-advice (one producer arm plus regression). Feel free to cherry-pick it during integration; I have left this PR head unchanged. Real consumer proof after deploying that module and certifying: AFT indexed 4 fixture entries at 1024 dimensions; its search made 1 live native embed call and returned refund_cancelled_purchase with semantic_score 0.58378899, no lexical-only fallback. Before the fix, AFT failed discovery with no usable recommended_batch rows/token_budget. Published values are the module-enforced limits: rows=8, token_budget=3072. Targeted regression, fmt, strict Clippy, and release build passed.

[synapse-alfonso]

Landed at ecfb092 through the CI train, with the report in docs/audits/pr15-cuda-verification.md.

The restart proof on a rented RTX 4090 ($0.20) holds on both trees. Master, after one owned-CUDA model.load and a clean restart, died with exactly the line I traced: synapse boot failed after HELLO_ACK: config: owned-cuda catalog entry is missing family. This branch booted with the persisted profile and served a 1,024-dimension owned-cuda embed with no new load. The certification evidence probe completed in 4.0 s through the blocking path with cuda evidence present, and both unit tests passed.

Two things the run recorded that are worth knowing, neither a defect in the PR:

• On master the CUDA model.load also failed artifact validation before reaching ready, so the pre-restart control embed was impossible there — the row was persisted malformed anyway, which is what makes the next boot refuse.
• On this branch the first post-restart embed was refused not_certified: persisting a new engine identity rotates the machine-profile hash, and serving fails closed until a probe recertifies. That is the designed behaviour; the prescribed probe run cleared it. Operators running model.load on a CUDA engine should expect one probe after the first restart.

Thank you — this one would have taken the daemon down for anyone using model.load on the CUDA lane.