Problem
Once #8089's advisory backtest check is live and has a real track record, the same question applies to Loopover: if hosted ORB (per the current ~1-2-months-out roadmap) ever uses a backtest result to justify an automated decision to a customer, "trust the maintainer's report" is a weaker claim than "here's a cryptographically-attested execution anyone can verify." This issue is tracking/research only — it does not commit to building anything yet.
Area
Research spike. No code changes are expected as the primary deliverable (a small proof-of-concept is acceptable if genuinely cheap, but is not required to close this issue).
Proposal
Three ordered research phases — each has a real, independently-useful answer, and a "no" at any phase is a legitimate reason to stop rather than proceed to the next:
Phase A — is TEE hardware attestation even the right mechanism, or does a cheaper alternative reach the same trust goal? A "reproducible backtest" approach — publish the corpus snapshot's checksum (#8084's BacktestCorpusManifest.checksum), the exact freeze point, and the script version, and let a skeptical third party independently re-run scoreBacktest/compareBacktestScores themselves and compare checksums — proves the SAME thing (the reported score is real, not fabricated) without any new infrastructure, for corpora that are safe to publish. Decide whether this covers the actual trust gap before investing in TEE-specific research; TEE is only necessary for cases this can't cover (e.g. a private/held-out corpus, per #8087, that can't be published for the reproducibility check to work against).
Phase B — if TEE is still needed (e.g. specifically for the held-out-corpus case Phase A can't cover): what's actually viable on this repo's real deploy targets? Survey Cloudflare Workers/Containers TEE or confidential-compute options. #7180 already contains a research comment surveying Cloudflare Containers vs. microVM isolation tiers for a different purpose (tenant execution sandboxing) — check whether that research transfers before redoing it. If Cloudflare has no viable option, the alternative is running the attested piece on a different provider entirely (a real infra-split cost to weigh, not just a technical detail).
Phase C — threat model check. Who is the actual verifier, and what are they trying to verify — that the ground-truth corpus wasn't tampered with, that the score computation ran the real published code, or that the reported number wasn't just made up? Different threats need different mechanisms (checksums solve tampering; attestation solves "did the real code actually run"; neither solves "do I trust the maintainer's infrastructure at all," which Phase A's own reproducibility check is the closest available answer to). Name which threat this is actually defending against before recommending a mechanism for it.
Produce a short written recommendation after all three phases: worth pursuing once hosted ORB ships; worth pursuing now; the Phase-A reproducibility approach is sufficient and TEE isn't needed; or not worth pursuing at all given the real threat model. Any of these four is an acceptable, complete outcome for this issue.
Deliverables
- A written recommendation (issue comment or a linked doc) covering all three phases and the final call.
Resources
Boundaries
No production code commitment. No hosted-product architecture decision made here beyond the written recommendation.
maintainer-only — research/strategy, not a build task for contributors.
Problem
Once #8089's advisory backtest check is live and has a real track record, the same question applies to Loopover: if hosted ORB (per the current ~1-2-months-out roadmap) ever uses a backtest result to justify an automated decision to a customer, "trust the maintainer's report" is a weaker claim than "here's a cryptographically-attested execution anyone can verify." This issue is tracking/research only — it does not commit to building anything yet.
Area
Research spike. No code changes are expected as the primary deliverable (a small proof-of-concept is acceptable if genuinely cheap, but is not required to close this issue).
Proposal
Three ordered research phases — each has a real, independently-useful answer, and a "no" at any phase is a legitimate reason to stop rather than proceed to the next:
Phase A — is TEE hardware attestation even the right mechanism, or does a cheaper alternative reach the same trust goal? A "reproducible backtest" approach — publish the corpus snapshot's checksum (#8084's
BacktestCorpusManifest.checksum), the exact freeze point, and the script version, and let a skeptical third party independently re-runscoreBacktest/compareBacktestScoresthemselves and compare checksums — proves the SAME thing (the reported score is real, not fabricated) without any new infrastructure, for corpora that are safe to publish. Decide whether this covers the actual trust gap before investing in TEE-specific research; TEE is only necessary for cases this can't cover (e.g. a private/held-out corpus, per #8087, that can't be published for the reproducibility check to work against).Phase B — if TEE is still needed (e.g. specifically for the held-out-corpus case Phase A can't cover): what's actually viable on this repo's real deploy targets? Survey Cloudflare Workers/Containers TEE or confidential-compute options. #7180 already contains a research comment surveying Cloudflare Containers vs. microVM isolation tiers for a different purpose (tenant execution sandboxing) — check whether that research transfers before redoing it. If Cloudflare has no viable option, the alternative is running the attested piece on a different provider entirely (a real infra-split cost to weigh, not just a technical detail).
Phase C — threat model check. Who is the actual verifier, and what are they trying to verify — that the ground-truth corpus wasn't tampered with, that the score computation ran the real published code, or that the reported number wasn't just made up? Different threats need different mechanisms (checksums solve tampering; attestation solves "did the real code actually run"; neither solves "do I trust the maintainer's infrastructure at all," which Phase A's own reproducibility check is the closest available answer to). Name which threat this is actually defending against before recommending a mechanism for it.
Produce a short written recommendation after all three phases: worth pursuing once hosted ORB ships; worth pursuing now; the Phase-A reproducibility approach is sufficient and TEE isn't needed; or not worth pursuing at all given the real threat model. Any of these four is an acceptable, complete outcome for this issue.
Deliverables
Resources
BacktestCorpusManifest.checksum, the reproducibility-check building block Phase A would reuse), calibration: deterministic seeded held-out/visible split of the backtest corpus #8087 (the held-out corpus that's the concrete case Phase A can't cover), Design + wire an advisory-only backtest check for PRs touching ORB/AMS deterministic rules #8089Boundaries
No production code commitment. No hosted-product architecture decision made here beyond the written recommendation.
maintainer-only — research/strategy, not a build task for contributors.