Skip to content

Sign articles and routed media with the Encypher provenance module - #1448

Draft
erik-sv wants to merge 15 commits into
IABTechLab:feature/55-scripts-in-cratesfrom
erik-sv:encypher/v2-provenance-module
Draft

erik-sv wants to merge 15 commits into
IABTechLab:feature/55-scripts-in-cratesfrom
erik-sv:encypher/v2-provenance-module

Conversation

@erik-sv

@erik-sv erik-sv commented Oct 24, 2026 •

Copy link
Copy Markdown

Summary

  • Sign eligible article text, routed binary responses and source-embedded HTML/CSS media through provenance.encypher, while keeping the core vendor-neutral.
  • Preserve exact source provenance and existing privacy, integrity and cache guards. Opt-in rehosting brings eligible references through the edge; it does not intercept every browser request.
  • Add an opt-in prepared large-media lane that hashes a version-pinned source in bounded chunks instead of uploading the complete asset for signing. Refuse that lane when the adapter cannot enforce a total deadline.

This contribution remains based on row 55 of the module-architecture stack tracked in #777: #1423, feature/55-scripts-in-crates at 5be114773345a03e207c1da2d3b163a23510985f. It changes no row of that stack and does not target the older module layout on main.

Changes

Area Change
crates/provenance/encypher Article signing, source data URI/SVG signing, exact-source preservation, bounded per-document work, shared-policy and per-item failure handling, prepared signing and configuration validation.
Core media signing and rehosting Routed binary signing; explicit rehosting with privacy/integrity guards; source-aware CSS relocation; bounded prepared large-media operations, including conditional revalidation.
Core streaming and middleware Ordered asynchronous completion, origin privacy/status propagation, source protection and vendor-neutral media-signing interfaces.
Adapters Runtime timer/KV and bounded response/body support across the supported adapters. Fastly's large-media lane remains closed without a whole-operation timer.
Operator docs and design Configuration defaults/ranges, upgrade behavior, supported selector grammar, partial-success semantics, coverage limits and independent review log.

Existing text-only module deployments also activate bounded media signing, subject to the documented 5 MB asset cap and eight uncached calls per document. There is no separate bounded-media disable switch. The large-media lane remains opt-in and off by default.

Shared policy refusal preserves the document unchanged. After admission, an individual article or embedded-media failure preserves that item's source while independently successful items may remain signed. Source bytes with existing provenance are protected from rewriting.

Closes

Closes #1456.

Spec

docs/superpowers/specs/2026-10-24-encypher-provenance-module-design.md

Two independent review rounds are recorded there. Round 1 found deadline, selector-admission and operator-contract defects. Round 2 found the corresponding missing deadline/no-timer guard on origin 304 revalidation. The failing regressions were reproduced before repair; the second reviewer re-inspected the repairs and reported no remaining findings. Review is separate from execution evidence below.

Test plan

All 12 gates required by this branch's AGENTS.md passed on 2026-10-27 after the source repairs: rustfmt, all eight target-matched clippy aliases, all five adapter test aliases, cross-adapter parity, JS build/tests/formatting, docs/root Markdown formatting, template-build digest tests/lint, WASM clock lint, shellcheck, docs-proposal checks and the core platform-name lint. The final evidence-only spec edit also passed Prettier.

  • cargo test-fastly and cargo test-axum, plus the other required adapter test aliases.
  • cargo clippy-fastly and cargo clippy-axum, plus the other six required clippy aliases.
  • cargo fmt --all -- --check.
  • JS build, Vitest and formatting checks.
  • Docs and root Markdown formatting checks.
  • Remaining canonical repository gates listed above.
  • Rebuilt Axum fixture, real signing API, PostgreSQL/Redis, six desktop/mobile Chromium flows and independent public-SDK verification of served bytes.
  • Manual testing through fastly compute serve for this media expansion.
  • Deployed Image Optimizer transformation/signing verification.
  • Public retrieval of the emitted detached-manifest URLs.

Real-runtime evidence at the current implementation

The latest run verified 582 of 659 media observations. These are observations, not 659 distinct assets:

  • 148/148 edge-served browser network observations.
  • 107/107 sequential network observations.
  • 138/138 source-inline SVG observations.
  • 186/186 data URI observations.
  • Two exact-source preservation observations and one generated-source observation.
  • All 490 recorded API requests returned 200. All six browser flows had zero broken images.

The whole-page coverage assertion still exits nonzero: 77 direct third-party response observations remained unsigned. Script-created SVGs are also outside this coverage claim. No exclusions or assertion changes were used to turn that result into a pass.

Detached manifests in this local run were retrieved through an explicitly declared HTTPS backchannel with the emitted canonical Host authority. This proves the local handler and signed bytes, not public URL reachability. The native signer used the canonical public verifier v1.8.0; the local API Python-extension diagnostics still reported 1.7.1, so this run is not presented as proof of a uniformly rebuilt deployment.

Checklist

  • Feature spec linked, ending in two independent review rounds.
  • Canonical branch gates passed; core remains vendor-neutral.
  • Behavioral regressions cover the repaired deadline, conditional-revalidation and selector boundaries.
  • Operator docs distinguish implemented routed coverage from whole-page and deployment claims.
  • No production credentials or customer fixtures are included in this contribution.

The API/release and deployed runtime are separate rollout dependencies. This PR does not claim that production or a publisher deployment has been updated.

A stream processor may implement StreamProcessor::async_finish over the
bytes its last process_chunk call returned. StreamProcessor::finish ends
a processor once; a processor made of others ends each in order, so each
is handed what the one before it finished with. Every publisher path that
can await now ends its processor this way: the no-hold pipeline, both
auction-hold loops, the serve pass on a shared template and the lazy
stream. A body read whole takes the same chunk pipeline, and the
synchronous stream_publisher_body path is removed.

A middleware is also told the response status and whether the origin
marked the response as one reader's, read before core writes its own
cache policy.
The Encypher signing integration becomes a serve middleware in
crates/provenance/encypher, selected by [provenance] module = "encypher"
and placed by a [[serve]] entry. Its preparation applies the request gates
with no I/O, and its stream processor signs the article in the
asynchronous finish core now awaits. The worker token is a module secret
setting. The enabled flag and path_prefixes give way to the selection and
the entry's path, and the module is offered on every adapter and the CLI.
…arrow the module's data claims

Adds a regression test that the parser-confirmed auction hold injects bids
into a document a middleware processor held and finished asynchronously.
States that the module skips pages the origin marks private but cannot
detect other personalized pages, so a deployment places it on public
article routes. Applies rustfmt and the docs formatter.
- Bound HTML scanning and canonical text extraction; one metadata pass
  saves the title, head close and manifest presence, and rendering
  allocates the signed output once at its exact size.
- Stop on malformed or absent head elements and empty manifest URLs
  without changing the page.
- Enforce signed-output and API-response limits incrementally in all
  four adapters, and document the Axum KV persistence behaviour.
- Fail open: any signing failure, timeout or private response passes the
  original page through unchanged.

Full repository checks pass (38 commands), plus a rebuilt Axum runtime
smoke of public, private, Set-Cookie and stalled-TLS cases.
Add a generic host for the signed prepared-hash probe release. It verifies
a release against the advertised contract (pinned Ed25519 key, revocation,
ABI 1, floor, 4 MiB artifact, SHA-256, sole env.host_read import, probe ABI
exports, probe_abi and version planner) and runs any operation by name.

Each host_read is checked against the source size and per-read bound in the
wasmi callback, then carried out as a resumable host trap, awaited outside
the interpreter straight into the release's buffer, and resumed, so reads
never block the executor and no operation is replayed. Calls run in fresh
instances bounded by fuel, memory, per-read, total-read and result limits.

Tests drive a WebAssembly-text stand-in with no format logic.
@erik-sv erik-sv changed the title Encypher text provenance as the provenance.encypher module Sign articles and routed media with the Encypher provenance module Oct 27, 2026
@erik-sv

erik-sv commented Oct 27, 2026

Copy link
Copy Markdown
Author

Updated to dde89ab, preserving the existing feature/55-scripts-in-crates base. All 12 required local gates and two independent review rounds are recorded in the PR and spec. The four GitHub workflow runs currently report action_required and need a maintainer to approve this fork contribution. This token has read-only upstream repository permissions, so it cannot approve those runs or merge. Please approve the workflows; the contribution remains subject to their results and normal review. The latest local real-API/browser run verified all 148 edge-served network observations; direct external and script-created media remain outside the coverage claim. This is source publication, not a production deployment.

@erik-sv

erik-sv commented Oct 27, 2026

Copy link
Copy Markdown
Author

The companion API implementation has landed, and its matching production probe release is published. The public verifier is released as v1.8.0.

Real staging signing passed for embedded PNG, detached BMP and large prepared PNG. The large source was 9,184,683 bytes; prepare and finalize uploaded 7,431 bytes, with the source unchanged. Independent public SDK verification checked signatures, hard bindings, supplied-root static trust and tamper refusal.

This staging proof used authenticated private operator transport and the existing signing-canary credential. It does not prove public manifest URL reachability, deployed Fastly behavior or complete page coverage. It does not replace the four upstream workflows still awaiting maintainer approval on this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant