Skip to content

Toolchain migration, sibling re-pin train, and supply-chain hardening - #90

Open
OBrezhniev wants to merge 57 commits into
masterfrom
chore/fastfile-idempotent-close
Open

Toolchain migration, sibling re-pin train, and supply-chain hardening#90
OBrezhniev wants to merge 57 commits into
masterfrom
chore/fastfile-idempotent-close

Conversation

@OBrezhniev

@OBrezhniev OBrezhniev commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Companion PR to iden3/snarkjs#628 / #634 and the sibling PRs (fastfile#138, ffjavascript#185, r1csfile#105, wasmcurves#75): binfileutils' side of this development cycle. Contents, in order of landing:

Core

  • readSection allocates SharedArrayBuffers where available and detects the runtime's real max buffer size (newer node raised the limits); the BigBuffer switchover is an explicit 1 GiB constant
  • fastfile/ffjavascript pinned to commit refs instead of file: paths

Toolchain

  • Rollup + Mocha → Vite 8 (Rolldown/Oxc) + Vitest 4, flat eslint 10 config; CI on lts/* and lts/-1 with job timeouts
  • rimraf dropped (fs.rmSync does the clean); LF pinned via .gitattributes

Sibling re-pin train

Every ffjavascript/fastfile advance this cycle re-pinned here bottom-up with the inlined browser bundle rebuilt (this repo's bundle inlines fastfile): direct-IO stack, http Range streaming with retry/backoff + stall timeout, connection cap, in-flight dedupe, persistent block cache and its cache descriptor option, descriptor-object cacheSize/pageSize fallbacks, idempotent close(), the fsqrt/dispatch-race CI-hang fixes, worker input-buffer transfer-back, and the in-house node Worker shim that removed web-worker — the stack's last non-iden3 runtime dependency.

Supply-chain hardening (cascade from snarkjs#634)

  • CI: actions SHA-pinned, permissions: contents: read, persist-credentials: false, npm ci --ignore-scripts with a 10-minute cap, npm audit signatures step
  • .npmrc: ignore-scripts (tree verified script-free) + save-exact; pretest hook folded into the test script
  • Dependabot: 7-day cooldown for npm and the pinned actions
  • Publishing: npm stage publish --provenance on SHA-pinned v6 actions and latest LTS — releases stage in the registry and go live only after a maintainer's npm stage approve with 2FA

Runtime dependencies are exactly two, both iden3-owned and commit-pinned: fastfile, ffjavascript. Suite green throughout (19 tests, node + browser projects).

🤖 Generated with Claude Code

https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp

OBrezhniev and others added 30 commits September 11, 2025 00:57
@OBrezhniev
Create SharedArrayBuffers in readSection
ba832e2
@OBrezhniev
New nodejs versions increased buffer size limits, so trying to detect…
5e4301b 
… it and use max available buffer size to minimize amount of work downstream switching between them. Debug logs.
@OBrezhniev @claude
Make the BigBuffer threshold an explicit 1 GiB constant
46fd2cd 
MAX_BUFFER_SIZE gated on `Buffer.constants.MAX_LENGTH`, but those constants live on
the `buffer` module, not the `Buffer` class, so the probe was always undefined and
fell back to `1 << 30`. Drop the dead check and set 1 GiB explicitly, matched to
BigBuffer's page size: sections at/above it are read into a paged BigBuffer instead
of one flat Uint8Array. No behaviour change (the runtime value was already 1 GiB).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@OBrezhniev
Update devDependencies: rollup 4, eslint 10 (flat config)
918fabe 
Migrates .eslintrc.cjs to flat eslint.config.mjs.
@OBrezhniev
Pin fastfile and ffjavascript to commit refs instead of file: paths
0a79631 
The committed manifest now resolves standalone (once the referenced
branches are pushed). For local development keep uncommitted file:../
overrides in the working tree; the lockfile still records the local
layout and gets regenerated after the branches are published.
@OBrezhniev
Remove readSection debug timers
fcf8d5b 
console.time printed one line per section read into every consumer's output.
@OBrezhniev
Rebuild bundle: silent readSection
b257f6b
@OBrezhniev
Re-pin ffjavascript (silent workers)
28b3274
@OBrezhniev
Regenerate lockfile against the pinned commit refs
1fb7356
@OBrezhniev
Switch fastfile/ffjavascript pins to git+https, re-pin ffjavascript
dee81e9 
git+https installs anonymously (git+ssh needs SSH credentials).
@OBrezhniev
Tidy stray blank lines left by debug-timer removal
41589e5
@OBrezhniev
Re-pin fastfile/ffjavascript to latest commits
4b19bb9
@OBrezhniev
test: add a baseline test suite (this repo had zero tests)
ce9bf23 
npm test ran mocha against an empty glob -- and mocha wasn't even a
declared devDependency, so a clean install couldn't run it at all
('mocha: command not found'). Every byte this library parses (magic
string, version, section table) sits upstream of every zkey/ptau/
wtns/r1cs file snarkjs reads.

Adds: happy-path round-trip, malformed-input rejection (wrong magic,
unsupported version, missing/duplicate section, section-size
mismatch), zero-length sections, copySection/sectionIsEqual/
writeBigInt/readBigInt correctness, and a regression test for the
just-fixed fastfile truncated-read corruption bug (readSection on a
truncated on-disk file must not silently misplace real data).

Also re-pins fastfile to the fix (see fastfile commit for the actual
bug/fix), adds mocha as a devDependency, and adds the standard
serialize-javascript/diff overrides other iden3 sibling repos already
carry for the same mocha-transitive audit findings.
@OBrezhniev
Re-pin ffjavascript (worker-error propagation fixes)
e982002
@OBrezhniev
Re-pin fastfile/ffjavascript (proving-phase hang-audit fixes)
892d9f2
@OBrezhniev
Re-pin ffjavascript (partitioned MSM)
c0b23ef
@OBrezhniev
Re-pin ffjavascript (endo cutoff)
a235b93
@OBrezhniev @claude
feat: re-pin fastfile with HTTP Range streaming; URL readBinFile tests
377a7a1 
readBinFile with an http(s) URL now streams: header scan via cached
pages, section reads via Range requests, with a buffer-it-all fallback
for servers without range support. No code change here -- the transport
lives in fastfile ffc97b3 -- but the URL path is now covered by tests
against a local server in both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
@OBrezhniev @claude
chore: re-pin fastfile (http/blob page-size cap)
5bad9bb 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
@OBrezhniev @claude
chore: re-pin ffjavascript (worker boot-failure give-up)
b097683 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
@OBrezhniev @claude
chore: re-pin ffjavascript (wasmcurves ESM migration)
372283f 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
@OBrezhniev @claude
chore: re-pin fastfile (idempotent close())
60dcebe 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@OBrezhniev @claude
test: cover section-protocol misuse guards and sectionIsEqual size mi…
e808955 
…smatch

99.4% lines / 98.0% branches / 100% functions. The >=1 GiB BigBuffer
readSection path is c8-ignored (unreachable with test-sized files).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile and ffjavascript (coverage pushes + dormant-bu…
44c1fe8 
…g fixes)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript (standards-vector tests)
705336c 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript (wasmcurves re-pin)
824e9f1 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile and ffjavascript (Range degrade, portable tear…
9a54502 
…down, Node 20 dispatch)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: restore transitive git deps in the lockfile (npm ci sync)
a1fbc51 
Local installs ran with --allow-git=root, which silently omits transitive
git dependencies (ffjavascript -> wasmcurves) from the lock; npm ci then
fails with EUSAGE. Regenerated with --allow-git=all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
Migrate build/test tooling to Vite + Vitest (rebased onto chore/fastf…
f71fb82 
…ile-idempotent-close)

Port of the feature/esm-tooling migration (dee038d) onto the current branch:
Rollup -> Vite (node CJS + new browser ESM build), Mocha -> Vitest with node +
browser (Playwright) projects, flat eslint.config.js, CI workflow. Our
post-fork coverage suite (test/binfileutils.test.js) runs under the node
project as-is (this.timeout dropped).

Re-pinned deps to the migrated siblings: ffjavascript 4ac1cba,
fastfile e7eb0f0. vitest family at ^4.1.11 (GHSA-p63j-vcc4-9vmv);
postcss/brace-expansion overrides added (npm audit clean).

19 tests pass (node + browser in Chromium); lint clean.

(cherry picked from commit dee038d)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (sequential vitest files upstream)
f8aedac 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
OBrezhniev and others added 27 commits August 27, 2026 00:39
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (upstream CI-hang fix)
9ee4f27 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (wasmcurves re-pin cascade)
7127862 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
ci: test on lts/-1 as well
c2f5302 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (lts/-1 cascade, vitest-artifac…
654676e 
…t cleanup)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (browser Blob streaming test)
c69440f 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (unref fix, fastfile docs)
6c2616f 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
ci: add timeout-minutes to the npm-publish job
102335c 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (direct-IO stack + persistent http block cache)
1880d15 
Now points at feature/direct_rw_optimization: the merged streaming/idempotent
line plus the direct-IO fast paths and the opt-in IndexedDB persistentCache
backend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
Rebuild bundles: fastfile persistentCache (the browser bundle inlines…
a066c87 
… fastfile)

The browser ESM bundle vendors fastfile's browser build, so a fastfile
re-pin without a rebuild ships the stale copy -- the persistentCache option
was silently ignored downstream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (in-flight cache dedupe); rebuild inlined bundle
dc38d51 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (range-fetch connection cap); rebuild inlined …
95b8591 
…bundle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (full-download persistent cache); rebuild inli…
8a30369 
…ned bundle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (conditional-probe CORS fallback); rebuild inl…
a13d1e7 
…ined bundle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (fsqrt c=0 hang fix); rebuild b…
1587193 
…undle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (dispatch-race fix); rebuild bu…
b04e62f 
…ndle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (PR #145 merged: range-request retry with back…
d74513c 
…off + stall timeout); rebuild inlined bundle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript + fastfile (verify-path concurrency fixes)…
3901abd 
…; rebuild bundle

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript (worker input-buffer transfer-back)
63e62fe 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (descriptor-object cacheSize/pageSize fallback…
baed18f 
…s); rebuild bundles

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (http cache option renamed persistentCache -> …
795f9c1 
…cache); rebuild bundles

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: pin LF line endings via .gitattributes
42d90f3 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: drop rimraf (fs.rmSync does the clean)
574972d 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: supply-chain hardening (cascade from snarkjs#634)
dd9a8ac 
CI: actions SHA-pinned, contents:read permissions, GITHUB_TOKEN never
persisted, npm ci --ignore-scripts with a 10-minute cap (npm applies no
timeout to git-dependency clones), npm audit signatures step.
.npmrc: ignore-scripts (tree verified script-free) + save-exact; any
pretest hook is folded into the test script since ignore-scripts also
disables pre/post hooks. Dependabot: 7-day cooldown for npm and the
pinned actions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
ci: publish via npm staged releases with provenance, on v6/latest LTS
51eca83 
npm stage publish (npm >= 11.15, GA May 2026) puts the tarball in the
registry staging queue with an OIDC provenance attestation; a
maintainer releases it with npm stage approve + 2FA, so a compromised
automation token can stage but never ship. Actions bumped to the same
SHA-pinned v6 as ci.yml, node pinned to latest LTS, id-token: write
scoped to this job for --provenance.

NOTE: the IDEN3_CIRCOM_NPM_PUBLISH_TOKEN granular token needs the
staging permission (--allow-stage-publish) granted before the next
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin ffjavascript (web-worker replaced by an in-house shim)
e8f1270 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin fastfile (align with the ffjavascript shim re-pin)
5c8e0d9 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
@OBrezhniev @claude
chore: re-pin siblings (0KIMS copyright headers)
34eb6c3 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@OBrezhniev