Skip to content

fix: rewrite preprocessor as standalone host exe, regenerate headers - #213

Open
samooth wants to merge 5 commits into
vrischmann:masterfrom
samooth:fix/preprocessor-rewrite
Open

samooth wants to merge 5 commits into
vrischmann:masterfrom
samooth:fix/preprocessor-rewrite

Conversation

@samooth

@samooth samooth commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

The header preprocessor (which generates c/loadable-ext-*.h) was broken since the 0.16 Io.

Reader migration — readAlloc always failed, dump() was never called, and CI never exercised the consumer path.

Rewrite as a standalone host executable that takes paths via stdin.

Also bumps SQLite to 3.53.4, regenerates the committed headers, and fixes the CI matrix.

Depends on PR2 (feat/compat-0.16-0.17) for the cross-compile legs.

@vrischmann

Copy link
Copy Markdown
Owner

Why exactly do you change it to a standalone executable ?

The build-time header preprocessor (which generates c/loadable-ext-*.h
for loadable extensions) was broken in three ways:

1. readOriginalData used readAlloc(allocator, 1024*1024) on a ~691KB
   file. The new 0.16/0.17 Io.Reader API reads exactly N bytes or
   returns EndOfStream — always failed.

2. processor.dump() was never called in sqlite3()/sqlite3ext(), so
   the output header was just a 47-byte comment banner.

3. test_c_bindings didn't depend on the preprocess step, so zig build
   test never exercised the consumer path.

Rewrite as a standalone host executable that reads paths via stdin
(no argv — works on Windows/POSIX). Uses readFileAlloc for the full
file. addPreprocessRun in build.zig builds and runs it via
addRunArtifact + setStdIn.

Also bumps build.zig.zon to SQLite 3.53.4 and regenerates the
committed c/loadable-ext-*.h headers to match.

Depends on the compat layer (feat/compat-0.16-0.17) for the cross-
compile legs to pass.
- Replace readAlloc with readFileAlloc (Io API change in 0.16/0.17)
- Call processor.dump() to actually write processed headers
- Add test_c_bindings dependency on preprocess step
- Keep inline Build step (no standalone exe)
@samooth
samooth force-pushed the fix/preprocessor-rewrite branch from 2c36fff to 22d264b Compare October 3, 2026 21:33
- Remove c_bindings/c_bindings_ext translate-c modules
- Use addIncludePath directly on modules (matches upstream)
- Remove test_c_bindings; tests use addIncludePath like upstream
- addPreprocessStep returns void (no return value)
- test_step.dependOn(&preprocess.step) so preprocessed headers exist
  before @cImport in c.zig runs during test compilation
- revert makeSQLiteLib to upstream signature (no module_suffix)
- remove duplicate addPreprocessStep call that caused step name conflict
- Remove c_bindings/c_bindings_ext translate-c modules
- Use addIncludePath directly on modules (matches upstream)
- Remove test_c_bindings; tests use addIncludePath like upstream
- Remove BoolFeatureOptions/DefineFeatureOptions/IntFeatureOptions
  (not in upstream; std.meta.fields deprecated in Zig 0.17+)
- Move PreprocessStep/addPreprocessStep before build() so io is used
- Remove test_step.dependOn(&preprocess.step) (upstream doesn't have it;
  causes cross-compile test failures in CI)
- Revert makeSQLiteLib to upstream signature (no module_suffix)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants