Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions .dagger/modules/e2e/fixtures/generate-deps/dep/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
{
"type": "module",
"dependencies": {
"typescript": "5.9.3"
}
"type": "module"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/sdk/** linguist-generated
/__dagger.entrypoint.ts linguist-generated
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
/sdk
/**/node_modules/**
/**/.pnpm-store/**
/.env
/__dagger.entrypoint.ts
13 changes: 13 additions & 0 deletions .dagger/modules/e2e/fixtures/generate-legacy-deps/app/dagger.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"name": "legacy-deps-app",
"engineVersion": "latest",
"sdk": {
"source": "typescript"
},
"dependencies": [
{
"name": "legacydep",
"source": "../dep"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"type": "module",
"dependencies": {
"typescript": "5.9.3"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { object, func } from "@dagger.io/dagger"

@object()
export class LegacyDepsApp {
@func()
hello(): string {
return "hello"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"compilerOptions": {
"target": "ES2022",
"moduleResolution": "Node",
"experimentalDecorators": true,
"strict": true,
"skipLibCheck": true,
"paths": {
"@dagger.io/dagger": ["./sdk/index.ts"],
"@dagger.io/dagger/telemetry": ["./sdk/telemetry.ts"]
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/sdk/** linguist-generated
/__dagger.entrypoint.ts linguist-generated
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
/sdk
/**/node_modules/**
/**/.pnpm-store/**
/.env
/__dagger.entrypoint.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"name": "legacydep",
"engineVersion": "latest",
"sdk": {
"source": "typescript"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"type": "module",
"dependencies": {
"typescript": "5.9.3"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { object, func } from "@dagger.io/dagger"

@object()
export class Legacydep {
@func()
value(): string {
return "dep"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"compilerOptions": {
"target": "ES2022",
"moduleResolution": "Node",
"experimentalDecorators": true,
"strict": true,
"skipLibCheck": true,
"paths": {
"@dagger.io/dagger": ["./sdk/index.ts"],
"@dagger.io/dagger/telemetry": ["./sdk/telemetry.ts"]
}
}
}
143 changes: 143 additions & 0 deletions .dagger/modules/e2e/generate.dang
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,149 @@ type GenerateChecks {
null
}

"""
A CLI 1.0 module is generated by this SDK, and what it writes has to satisfy
the engine runtime by itself: that runtime does no codegen for a
dagger-module.toml module, it mounts sdk/ as @dagger.io/dagger and runs the
committed entrypoint.

So this asserts the whole contract rather than a single marker file — the
bundled library the bindings import, the bindings themselves, the dispatch
entrypoint, and the config the runtime reads. A missing piece here is a module
that fails to load at `dagger call` time, long after generate.
"""
generateWorkspaceModuleCheck(ws: Workspace!): Void @check {
let path = fixtures.depAppModule
let changes = typescriptSdk.mod(ws, path: path).generate(ws)
let tree = changes.after.directory(path)

Asserts.generated(changes, path + "/sdk/client.gen.ts")
Asserts.generated(changes, path + "/sdk/core.js")
Asserts.generated(changes, path + "/sdk/core.d.ts")
Asserts.generated(changes, path + "/sdk/index.ts")
Asserts.generated(changes, path + "/sdk/telemetry.ts")
Asserts.generated(changes, path + "/__dagger.entrypoint.ts")

# package.json and tsconfig.json are asserted on the resulting tree rather
# than the changeset: this fixture already carries correct ones, so a
# correctly-behaving generate leaves them untouched and out of the diff.

# The bindings must reach the runtime through the bundle sitting beside
# them, not through the npm package: sdk/ *is* @dagger.io/dagger here.
Asserts.stringContains(
tree.file("sdk/client.gen.ts").contents,
"from \"./core.js\"",
"module bindings should import the bundled runtime",
)

# The entrypoint dispatches by importing the user's classes from where they
# were declared, which only the source scan knows.
Asserts.stringContains(
tree.file("__dagger.entrypoint.ts").contents,
"from \"./src/index\"",
"entrypoint should import the module's own source",
)

# This fixture arrives already pinned, so the pin only proves generation
# preserved it. That generation *writes* one is generateTypescriptPinCheck.
Asserts.stringContains(
tree.file("package.json").contents,
"\"typescript\"",
"package.json should keep the module's typescript pin",
)
Asserts.stringContains(
tree.file("tsconfig.json").contents,
"./sdk/index.ts",
"tsconfig should alias @dagger.io/dagger to the generated sdk",
)

null
}

"""
Generating a module that has not pinned typescript should add the pin.

Run against the dependency fixture, whose package.json carries nothing but
`type: module` — the shape a module has before its first generate. That is the
only way to tell writing from preserving: a fixture that arrives pinned goes
on passing after generation stops writing the pin. The cost of losing it is
silent — the runtime mounts its prebuilt compiler
only when the version matches its default, so an unpinned module installs
typescript on every single call.
"""
generateTypescriptPinCheck(ws: Workspace!): Void @check {
let path = fixtures.depLibModule
let changes = typescriptSdk.mod(ws, path: path).generate(ws)

Asserts.stringNotContains(
ws.directory("/" + path).file("package.json").contents,
"typescript",
"fixture must start unpinned for this check to mean anything",
)
Asserts.stringContains(
changes.after.directory(path).file("package.json").contents,
"\"typescript\"",
"generation should pin typescript for a module that has not",
)

null
}

"""
Regenerating should remove bindings for dependencies the module no longer has.

Layering the generated tree over the existing one is what keeps a module's own
source and config out of the diff, but it also leaves behind any file we no
longer generate: drop a dependency and its `<dep>.gen.ts` stays, exporting an
API against a module that is no longer served. A default `tsc` compiles the
whole sdk/ directory and rejects it.

Staged rather than fixtured because sdk/ is gitignored in every module, so
there is no committed stale file to start from — the check plants one and
asserts regeneration takes it away.
"""
generatePrunesStaleBindingsCheck(ws: Workspace!): Void @check {
let path = fixtures.depAppModule
let stale = path + "/sdk/departed.gen.ts"

let dirty = ws.withNewFile("/" + stale, "export const gone = true\n")
let changes = typescriptSdk.mod(dirty, path: path).generate(dirty)

Asserts.generated(changes, path + "/sdk/gendep.gen.ts")
Asserts.assert(
Asserts.contains(changes.removedPaths, stale),
"regeneration should remove bindings for a dependency that is gone, got removed [" +
changes.removedPaths.join(", ") + "]",
)

null
}

"""
Generating a legacy module with a local dependency should write that module's
files and nothing else.

The dependency closure has to be staged before generation — a dependency's
schema is only loadable once its own generated files exist — but that staging
is scaffolding, not output. A legacy module routes to the engine, which hands
back a whole workspace rather than a changeset, so what is taken out of that
workspace is the only thing keeping the dependency's freshly generated files
from being written into the user's tree alongside the module they asked for.
"""
generateLegacyDependencyCheck(ws: Workspace!): Void @check {
let changes = typescriptSdk.mod(ws, path: fixtures.legacyDepAppModule).generate(ws)

Asserts.generated(changes, fixtures.legacyDepAppModule + "/sdk/index.ts")

Asserts.notAdded(
changes,
fixtures.legacyDepLibModule + "/sdk/index.ts",
"generating a legacy dependent should not also write its dependency's files",
)

null
}

"""
Generating a module with a local dependency must resolve that dependency and
emit its typed bindings alongside the module's own — otherwise the module's
Expand Down
8 changes: 8 additions & 0 deletions .dagger/modules/e2e/util.dang
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,12 @@ type Fixtures {
let depAppModule: String! = root + "/generate-deps/app"
let depLibModule: String! = root + "/generate-deps/dep"

# The same edge in the legacy format: a dagger.json module with a local
# dagger.json dependency. Generation routes to the engine here, and its
# dependency staging is what a changeset must not carry back.
let legacyDepAppModule: String! = root + "/generate-legacy-deps/app"
let legacyDepLibModule: String! = root + "/generate-legacy-deps/dep"

# Module config in <root>/.dagger/modules/app, implementation source in
# <root>/ci — the split layout `dagger setup` migration produces.
let parentSourceRoot: String! = root + "/parent-source"
Expand Down Expand Up @@ -185,6 +191,8 @@ type Fixtures {
managedTomlModule,
depAppModule,
depLibModule,
legacyDepAppModule,
legacyDepLibModule,
parentSourceModule,
]
}
9 changes: 9 additions & 0 deletions dagger.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@ path = ".dagger/modules/e2e/fixtures/generate-deps/app"
[[modules.typescript-sdk.as-sdk.modules]]
path = ".dagger/modules/e2e/fixtures/generate-deps/dep"

# The legacy-format counterpart of generate-deps: a dagger.json module with a
# local dagger.json dependency. Generation routes to the engine for these, the
# one path where the SDK hands back a workspace it did not assemble itself.
[[modules.typescript-sdk.as-sdk.modules]]
path = ".dagger/modules/e2e/fixtures/generate-legacy-deps/app"

[[modules.typescript-sdk.as-sdk.modules]]
path = ".dagger/modules/e2e/fixtures/generate-legacy-deps/dep"

[[modules.typescript-sdk.as-sdk.modules]]
path = ".dagger/modules/e2e/fixtures/parent-source/.dagger/modules/app"

Expand Down
52 changes: 39 additions & 13 deletions design/module-gen.md
Original file line number Diff line number Diff line change
Expand Up @@ -559,15 +559,40 @@ engine builds its own from whatever it resolves.

This is the cutover's safety net and it is cheap while both paths exist.

**Phase 3 — cutover, for `dagger-module.toml` modules only.** Flip
`Mod.generate` to the local path when the module's config is
`dagger-module.toml`; keep delegating to `polyfill.moduleSource(...).generate`
for `dagger.json` ones (§8). Keep `generateLocalDependencies` staging (still
required: a dependent's schema can only be built if its local deps' generated
files exist, and dep generation may cross SDKs). Update the e2e assertions from
`sdk/index.ts` to the full expected tree, and add `__dagger.entrypoint.ts` +
`sdk/core.js` assertions. Verify `dagger call` on a generated fixture actually
runs — the runtime contract in §2.2 is only really proven by executing a module.
**Phase 3 — cutover, for `dagger-module.toml` modules only.** ✅ Done.
`Mod.generate` takes the local path when the module's config is
`dagger-module.toml` and keeps delegating to
`polyfill.moduleSource(...).generate` for `dagger.json` ones (§8).
`generateLocalDependencies` staging is kept for both: a dependent's schema can
only be loaded once its local deps' generated files exist, and dep generation
may cross SDKs.

Validated three ways rather than one:

- **Byte-for-byte against the engine.** Generating the same fixture both ways
produces identical `client.gen.ts`, `__dagger.entrypoint.ts`, `index.ts`,
`telemetry.ts`, `core.d.ts`, `package.json` and `tsconfig.json`. The one
exception is `core.js`, which is *ours* by construction: built from the
vendored lockfile at 4.3 MB against the engine's unpinned 5.4 MB (§4.2).
Getting there required three generator fixes the diff surfaced (§ the
generator was ported from an older upstream commit than the engine we pin).
- **By running a module.** A `dagger call` on a generated fixture returns its
value, exercising our bindings, bundle and entrypoint through the engine
runtime, dependency bindings included. The runtime contract in §2.2 is only
really proven by executing a module.
- **By an e2e check** asserting the whole tree contract, not one marker file.

**VCS files are not written** (decided): no `.gitignore`, no `.gitattributes`.
The engine appends to both around codegen, but for a workspace module the
ignore list is reduced to `node_modules`/`.pnpm-store` anyway, which is the
user's business rather than codegen's.

Worth knowing, since it looks like our bug when it happens: a module whose
`.gitignore` still ignores `sdk/` — a legacy `dagger.json` module migrated to
`.toml` — fails to load with *"committed generated file sdk/client.gen.ts is
missing"* even though the file is on disk, because the ignore keeps it out of
the module context. The engine has the same problem: it only avoids *adding*
generated paths for toml modules, it never removes one already there.

**Phase 4 — upstream cleanup** (separate `dagger/dagger` PR, §9).

Expand Down Expand Up @@ -650,15 +675,16 @@ late):
Bundled without `--compile`, with the trimmed compiler at
`node_modules/typescript`, it scans a fixture module and emits a
`typedef.json` with the `location` data the entrypoint needs.
2. **Does our `module` mode reproduce the engine's `sdk/client.gen.ts`
byte-for-byte** for a fixture, given `ModuleSource.introspectionSchemaJSON`?
This is the differential check of §7 Phase 2, run by hand once, first.
2. ~~**Does our `module` mode reproduce the engine's `sdk/client.gen.ts`
byte-for-byte**~~ — **yes**, once the generator was synced with the engine
it targets (§7 Phase 3). It did not before: enum members were miscased,
`arguments` went unescaped, and the entrypoint carried no source maps.
3. ~~**Is `bun build` output reproducible enough**~~ — **yes**, with the image
pinned by digest and the vendored lockfile in place: a second packager run
over an unchanged tree reports no changes. Dropping the lockfile is what
breaks it (§4.2).

Only the differential check is left, and it belongs to Phase 2 anyway.
All three are now answered; the work they were guarding is done.
(For the record on the fetch alternative in §4.1: dang does support
`@cache(policy:, ttl:)` → `withCachePolicy`, but a plain container exec is
already content-addressed by the engine, so the decorator would mostly buy a TTL
Expand Down
Loading