Modules have clients, not dependencies: generated clients for dependencies, self, and standalone use - #17
Draft
eunomie wants to merge 17 commits into
Draft
Modules have clients, not dependencies: generated clients for dependencies, self, and standalone use#17eunomie wants to merge 17 commits into
eunomie wants to merge 17 commits into
Conversation
A module's dependency and a standalone generated client are the same artifact built twice. Record the design that collapses them: the hand-written runtime moves to io.dagger.sdk, core types are generated into io.dagger.core, each bound module gets io.dagger.client.<module> generated from the engine's client-facing schema, and an idempotent serve preamble makes the emitted bytes identical whether the client is a module's dependency, the module's own self client, or a standalone artifact. Decisions D1-D5 are recorded in the document, each checked against the engine source or the Go SDK rather than assumed: the client schema really does exclude the bound module's dependencies, and it installs that module namespaced rather than promoting it to the Query root, which is what makes the self client fall out of the same code path. Signed-off-by: Yves Brissaud <yves@dagger.io>
The introspection schema already records which module contributed each type and each field: the engine emits @sourcemap(module: "<name>") on both, and core carries none. That is the exact partition a per-module client generator needs, so expose it rather than inventing a name-prefix heuristic later. The directive value arrives JSON-encoded, so it is unquoted the same way getExpectedType already does, and an empty module reads as core. Tests are shaped after a real clientSchemaIntrospectionJSON dump, which is where the interesting case lives: a module-owned field on a core type (Query.e2E, Binding.asE2E) belongs to the module while the type hosting it stays core. Signed-off-by: Yves Brissaud <yves@dagger.io>
A generated package is one slice of a schema: core is every unowned type with owned fields stripped, and a module's client is every type it owns plus its fields on core types. Those fields (Query.hello, Binding.asHello) are how a module extends core; they have no home of their own in Java, so the partition keeps them aside as extensions for the entry point to emit as shims rather than silently dropping the LLM/agent surface. Version and the IDAble helpers are not schema types, so they do not fall out of the partition; they belong to core alone, or every client package would carry a colliding copy. The full schema stays reachable for lookups; only the emitted types are narrowed, on copies, so the schema is never mutated. A client partition for a module the schema does not contain is an error: an empty client is a misconfiguration, never a result. Signed-off-by: Yves Brissaud <yves@dagger.io>
Every generated reference used ClassName.bestGuess on a simple name, which is only right because everything landed in one package. Route schema types, the hand-written runtime, and the type being generated through a TypeRegistry instead, and let CodeWriter take its package from it. That is the seam a second package needs; nothing else changes here. TypeRef is the actual type-reference resolver and goes through the registry too, as do the string-interpolated class names ($L) that only worked because the name was in scope: with a package on the ClassName, $L would print it fully qualified, so they become $T. Behaviour is unchanged. The registry is built with every package set to io.dagger.client, and regenerating the vendored client from a real engine schema before and after this patch differs in exactly one way: executeQuery(java.lang.String.class) is now executeQuery(String.class), because $T elides the implicit java.lang import where $L printed the qualified name. The two in-scope references to a method's nested *Arguments class stay package-less on purpose: they name a member of the class being written, not something in another package. Signed-off-by: Yves Brissaud <yves@dagger.io>
Generated code is about to leave io.dagger.client for packages of its own, and every generated type is built on a QueryBuilder, chains through it, implements InputValue, merges Arguments and converts Scalars. All of those were package-private, which was right while the generated code sat next to them and is impossible once it does not: a class cannot implement a non-public interface from another package. So QueryBuilder and its chain/execute methods, InputValue, Arguments.merge and Scalar.convert become public, the generated constructors that take a QueryBuilder or a Connection become public, and the generated Client gains a queryBuilder() accessor for code that has to start a chain from the root — the serve preamble of a generated module client, chiefly. This deliberately reverses the decision in hack/designs/2026-08-17-nullable-object-returns.md to keep QueryBuilder package-private: a public transport is the price of generating into more than one package, and the javadoc says it is not a user-facing API. Signed-off-by: Yves Brissaud <yves@dagger.io>
io.dagger.client is about to mean "a generated client": one package segment per bound module, nothing else. QueryBuilder is not a client, and leaving the transport under that prefix would make the package name a lie, so the hand-written runtime moves to io.dagger.sdk with its subpackages (engineconn, exception, graphql, telemetry) intact. Nothing generated moves yet: the vendored bindings stay in io.dagger.client, which is why Dagger and AutoCloseableClient now import the generated Client, and why the annotation processor, templates, samples, e2e fixture and README only change the imports of runtime classes. The codegen resolves the runtime through the registry already; the statements that still named QueryBuilder and Arguments as literal text become $T so the emitted import follows the package. Mechanical: every moved file is its previous content with the package and runtime imports rewritten, and the full reactor test path is green. Signed-off-by: Yves Brissaud <yves@dagger.io>
ModuleBinding.ensureServed is the serve preamble every generated module client calls before its bindings can resolve: moduleSource(ref, pin) for a git module, currentWorkspace.moduleSource(path) for a local one, then withName(finalName).asModule.serve. It serves unconditionally. The engine keys served modules by name, deduplicates a repeat of the same source and pin, and rejects a different source under the same name — so a probe would only ever hide the one conflict worth reporting, and a cache would only ever race. Inside a module, where the engine has already served every dependency and the module itself, the call is a no-op by the engine's own rule; in a standalone client it is the bootstrap. The name is the module's final one, after any dependency alias, because that is what namespaces its types. Connection.get opens a session again when the environment has none: DAGGER_SESSION_PORT/TOKEN first, else `dagger session` from _EXPERIMENTAL_DAGGER_CLI_BIN or the PATH, parsed from its announcement line and stopped with the connection (and at JVM exit, for the global client that is never closed). The Go and TypeScript SDKs do the same and additionally download a CLI when none is found; that is left out, like a test framework that uses the Docker the host has. ProcessBuilder is enough, so the fluent-process dependency that the old CLIRunner needed stays out. Signed-off-by: Yves Brissaud <yves@dagger.io>
Formatter-only changes to files the build reformats on every run, so that later patches carry only their own edits. Signed-off-by: Yves Brissaud <yves@dagger.io>
A module client is more than its types. Its root type gains a static from(Client, <constructor args>) that serves the bound module through ModuleBinding and returns the root, and an alias named after the module that delegates to it — the static-import form, hello(dag()), is what keeps a self call readable inside the module, where the authored Hello is already in scope. Both take the module's constructor arguments, optional ones included, exactly as the instance methods would, because they are generated by the same code: a field method now chains from a receiver, which is this for an instance method and a first parameter for a static one, and declares the serve call's exceptions when it carries that preamble. Java has no extension methods, so the fields a module adds to Binding and Env are emitted the same way, as static shims on the root type taking the core object first. Without them the whole LLM surface of a module's types would silently vanish from the partition. Every generated object exposes its query builder for that reason. The root type and the entry field are read off the schema — the return type of the one Query field the module owns — never derived from the module name, which gives E2e where the engine says E2E. The binding baked into the preamble is the module's final name, kind, ref and pin. Signed-off-by: Yves Brissaud <yves@dagger.io>
The cutover, in one patch because the tree cannot build between its halves. The generator now runs a plan: one core entry and any number of client entries, each with its own schema and, for a client, the bound module's name and binding; all of it lands in one output tree from one Maven invocation, so a module with many dependencies does not pay one Maven run per package. The schema-only form the reactor and the packager use is a one-entry core plan, and the CLI-query fallback still stands behind it. Each package root is cleaned before it is written and no other root is touched, which is what a second, one-entry pass for the self client relies on. Core is generated into io.dagger.core, from whatever schema the consumer is entitled to: a module's own module-facing schema keeps Host and the other hidden types out of module code; a standalone client's schema hides nothing. A client goes into io.dagger.client.<module>, resolving every type it does not own to core. The runtime, the annotation processor, the templates, the e2e fixture and the README follow the generated types to io.dagger.core. Signed-off-by: Yves Brissaud <yves@dagger.io>
mod.dang prefers the committed plugin repository under prebuilt/m2 whenever it exists, so every codegen change in this series is inert in module generation until the jar is rebuilt. This is packager:generate's reproducible output for the plugin as it now stands. Signed-off-by: Yves Brissaud <yves@dagger.io>
Module generation becomes plan-driven. The per-module flow builds a plan — core from the module's own module-facing schema, so the types hidden from module code stay hidden, plus one client entry per declared dependency from that dependency's client-facing schema — and hands it to one codegen run. The mechanics move to a Codegen type shared with client generation, since the two are the same generator. The self client needs the module's own client-facing schema, and the engine only produces that by loading the module, which means building it: a circularity broken by bootstrapping. The first pass generates core and the dependency clients with the previous generation's client packages carried in and the module's own kept, so module code that already calls its self client still compiles; the vendored SDK and the entrypoint are then staged onto the workspace, the engine loads the module from there and hands back its schema, and a second, one-entry pass writes the self client over the first pass's output — so the vendored tree is exactly core, the dependency clients and this client. A dependency declared below v1.0.0-0 is refused up front: the engine renders core through the dependency's declared version, and a pre-1.0 view carries per-type ID scalars io.dagger.core does not have. Signed-off-by: Yves Brissaud <yves@dagger.io>
generateClient(ws, module, path) writes a plain Maven project bound to one module: the SDK runtime, io.dagger.core and io.dagger.client.<module> under sdk/, exactly as a module receives them, and a pom seeded from templates/client when the directory has none. Both core and the client come from the bound module's client-facing schema, which hides nothing: a client is allowed everything the CLI is. The client package is the same plan entry a dependency produces, so it is byte-identical to what a module depending on that module vendors. initClient seeds the pom for a client the engine registers in workspace config, and generateAllClient is the @generate hook that regenerates every registered client visible from the caller's location, with the engine owning the list and the cwd policy as it does for modules. The three mirror the Go SDK's surface. Signed-off-by: Yves Brissaud <yves@dagger.io>
The module tree gains io.dagger.core and one io.dagger.client.<module> per dependency plus the module's own; dependencies become clients and a self call goes through the engine like any other; standalone clients are generated by generate-client and registered clients regenerated by dagger generate; and, with no shim, the import moves every existing module needs are given as a sed one-liner. Signed-off-by: Yves Brissaud <yves@dagger.io>
…red clients Two real Java modules under fixtures/clients — dep, and app depending on it — carry their own workspace config, which the checks place at the workspace root so the SDK's module registry sees them without touching the module inventory the discovery checks assert. clients-generate-check generates app from nothing and expects core, the dep client with its serve preamble bound by workspace path, and app's own client; then swaps in a source that calls app through that client and generates again, which is the carry-over case; then generates once more with no edits and expects an empty changeset. standalone-client-check generates a client for dep on its own, expects it byte-identical to the one app vendors, expects Host present — a client hides nothing — and builds the project with a plain mvn package around a main that uses it. registered-client-check seeds a client with initClient and materializes it through the generate hook off an as-sdk.clients entry. Signed-off-by: Yves Brissaud <yves@dagger.io>
Every generate now runs two installs into the shared ~/.m2 volume, and
concurrent installs from parallel checks corrupt maven-metadata-local.xml
("in epilog non whitespace content is not allowed"). Every mount of the
volume is CacheSharingMode.LOCKED, so the SDK's containers serialize on
it; this is the packager's side of it, the SDK's mounts carry it in
their own patches.
Signed-off-by: Yves Brissaud <yves@dagger.io>
The feature is implemented and its clients e2e is green in the engine; move the design and plan into done/. Signed-off-by: Yves Brissaud <yves@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modules have clients, not dependencies
A module's dependency and a standalone generated client were already the same artifact, built twice. This series collapses them: a dependency declared in
dagger-module.tomlbecomes a generated client, produced by the same generator — and the same plan entry — that produces a standalone client for any module. A module also gets a client for itself, and that is how it calls itself.What changes for a Java module
io.dagger.client→io.dagger.sdk; the generated core API (Container,Directory, …) lives inio.dagger.core; each bound module getsio.dagger.client.<module>.from(Client, <constructor args>), a static-import alias named after the module (hello(dag())), and static shims for the module's fields on core types (Binding/Env). The alias is the idiom for self calls:app(dag()).build(source).io.dagger.sdk.ModuleBinding— unconditionally on first use (the engine deduplicates an identical binding and rejects a conflicting one), cached per client afterwards.Connection.getopens adagger sessionwhen the environment has none (_EXPERIMENTAL_DAGGER_CLI_BINordaggeronPATH), so a generated client runs outside the engine.sedmigration for the import moves.What changes in generation
@sourceMap-attributed type and field is partitioned exactly; module-owned fields on core types become shims rather than being dropped.mod.danggeneratesio.dagger.corefrom the module-facing schema (soHostand the other hidden types stay hidden from module code), one client per dependency from that dependency's client-facing schema, then bootstraps the module's own client: with the SDK and the entrypoint staged, the engine loads the module from the staged workspace and hands back its client-facing schema, and a second one-entry pass writes the self client. The previous self client is carried through the first pass so module code that already calls it keeps compiling.generateLocalDependencies, which returns an empty generator group for value workspaces and never carried this SDK's generators in this repository.generateClient,initClientandgenerateAllClientmirror the Go SDK's surface; a standalone client is a plain Maven project with everything generated undersdk/.Verification
.dagger/modules/e2e): generation from nothing with core, the dependency client, an aliased second client of the same dependency and the self client; a self call added and picked up; an idempotent rerun; a stale package removed; the standalone client byte-identical to the vendored dependency client and building withmvn package; a registered client seeded byinitClientand materialized by the rollup, idempotently.dagger checksuite green locally (e2e, packager, sdk-sdk contract and chain checks).Known limits (recorded in the design doc)
Deserializers resolve through theDagger.dag()singleton, so a standalone client opened withDagger.connect()cannot deserialize IDs viaJsonConverteragainst its own session.CLISession.startblocks until the CLI announces a session or exits.CI note (not a code issue in this branch)
The current CI run shows the four
sdk-sdk:contract:*checks failing withoci-sha lockfile: parsing lock: ... cannot unmarshal object into Go value of type string, thrown at workspace load — before any code in this branch runs. This is a mismatch between the repository's committeddagger.lock(object-valued v2 entries) and the CLI the contract harness runs, and it is not introduced here:dagger.lockis byte-identical toupstream/main; this branch does not touch it.JavaSdk.initModule(the function under test) is byte-identical toupstream/mainand reads no lock; the error precedes it.upstream/main; the wholedagger checksuite is otherwise green (unit tests, alle-2-e:*including the new client checks, the rest ofsdk-sdk:*).The fix, if any, is a repository-wide lock/CLI alignment outside this feature's scope. The PR stays a draft pending that.
Design and plan:
hack/designs/2026-08-26-modules-have-clients.md(decisions D1–D9).