Repository navigation
build(image): ship one binary and keep the .so files out of the image - #9291
Merged
Merged
Conversation
`make build` is now `build-server`: the framework, the API and every plugin in the generated registry link into a single bin/lake, so the runtime stage has no plugin directory to copy. - backend/Dockerfile: copy only /app/bin/lake instead of the whole bin/, and create an empty /app/bin/plugins as the sideload mount point. It is created as root, before the chgrp that makes /app group-writable, because the stage otherwise runs as the devlake user. - Drop `COPY --from=build /go/bin /usr/bin`: mockery and swag are build tools and nothing at runtime uses them. - Dockerfile.local: one build step instead of build-plugin plus build-server, and copy only the binary. - build-server.sh: add -trimpath and -s -w. A DEVLAKE_DEBUG build keeps the symbols so delve still works. - Makefile: build, dev and godev no longer build plugins; build-plugin stays for sideload and core-only builds. - env.example: PLUGIN_DIR now documents the sideload directory. GO_PLUGINS keeps working: it reaches the registry generator, so it now selects which plugins are compiled in rather than which .so files are built. plugins-registry clears GOOS/GOARCH: the Dockerfile exports them to cross-compile the server, and `go run` would otherwise build the generator for the target platform and fail with "exec format error" on an amd64 build host.
Startrekzky
pushed a commit
that referenced
this pull request
Oct 31, 2026
…stry (#9288) * feat(build): link plugins into the server binary via a generated registry Plugins were only discoverable at runtime, through the .so files in PLUGIN_DIR. Add a compile-time path so the whole plugin set can be linked into one binary: - core/plugin.MustRegister registers a plugin under the name it reports. - tools/pluginsgen scans plugins/*/impl for the type implementing plugin.PluginMeta and generates plugins/all, which imports each plugin and registers it from an init function. It honours DEVLAKE_PLUGINS so subset builds keep working, and emits an empty package for "none". - server/main.go blank-imports the registry. Building with -tags devlake_external_plugins swaps in an empty package for the shared-object-only build. - runner.LoadGoPlugins treats a missing PLUGIN_DIR as "no shared-object plugins" instead of failing startup, so a single-binary deployment does not need to ship an empty directory. - CI regenerates the registry and fails when the checked-in file drifts. Build scripts and Dockerfiles are deliberately untouched: `make build` still produces the .so files, and they register over the compiled-in ones harmlessly. Repackaging follows separately. * feat(plugin): treat sideloaded .so plugins as a supported extension path Reframe the runtime loader around the shipping model: the official binary compiles in every official plugin, and PLUGIN_DIR is where users drop additional or replacement plugins. - A broken or incompatible .so is logged and skipped instead of aborting startup. Sideloaded plugins are third-party code built outside this repo; one bad file must not take the server down. - A shared object reporting a name that is already registered (an official plugin) overrides it, with a warning. That is the existing behaviour and it is what makes patching an official plugin possible, so keep it but make it visible. - Extract open/lookup/assert into openGoPlugin and stop the loader goroutine from writing the walk's `err` variable. - Update the generated registry and the core-only build tag docs to describe sideloading rather than presenting .so as a legacy fallback. * build(image): ship one binary and keep the .so files out of the image (#9291) `make build` is now `build-server`: the framework, the API and every plugin in the generated registry link into a single bin/lake, so the runtime stage has no plugin directory to copy. - backend/Dockerfile: copy only /app/bin/lake instead of the whole bin/, and create an empty /app/bin/plugins as the sideload mount point. It is created as root, before the chgrp that makes /app group-writable, because the stage otherwise runs as the devlake user. - Drop `COPY --from=build /go/bin /usr/bin`: mockery and swag are build tools and nothing at runtime uses them. - Dockerfile.local: one build step instead of build-plugin plus build-server, and copy only the binary. - build-server.sh: add -trimpath and -s -w. A DEVLAKE_DEBUG build keeps the symbols so delve still works. - Makefile: build, dev and godev no longer build plugins; build-plugin stays for sideload and core-only builds. - env.example: PLUGIN_DIR now documents the sideload directory. GO_PLUGINS keeps working: it reaches the registry generator, so it now selects which plugins are compiled in rather than which .so files are built. plugins-registry clears GOOS/GOARCH: the Dockerfile exports them to cross-compile the server, and `go run` would otherwise build the generator for the target platform and fail with "exec format error" on an amd64 build host.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #9288 (base branch
feat/single-binary), because it needs the generated plugin registry to have anything to link in.make buildnow meansbuild-server: the framework, the API and every plugin inplugins/alllink into a singlebin/lake. The image build stops producing.sofiles entirely.How the image builds now
buildmake all→build-plugin(47 ×go build -buildmode=plugin) thenbuild-servermake all→build-server→ onego build ./server/app/bin/lake+/app/bin/plugins/*/*.so/app/bin/lakeCOPY --from=build /app/bin /app/binCOPY --from=build /app/bin/lake /app/bin/lakeThe runtime stage gets an empty
/app/bin/pluginsas the sideload mount point. It is created as root, before thechgrp -R 0 /app, because the stage otherwise runs as thedevlakeuser and could not create it.Measured
So the Go payload goes from ~2.75 GB to ~107 MB. Against the current
apache/devlake:main_*image (3.06 GB, of which the:basestage is 306 MB) the image should land around ~415 MB, roughly 7× smaller. The base stage is now the largest single piece.Also in here
build-server.shgains-trimpathand-s -w. ADEVLAKE_DEBUG=1build keeps the symbols somake debugstill works.COPY --from=build /go/bin /usr/bin.mockeryandswagare build tools; nothing at runtime uses them.dev,godevandbuildno longer build plugins.build-pluginstays for sideload and core-only builds.plugins-registryclearsGOOS/GOARCH.backend/Dockerfileexports them to cross-compile the server, andgo runwould otherwise build the generator for the target platform and fail withexec format erroron an amd64 host building the arm64 image.GO_PLUGINSkeeps working: it reaches the registry generator, so it now chooses which plugins are compiled in rather than which.sofiles are built.GO_PLUGINS=noneyields a core-only binary.Does this close any open issues?
Part of #9276.
Other Information
Verified locally:
The arm64 image build itself is not exercised on PRs (
.github/workflows/build.ymlruns onmainand tags), so that path needs a careful look at merge time.