From 21793bea5aefe34e6f02d8de7acf2d94cd84b36b Mon Sep 17 00:00:00 2001 From: Klesh Wong Date: Sat, 31 Oct 2026 11:59:22 +0800 Subject: [PATCH 1/3] docs: describe the single binary and plugin sideloading `make build` now produces one `bin/lake` with the framework, the API and every official plugin compiled in, and a deployed binary still loads extra `.so` plugins from PLUGIN_DIR. - Rewrite the "Compiling" section, which was rendered as a code block and documented two things that do not exist: `PLUGIN=` (the variable is DEVLAKE_PLUGINS) and `make build-worker`. - Add a "Sideloading plugins" section: where the .so files go, why they must be built with the matching toolchain, and that a plugin which fails to load is skipped rather than taking the server down. - Document PLUGIN_DIR in the environment variables reference, including the container mount path and the override warning. --- docs/DeveloperManuals/DeveloperSetup.md | 36 ++++++++++++++++++++++--- docs/GettingStarted/Environment.md | 14 ++++++++++ 2 files changed, 46 insertions(+), 4 deletions(-) diff --git a/docs/DeveloperManuals/DeveloperSetup.md b/docs/DeveloperManuals/DeveloperSetup.md index 6d0bbd4bac8..5d41f9386c2 100644 --- a/docs/DeveloperManuals/DeveloperSetup.md +++ b/docs/DeveloperManuals/DeveloperSetup.md @@ -121,10 +121,38 @@ For provisioning, customizing, and creating dashboards, please refer to our [Gra ## Compiling - - Compile all plugins: `make build-plugin` - - Compile specific plugins: `PLUGIN= make build-plugin` - - Compile server: `make build` - - Compile worker: `make build-worker` +`make build` produces a single `bin/lake` binary: the framework, the API and every official plugin are compiled into it. + +```sh +cd backend +make build +``` + +While iterating you usually do not want all of them. `DEVLAKE_PLUGINS` takes a comma-separated list: + +```sh +DEVLAKE_PLUGINS=jira,gitlab make build +``` + +To build plugins as shared objects instead — to sideload them into an existing binary, or to ship a server without any plugin compiled in — use: + +```sh +make build-plugin # every plugin +DEVLAKE_PLUGINS=jira,gitlab make build-plugin # a subset +``` + +The output lands in `bin/plugins//.so`. + +## Sideloading plugins + +A released `lake` binary already contains every official plugin, and it still accepts extras. Any `.so` found under `PLUGIN_DIR` (default `bin/plugins`) is opened at startup and registered on top of the compiled-in set, so you can add a custom plugin — or replace an official one — without rebuilding DevLake. + +Two things to know before you try it: + +- The `.so` has to be built with `go build -buildmode=plugin` against the **same DevLake version, Go toolchain, build flags and dependency versions** as the binary that loads it. Go rejects a plugin whose copy of a shared package differs, so build it in the matching `lake-builder` image rather than on your host. +- A plugin that fails to load is logged and skipped. The server keeps running without it, so check the log after adding one. + +The plugin entry point (`PluginEntry`, implementing `plugin.PluginMeta`) and the interfaces a plugin can implement are described in the [plugin development manual](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md). ## References diff --git a/docs/GettingStarted/Environment.md b/docs/GettingStarted/Environment.md index cfc13ad879c..991319ecce2 100644 --- a/docs/GettingStarted/Environment.md +++ b/docs/GettingStarted/Environment.md @@ -8,6 +8,20 @@ This document explains how to set environment variables for Apache DevLake and w ## Environment Variables +### PLUGIN_DIR + +Directory scanned at startup for shared-object plugins (`.so`). The released binary already contains every official plugin; anything found here is loaded on top of that set, which makes this the directory to use for a custom or third-party plugin. + +| Environment Variable | Description | Default Value | +| -------------------- | ------------------------------------------------------------------------------------ | ------------- | +| `PLUGIN_DIR` | Directory scanned for sideloaded `.so` plugins. A missing directory is not an error. | `bin/plugins` | + +- In the Docker image, mount your plugins over `/app/bin/plugins`. +- A plugin that fails to load — usually a toolchain or dependency mismatch — is logged and skipped; the server keeps running without it. +- A sideloaded plugin whose name matches an official plugin overrides it, and the override is logged as a warning. + +See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for how to build one. + ### ENABLE_SUBTASKS_BY_DEFAULT This environment variable is used to enable or disable the execution of subtasks. From c144949ad7779fc601753aa7696014f03c4f1da7 Mon Sep 17 00:00:00 2001 From: Klesh Wong Date: Sat, 31 Oct 2026 12:54:29 +0800 Subject: [PATCH 2/3] docs: show how to sideload a plugin into the Docker image The image ships one binary with every official plugin, so adding a plugin means mounting a directory over /app/bin/plugins rather than rebuilding it. Adds an FAQ entry with the compose snippet, the naming/override rule and the "build it in the matching lake-builder image" constraint. --- docs/GettingStarted/DockerComposeSetup.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/GettingStarted/DockerComposeSetup.md b/docs/GettingStarted/DockerComposeSetup.md index f349a660374..3b51707585b 100644 --- a/docs/GettingStarted/DockerComposeSetup.md +++ b/docs/GettingStarted/DockerComposeSetup.md @@ -174,6 +174,21 @@ config-ui: 6. Final step: `docker compose up -d` +### Can I add my own plugin to the DevLake image? + +The `apache/devlake` image ships a single binary that already contains every official plugin, so there is nothing to install for the supported data sources. If you need one that is not included, build it as a shared object and mount it into the container: + +```yaml +devlake: + ... + volumes: + - ./plugins:/app/bin/plugins +``` + +The directory is scanned at startup and each `.so` is registered on top of the built-in plugins; one that reuses an existing plugin's name replaces it. A plugin that fails to load is logged and skipped, and the server keeps running without it. + +The plugin has to be built against the same DevLake release and Go toolchain as the running image, or it will be rejected at load time — build it in the matching `lake-builder` image rather than on your host. See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the details. + ## Troubleshooting If you run into any problem, please check the [Troubleshooting](/Troubleshooting/Installation.md) or [create an issue](https://github.com/apache/devlake/issues) From 0598746545fe25ad9006a8a23e2fc69f39b461fa Mon Sep 17 00:00:00 2001 From: Klesh Wong Date: Sun, 1 Nov 2026 19:43:01 +0800 Subject: [PATCH 3/3] docs: explain how plugins are compiled into the server binary The build no longer produces one .so per plugin for the release: a generated registry links every official plugin into bin/lake, and the image ships that binary alone. Document the model where plugin authors and operators will look for it. - Architecture: state that official plugins are compiled into the lake binary, and add a "Plugin packaging" section covering the generated registry, DEVLAKE_PLUGINS/GO_PLUGINS selection, Python plugins, and how PLUGIN_DIR sideloading fits on top. - Plugin implementation: describe how a plugin gets into the registry, how to regenerate it, the core-only builds, and when to sideload instead. - Developer setup: note the checked-in registry and the CI drift check, the debug build flag, and document how to build a core-only server. --- docs/DeveloperManuals/DeveloperSetup.md | 20 +++++++++++++++++-- docs/DeveloperManuals/PluginImplementation.md | 16 +++++++++++++++ docs/GettingStarted/DockerComposeSetup.md | 2 +- docs/GettingStarted/Environment.md | 6 +++--- docs/Overview/Architecture.md | 10 +++++++++- 5 files changed, 47 insertions(+), 7 deletions(-) diff --git a/docs/DeveloperManuals/DeveloperSetup.md b/docs/DeveloperManuals/DeveloperSetup.md index 5d41f9386c2..471adf1b488 100644 --- a/docs/DeveloperManuals/DeveloperSetup.md +++ b/docs/DeveloperManuals/DeveloperSetup.md @@ -134,14 +134,30 @@ While iterating you usually do not want all of them. `DEVLAKE_PLUGINS` takes a c DEVLAKE_PLUGINS=jira,gitlab make build ``` -To build plugins as shared objects instead — to sideload them into an existing binary, or to ship a server without any plugin compiled in — use: +That variable also decides what the registry generator writes to `backend/plugins/all/all.go`, which is checked in. Run `make plugins-registry` without `DEVLAKE_PLUGINS` before committing, or CI's drift check will fail. + +Release builds strip symbols. Set `DEVLAKE_DEBUG=1` to keep them, and use `make debug` to start the server under `dlv`. + +To build plugins as shared objects instead — for sideloading, or for a server without any plugin compiled in — use: ```sh make build-plugin # every plugin DEVLAKE_PLUGINS=jira,gitlab make build-plugin # a subset ``` -The output lands in `bin/plugins//.so`. +The output lands in `bin/plugins//.so`; set `PLUGIN_DIR` to write it somewhere else. + +A server with no plugin compiled in is built with an empty registry: + +```sh +DEVLAKE_PLUGINS=none make build +``` + +The `devlake_external_plugins` build tag selects the same empty registry and makes the intent explicit: + +```sh +go build -tags devlake_external_plugins -o bin/lake ./server +``` ## Sideloading plugins diff --git a/docs/DeveloperManuals/PluginImplementation.md b/docs/DeveloperManuals/PluginImplementation.md index 2c3e1023062..df2835631e0 100644 --- a/docs/DeveloperManuals/PluginImplementation.md +++ b/docs/DeveloperManuals/PluginImplementation.md @@ -12,3 +12,19 @@ support developers who prefer it using it. PyDevLake is relatively brand new, an The Go development manual can be found [here](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md). This manual also covers the framework in detail. The Python development manual can be found [here](https://github.com/apache/devlake/blob/main/backend/python/README.md). + +## How a plugin becomes part of DevLake + +DevLake ships as a single `lake` binary: the framework, the API server and the official plugins are linked at build time. A Go plugin joins that binary when its package lives under `backend/plugins/` and its `impl` package exports a type implementing [`plugin.PluginMeta`](https://github.com/apache/devlake/blob/main/backend/core/plugin/plugin_meta.go). + +Registration is generated rather than hand-written. `make plugins-registry` scans `plugins/*/impl`, writes `backend/plugins/all/all.go`, and the server blank-imports that package so each listed plugin registers itself on startup. + +- `make build` compiles in every plugin and writes `bin/lake`. +- `DEVLAKE_PLUGINS=jira,gitlab make build` compiles in a subset. The variable also drives the generator, so run `make plugins-registry` without it before you commit. +- `DEVLAKE_PLUGINS=none make build` produces a binary with no plugin compiled in. + +When a plugin is added, removed or renamed, regenerate the registry and commit the result; CI fails when `plugins/all/all.go` is out of date. + +A plugin developed outside this repository is not compiled in. Build it as a Go shared object — `make build-plugin`, or `go build -buildmode=plugin` — and place it under `PLUGIN_DIR`. The server loads it at startup on top of the compiled-in plugins. See [Sideloading plugins](./DeveloperSetup.md#sideloading-plugins) for the toolchain and version requirements. + +Python plugins do not go through this registry. They are discovered at runtime under `REMOTE_PLUGIN_DIR`. diff --git a/docs/GettingStarted/DockerComposeSetup.md b/docs/GettingStarted/DockerComposeSetup.md index 3b51707585b..476ae20b9bd 100644 --- a/docs/GettingStarted/DockerComposeSetup.md +++ b/docs/GettingStarted/DockerComposeSetup.md @@ -185,7 +185,7 @@ devlake: - ./plugins:/app/bin/plugins ``` -The directory is scanned at startup and each `.so` is registered on top of the built-in plugins; one that reuses an existing plugin's name replaces it. A plugin that fails to load is logged and skipped, and the server keeps running without it. +The directory is scanned at startup and each `.so` is registered on top of the built-in plugins; one that reuses an existing plugin's name replaces it. A plugin that fails to load is logged and skipped, and the server keeps running without it. `PLUGIN_DIR` already defaults to `/app/bin/plugins` in the image, so the mount is all you need. The plugin has to be built against the same DevLake release and Go toolchain as the running image, or it will be rejected at load time — build it in the matching `lake-builder` image rather than on your host. See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the details. diff --git a/docs/GettingStarted/Environment.md b/docs/GettingStarted/Environment.md index 991319ecce2..7b88d85ec1f 100644 --- a/docs/GettingStarted/Environment.md +++ b/docs/GettingStarted/Environment.md @@ -12,9 +12,9 @@ This document explains how to set environment variables for Apache DevLake and w Directory scanned at startup for shared-object plugins (`.so`). The released binary already contains every official plugin; anything found here is loaded on top of that set, which makes this the directory to use for a custom or third-party plugin. -| Environment Variable | Description | Default Value | -| -------------------- | ------------------------------------------------------------------------------------ | ------------- | -| `PLUGIN_DIR` | Directory scanned for sideloaded `.so` plugins. A missing directory is not an error. | `bin/plugins` | +| Environment Variable | Description | Default Value | +| -------------------- | -------------------------------------------------------------------------------------- | ------------- | +| `PLUGIN_DIR` | Directory scanned for sideloaded `.so` plugins. A missing directory is not an error. | `bin/plugins` | - In the Docker image, mount your plugins over `/app/bin/plugins`. - A plugin that fails to load — usually a toolchain or dependency mismatch — is logged and skipped; the server keeps running without it. diff --git a/docs/Overview/Architecture.md b/docs/Overview/Architecture.md index b0eb280c296..630b629cb66 100755 --- a/docs/Overview/Architecture.md +++ b/docs/Overview/Architecture.md @@ -16,9 +16,17 @@ A DevLake installation typically consists of the following components: - API Server: The main programmatic interface of DevLake. - Runner: The runner does all the heavy-lifting for executing tasks. In the default DevLake installation, it runs within the API Server, but DevLake provides a temporal-based runner (beta) for production environments. - Database: The database stores both DevLake's metadata and user data collected by data pipelines. DevLake supports MySQL and PostgreSQL as of v0.11. -- Plugins: Plugins enable DevLake to collect and analyze dev data from any DevOps tools with an accessible API. DevLake community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself. +- Plugins: Plugins enable DevLake to collect and analyze dev data from any DevOps tools with an accessible API. The official plugins are compiled into the `lake` binary alongside the framework, the API server and the runner, so a standard installation runs a single executable. DevLake community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself. - Dashboards: Dashboards deliver data and insights to DevLake users. A dashboard is simply a collection of SQL queries along with corresponding visualization configurations. DevLake's official dashboard tool is Grafana and pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to swap for their own choice of dashboard/BI tool if desired. +## Plugin packaging + +DevLake ships as one `lake` binary. At build time a generated registry in `backend/plugins/all` imports every plugin under `backend/plugins` and registers it with the framework, so the framework, the API and all official plugins are linked into that single executable. The container image therefore contains no plugin `.so` files. + +`DEVLAKE_PLUGINS` selects which plugins are compiled in: leave it unset for all of them, pass a comma-separated list for a subset, or `none` for a core-only binary. When the container image is built, the same selection is exposed as the `GO_PLUGINS` build argument. Python plugins are not part of this mechanism; they are discovered at runtime under `REMOTE_PLUGIN_DIR`. + +A plugin that is not part of the release can still be sideloaded. `PLUGIN_DIR` is scanned at startup and every shared object found there is registered on top of the compiled-in plugins: a `.so` with a new name adds a plugin, and one that reuses an official plugin's name replaces it. A file that fails to load — usually because it was built with a different toolchain or DevLake version — is logged and skipped instead of stopping the server. See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the build steps. + ## Dataflow