diff --git a/docs/DeveloperManuals/DeveloperSetup.md b/docs/DeveloperManuals/DeveloperSetup.md index 6d0bbd4bac8..471adf1b488 100644 --- a/docs/DeveloperManuals/DeveloperSetup.md +++ b/docs/DeveloperManuals/DeveloperSetup.md @@ -121,10 +121,54 @@ For provisioning, customizing, and creating dashboards, please refer to our [Gra ## Compiling - - Compile all plugins: `make build-plugin` - - Compile specific plugins: `PLUGIN= make build-plugin` - - Compile server: `make build` - - Compile worker: `make build-worker` +`make build` produces a single `bin/lake` binary: the framework, the API and every official plugin are compiled into it. + +```sh +cd backend +make build +``` + +While iterating you usually do not want all of them. `DEVLAKE_PLUGINS` takes a comma-separated list: + +```sh +DEVLAKE_PLUGINS=jira,gitlab make build +``` + +That variable also decides what the registry generator writes to `backend/plugins/all/all.go`, which is checked in. Run `make plugins-registry` without `DEVLAKE_PLUGINS` before committing, or CI's drift check will fail. + +Release builds strip symbols. Set `DEVLAKE_DEBUG=1` to keep them, and use `make debug` to start the server under `dlv`. + +To build plugins as shared objects instead — for sideloading, or for a server without any plugin compiled in — use: + +```sh +make build-plugin # every plugin +DEVLAKE_PLUGINS=jira,gitlab make build-plugin # a subset +``` + +The output lands in `bin/plugins//.so`; set `PLUGIN_DIR` to write it somewhere else. + +A server with no plugin compiled in is built with an empty registry: + +```sh +DEVLAKE_PLUGINS=none make build +``` + +The `devlake_external_plugins` build tag selects the same empty registry and makes the intent explicit: + +```sh +go build -tags devlake_external_plugins -o bin/lake ./server +``` + +## Sideloading plugins + +A released `lake` binary already contains every official plugin, and it still accepts extras. Any `.so` found under `PLUGIN_DIR` (default `bin/plugins`) is opened at startup and registered on top of the compiled-in set, so you can add a custom plugin — or replace an official one — without rebuilding DevLake. + +Two things to know before you try it: + +- The `.so` has to be built with `go build -buildmode=plugin` against the **same DevLake version, Go toolchain, build flags and dependency versions** as the binary that loads it. Go rejects a plugin whose copy of a shared package differs, so build it in the matching `lake-builder` image rather than on your host. +- A plugin that fails to load is logged and skipped. The server keeps running without it, so check the log after adding one. + +The plugin entry point (`PluginEntry`, implementing `plugin.PluginMeta`) and the interfaces a plugin can implement are described in the [plugin development manual](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md). ## References diff --git a/docs/DeveloperManuals/PluginImplementation.md b/docs/DeveloperManuals/PluginImplementation.md index 2c3e1023062..df2835631e0 100644 --- a/docs/DeveloperManuals/PluginImplementation.md +++ b/docs/DeveloperManuals/PluginImplementation.md @@ -12,3 +12,19 @@ support developers who prefer it using it. PyDevLake is relatively brand new, an The Go development manual can be found [here](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md). This manual also covers the framework in detail. The Python development manual can be found [here](https://github.com/apache/devlake/blob/main/backend/python/README.md). + +## How a plugin becomes part of DevLake + +DevLake ships as a single `lake` binary: the framework, the API server and the official plugins are linked at build time. A Go plugin joins that binary when its package lives under `backend/plugins/` and its `impl` package exports a type implementing [`plugin.PluginMeta`](https://github.com/apache/devlake/blob/main/backend/core/plugin/plugin_meta.go). + +Registration is generated rather than hand-written. `make plugins-registry` scans `plugins/*/impl`, writes `backend/plugins/all/all.go`, and the server blank-imports that package so each listed plugin registers itself on startup. + +- `make build` compiles in every plugin and writes `bin/lake`. +- `DEVLAKE_PLUGINS=jira,gitlab make build` compiles in a subset. The variable also drives the generator, so run `make plugins-registry` without it before you commit. +- `DEVLAKE_PLUGINS=none make build` produces a binary with no plugin compiled in. + +When a plugin is added, removed or renamed, regenerate the registry and commit the result; CI fails when `plugins/all/all.go` is out of date. + +A plugin developed outside this repository is not compiled in. Build it as a Go shared object — `make build-plugin`, or `go build -buildmode=plugin` — and place it under `PLUGIN_DIR`. The server loads it at startup on top of the compiled-in plugins. See [Sideloading plugins](./DeveloperSetup.md#sideloading-plugins) for the toolchain and version requirements. + +Python plugins do not go through this registry. They are discovered at runtime under `REMOTE_PLUGIN_DIR`. diff --git a/docs/GettingStarted/DockerComposeSetup.md b/docs/GettingStarted/DockerComposeSetup.md index f349a660374..476ae20b9bd 100644 --- a/docs/GettingStarted/DockerComposeSetup.md +++ b/docs/GettingStarted/DockerComposeSetup.md @@ -174,6 +174,21 @@ config-ui: 6. Final step: `docker compose up -d` +### Can I add my own plugin to the DevLake image? + +The `apache/devlake` image ships a single binary that already contains every official plugin, so there is nothing to install for the supported data sources. If you need one that is not included, build it as a shared object and mount it into the container: + +```yaml +devlake: + ... + volumes: + - ./plugins:/app/bin/plugins +``` + +The directory is scanned at startup and each `.so` is registered on top of the built-in plugins; one that reuses an existing plugin's name replaces it. A plugin that fails to load is logged and skipped, and the server keeps running without it. `PLUGIN_DIR` already defaults to `/app/bin/plugins` in the image, so the mount is all you need. + +The plugin has to be built against the same DevLake release and Go toolchain as the running image, or it will be rejected at load time — build it in the matching `lake-builder` image rather than on your host. See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the details. + ## Troubleshooting If you run into any problem, please check the [Troubleshooting](/Troubleshooting/Installation.md) or [create an issue](https://github.com/apache/devlake/issues) diff --git a/docs/GettingStarted/Environment.md b/docs/GettingStarted/Environment.md index cfc13ad879c..7b88d85ec1f 100644 --- a/docs/GettingStarted/Environment.md +++ b/docs/GettingStarted/Environment.md @@ -8,6 +8,20 @@ This document explains how to set environment variables for Apache DevLake and w ## Environment Variables +### PLUGIN_DIR + +Directory scanned at startup for shared-object plugins (`.so`). The released binary already contains every official plugin; anything found here is loaded on top of that set, which makes this the directory to use for a custom or third-party plugin. + +| Environment Variable | Description | Default Value | +| -------------------- | -------------------------------------------------------------------------------------- | ------------- | +| `PLUGIN_DIR` | Directory scanned for sideloaded `.so` plugins. A missing directory is not an error. | `bin/plugins` | + +- In the Docker image, mount your plugins over `/app/bin/plugins`. +- A plugin that fails to load — usually a toolchain or dependency mismatch — is logged and skipped; the server keeps running without it. +- A sideloaded plugin whose name matches an official plugin overrides it, and the override is logged as a warning. + +See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for how to build one. + ### ENABLE_SUBTASKS_BY_DEFAULT This environment variable is used to enable or disable the execution of subtasks. diff --git a/docs/Overview/Architecture.md b/docs/Overview/Architecture.md index b0eb280c296..630b629cb66 100755 --- a/docs/Overview/Architecture.md +++ b/docs/Overview/Architecture.md @@ -16,9 +16,17 @@ A DevLake installation typically consists of the following components: - API Server: The main programmatic interface of DevLake. - Runner: The runner does all the heavy-lifting for executing tasks. In the default DevLake installation, it runs within the API Server, but DevLake provides a temporal-based runner (beta) for production environments. - Database: The database stores both DevLake's metadata and user data collected by data pipelines. DevLake supports MySQL and PostgreSQL as of v0.11. -- Plugins: Plugins enable DevLake to collect and analyze dev data from any DevOps tools with an accessible API. DevLake community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself. +- Plugins: Plugins enable DevLake to collect and analyze dev data from any DevOps tools with an accessible API. The official plugins are compiled into the `lake` binary alongside the framework, the API server and the runner, so a standard installation runs a single executable. DevLake community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself. - Dashboards: Dashboards deliver data and insights to DevLake users. A dashboard is simply a collection of SQL queries along with corresponding visualization configurations. DevLake's official dashboard tool is Grafana and pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to swap for their own choice of dashboard/BI tool if desired. +## Plugin packaging + +DevLake ships as one `lake` binary. At build time a generated registry in `backend/plugins/all` imports every plugin under `backend/plugins` and registers it with the framework, so the framework, the API and all official plugins are linked into that single executable. The container image therefore contains no plugin `.so` files. + +`DEVLAKE_PLUGINS` selects which plugins are compiled in: leave it unset for all of them, pass a comma-separated list for a subset, or `none` for a core-only binary. When the container image is built, the same selection is exposed as the `GO_PLUGINS` build argument. Python plugins are not part of this mechanism; they are discovered at runtime under `REMOTE_PLUGIN_DIR`. + +A plugin that is not part of the release can still be sideloaded. `PLUGIN_DIR` is scanned at startup and every shared object found there is registered on top of the compiled-in plugins: a `.so` with a new name adds a plugin, and one that reuses an official plugin's name replaces it. A file that fails to load — usually because it was built with a different toolchain or DevLake version — is logged and skipped instead of stopping the server. See [Sideloading plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the build steps. + ## Dataflow