feat(compose): support additional compose files (-f a -f b) - #5457
OrtegaMatias wants to merge 2 commits into
Conversation
Closes the follow-ups left open on Dokploy#1727 after it was closed as 'already supported via custom command': the custom-command workaround only patches the deploy command, so Dokploy's own service discovery (getComposePath/loadDockerCompose/readComposeFile) still only reads the primary composePath. Any service that only exists in the second file is invisible to the Domains picker and collision checks, exactly what was reported in the thread after the close. - compose.composePathAdditional: text[] column (new migration), mirrors the existing watchPaths array field. - domain.ts: loadDockerCompose/loadDockerComposeRemote now read every additional file and shallow-merge their services on top of the primary one via the new mergeComposeSpecifications, so anything that reads service names through loadServices (Domains picker, collision checks) sees the full merged set - not just the primary file. - builders/compose.ts: the generated deploy command now appends '-f <path>' for each additional file, same as 'docker compose -f a.yml -f b.yml up'. This is the native alternative to hand-writing a Command override for this exact case. - UI: 'Additional Compose Files' field on the GitHub provider form (compose/general/generic/save-github-provider-compose.tsx), same tag-input pattern already used for Watch Paths. Scoped to GitHub for this PR since it is the primary provider I could verify end-to-end; the other four provider forms (gitlab/gitea/bitbucket/git) carry an identical composePath+watchPaths block today and would need the same ~90 line addition, happy to extend in this PR or a fast-follow if preferred. - Test: mergeComposeSpecifications (6 cases - null handling, additive services, key-level override, multi-file ordering). Verified against a real 34-service compose file (base + a ~200 line override adding build:/volumes:/profiles: to 22 services): 'docker compose -f base.yml -f override.yml config' resolves correctly with the generated command, and loadServices returns all 34 service names instead of only the base file's set. typecheck (server + dokploy) and biome check both clean; existing should-deploy/traefik test suites untouched.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const { stdout, stderr } = await execAsyncRemote( | ||
| compose.serverId as string, | ||
| `cat ${path}`, | ||
| ); |
There was a problem hiding this comment.
A user permitted to update a Compose resource can save override.yml; id # in composePathAdditional and trigger cached service discovery. The resulting cat .../override.yml; id # executes the second command under the managed server’s SSH account without deploying anything. Although the primary path already had this unsafe pattern, this change extends it to the newly accepted additional paths. Quote the complete path with quote([path]) or read it through SFTP.
How this was verified: The update schema accepts unrestricted strings, the update persists them, and cached service discovery passes each joined path unescaped to the SSH command executor.
| const { stdout, stderr } = await execAsyncRemote( | |
| compose.serverId as string, | |
| `cat ${path}`, | |
| ); | |
| const { stdout, stderr } = await execAsyncRemote( | |
| compose.serverId as string, | |
| `cat ${quote([path])}`, | |
| ); |
| merged.services[serviceName] = { | ||
| ...(merged.services[serviceName] ?? {}), | ||
| ...serviceDefinition, | ||
| }; |
There was a problem hiding this comment.
This merge is also used for deployment, not just service discovery. If the base service has environment: { DATABASE_URL: "...", LOG_LEVEL: "info" } and the override contains only environment: { LOG_LEVEL: "debug" }, this spread drops DATABASE_URL. writeDomainsToCompose then overwrites the primary file with that result before Docker reads either file, so Docker cannot recover the base value. Nested build/deploy settings and additive collections have the same problem. Preserve Compose merge semantics before writing the deployment model, or keep this discovery-only approximation out of the write-back path.
Knowledge Base Used: Build and Compose workflows
| const merged: ComposeSpecification = { ...validSpecs[0] }; | ||
| merged.services = { ...(validSpecs[0]?.services ?? {}) }; | ||
|
|
||
| for (const spec of validSpecs.slice(1)) { | ||
| for (const [serviceName, serviceDefinition] of Object.entries( | ||
| spec.services ?? {}, |
There was a problem hiding this comment.
An override that adds a database service using data:/var/lib/postgresql/data and declares volumes: { data: {} } produces a rewritten primary file referencing an undeclared volume: this merge retains the service but discards the override’s root declarations. Docker Compose deployment recovers the declaration by reading the additional file, but Start reads only the rewritten primary and fails. Stack deployment also reads only the primary and fails immediately. Merge additional root volumes, networks, configs, and secrets into the prepared model instead of retaining only the first specification’s root properties.
Knowledge Base Used: Build and Compose workflows
| const additionalFileFlags = (compose.composePathAdditional ?? []) | ||
| .filter((additionalPath): additionalPath is string => | ||
| Boolean(additionalPath?.trim()), | ||
| ) | ||
| .map((additionalPath) => `-f ${quote([additionalPath])} `) | ||
| .join(""); | ||
| command = `compose -p ${quote([appName])} ${projectDirectoryFlag}${envFileFlag}-f ${quote([path])} ${additionalFileFlags}up -d --build --remove-orphans${pullFlag}`; |
There was a problem hiding this comment.
With randomize enabled and a suffix set, preparation renames merged service web to web-<suffix> and writes it into the primary file. The additional file still declares web, so appending it here makes Docker treat the override as another service rather than applying it to the renamed service. An environment-only override then fails because the extra service has neither an image nor a build; a complete definition can start an unintended duplicate. Merge the source files before transformation and deploy the prepared model without replaying the original overrides.
Knowledge Base Used: Build and Compose workflows
| } | ||
| return null; | ||
|
|
||
| return mergeComposeSpecifications([primary, ...additionalPaths.map(loadOne)]); |
There was a problem hiding this comment.
When an enabled patch supplies the primary Compose file, the Domains picker now offers additional-file services, but deployment preparation loses them. addDomainToCompose loads this merged result and then replaces it entirely with applyComposeFilePatch(compose). A domain attached to an override-only service consequently fails with “does not exist in the compose,” even though the configured additional file contains that service. Apply the primary-file patch before merging additional specifications so discovery and deployment preparation use the same service set. The remote loader has the same ordering problem.
Knowledge Base Used: Build and Compose workflows
| return primary; | ||
| } | ||
|
|
||
| const additional = await Promise.all(additionalPaths.map(loadOne)); |
There was a problem hiding this comment.
Each additional file opens a separate SSH connection, and this Promise.all starts every connection simultaneously. A longer list creates a burst of SSH handshakes on every service-discovery or preparation request, while connection failures are silently treated as absent specifications. As a non-blocking improvement, use bounded concurrency or a shared connection, and surface failures for explicitly configured files rather than returning an incomplete service set.
Addresses the review on this PR (5 P1 + 1 P2):
- Command injection: additional-file paths were interpolated into
`cat ${path}` unescaped when read over SSH. Every path is now quoted
with shell-quote before it reaches a shell command.
- Shallow merge deletes base configuration / additional root
declarations are lost: the hand-rolled per-service spread only
merged `services`, dropped root-level volumes/networks/configs/
secrets declared only in an additional file, and replaced nested
maps (environment, labels, deploy...) instead of merging them.
loadDockerCompose/loadDockerComposeRemote now resolve every file
through `docker compose -f a -f b ... config --no-interpolate`,
Compose's own merge engine, instead of reimplementing it in JS.
--no-interpolate keeps ${VAR} references literal so env resolution
still happens at actual `docker compose up` time.
- Primary patches discard additional services: addDomainToCompose
loaded the merge, then unconditionally replaced it with the patched
primary alone, dropping every additional-file service. A pending
patch is now materialized to a temp file and merged with the
additional files the same way the real primary would be, then
cleaned up.
- Overrides break randomized deployments: writeDomainsToCompose
already bakes the full merge into the primary file before
`docker compose up` runs, so createCommand no longer re-adds
`-f additionalFile` flags at deploy time -- re-layering them a
second time duplicated array-typed keys (ports/volumes) and broke
`randomize` (merged primary has suffixed names the un-suffixed
additional file no longer matches).
- Unbounded connections / swallowed failures: additional-file
existence is now resolved with a single batched remote probe
instead of one SSH connection per file via Promise.all, and a
genuine docker compose config failure (auth, malformed YAML) now
throws instead of being silently treated as an empty spec.
mergeComposeSpecifications is removed; its test file is replaced with
an integration test that exercises loadDockerCompose against real
compose files and the real docker compose binary (services merge
across files, environment deep-merges, root volumes only in an
additional file survive, a not-yet-pushed additional file is skipped,
a missing primary returns null).
|
Thanks for the review -- all 6 points addressed in 2760ac8:
|
1 participant
Problem
#1727 asked for native support for multiple compose files (
docker compose -f a -f b). It was closed as "already supported via custom command" — but that workaround only patches the deploy command. Two follow-up comments on the same thread after the close report what's still broken:Root cause:
getComposePath/loadDockerCompose/readComposeFileindomain.tsonly ever resolvecompose.composePath(the single UI field).loadServices(used by the Domains picker and collision checks) calls straight into that, so any service defined only in the second-ffile is invisible — even if the custom command correctly builds/starts it.What this adds
compose.composePathAdditional: text[]— new column, same shape as the existingwatchPathsarray.domain.ts:loadDockerCompose/loadDockerComposeRemotenow read every additional file and shallow-merge theirserviceson top of the primary one (newmergeComposeSpecifications), soloadServicessees the full set.builders/compose.ts: the generated deploy command appends-f <path>per additional file — a native alternative to hand-writing a Command override for exactly this case.mergeComposeSpecifications(null handling, additive services, key-level override across files, ordering, mixed-null input).Scope note
I scoped the UI to the GitHub provider form only for this PR (it's the one I could verify end-to-end against a real 34-service compose file). GitLab/Gitea/Bitbucket/Git carry an identical
composePath+watchPathsblock today and would need the same ~90-line addition — happy to add all four here if you'd rather land it in one PR, or as a fast-follow.Verification
docker-compose.yml(34 services) + a ~200-line override addingbuild:/volumes:/profiles:to 22 of them.docker compose -f base.yml -f override.yml configresolves correctly with the command this PR generates.pnpm --filter=@dokploy/server typecheckandpnpm --filter=dokploy typecheck: clean.biome check: clean on all touched files.drizzle-kit generate(not hand-written) against the current schema.This PR should not merge until remote path execution is secured and multi-file deployment preparation preserves a complete, consistently transformed configuration.
Summary
Reviews (1) · Last reviewed commit: "feat(compose): support additional compos..."