Skip to content

fix(cli): keep multiline function env working on restarts (CLI-3131) #7607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
7ttp wants to merge 2 commits into
base: develop
Choose a base branch
from
+129 −207
Open

fix(cli): keep multiline function env working on restarts (CLI-3131) #7607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14b6c1f
fix(cli): keep multiline function env working on restarts (CLI-3131)
7ttp Nov 12, 2026
f821aa0
fix(cli): only remove legacy multiline-env dir on bring-up
7ttp Nov 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions apps/cli/src/commands/functions/serve/SIDE_EFFECTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,12 @@

## Files Written

| Path | Format | When |
| ---------------------------------------------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `~/.supabase/telemetry.json` | JSON | always, at command exit via `Effect.ensuring` |
| `<workdir>/supabase/.temp/start-secrets/supabase_edge_runtime_<project>/env/docker.env` | dotenv | per start, when single-line container env exists; passed via `--env-file`; mode `0600`; removed after the run |
| `<workdir>/supabase/.temp/start-secrets/supabase_edge_runtime_<project>/multiline-env/…` | shell + raw | per start, only when an env value contains a newline; bind-mounted read-only into the container; mode `0600`; removed after the run |
| Path | Format | When |
| --------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------- |
| `~/.supabase/telemetry.json` | JSON | always, at command exit via `Effect.ensuring` |
| `<workdir>/supabase/.temp/start-secrets/supabase_edge_runtime_<project>/env/docker.env` | dotenv | per start, when single-line container env exists; passed via `--env-file`; mode `0600`; removed after the run |

The env files hold secrets (JWT secret, anon/service-role keys, JWKS), so they are
The env file holds secrets (JWT secret, anon/service-role keys, JWKS), so it is
written owner-only (`0600`, in `0700` directories) under the project's own
`supabase/.temp/` (gitignored) — a deterministic, persistent path rather than
`os.tmpdir()`, so `supabase stop` and a failed-start rollback can reclaim it via
Expand Down Expand Up @@ -176,7 +175,7 @@ per-function `import_map.json` is not read and does not fail the command.
- A container that stops on its own with exit code `0`, or that is torn down by an external supervisor (exit `129`/`130`/`131`/`143`, or `15` from edge-runtime's own `SIGTERM` exit, e.g. `supabase stop` or `docker stop` in another terminal), or that is already gone by the time a follow-up inspect runs, all end the command successfully — each prints its own distinct line (see Output above) rather than the user-initiated `Stopped serving …` line, so scrollback can tell "I stopped it" from "the runtime walked out" or "a supervisor tore it down". In a `functions serve &` CI step this means a runtime that exits on its own does not fail the step; the distinct message is the only signal, and a downstream failure otherwise only surfaces later as connection-refused. Exit `137` (SIGKILL) fails the command: `State.OOMKilled` separates a container that hit its memory limit, which reports the memory-allocation remediation, from one killed by something the CLI cannot identify. A non-OOM `137` gets one re-inspect first — if `supabase stop` force-killed a runtime that ignored `SIGTERM` and the follow-up prune has already removed the container, the command still ends successfully instead of failing on a container that is already gone. Any other non-zero container exit fails the command; the error message includes the container id. Only a watched-file change restarts the container itself — none of these outcomes ever restart it.
- A `docker logs -f` re-attach (the daemon can close the stream while the container keeps running) resumes with `--since <last forwarded log timestamp>` instead of replaying the full log history, and is capped at 5 consecutive re-attaches that forward no new output; exceeding the cap fails the command with a tagged error instead of looping forever.
- On the log-stream path, the Docker-daemon-unreachable classification comes from a follow-up `docker container inspect` failure, not from `docker logs -f`'s own stderr text. That same follow-up inspect's `State.OOMKilled` also drives the memory-limit remediation on this path, so a `docker logs -f` failure over a container killed for exceeding its memory limit reports the same suggestion as the clean-exit case above, instead of a bare unattributable failure.
- The worker bootstrap template (`serve.main.ts`) is bundled into a single self-contained module with `jose` and the local path/status helpers inlined, so the edge-runtime worker boots without any network access (supabase/supabase#45570). The bundle is embedded at build time for shipped binaries and produced on demand (esbuild) when running from source. It is delivered into the created (not yet started) container as a `docker cp` stdin tar archive at `/root/index.ts` — never a single-file host bind mount, which materializes as an empty directory on daemons that cannot see the client's filesystem (remote `DOCKER_HOST`/Docker-context daemons, podman machines) and breaks bring-up with edge-runtime's "failed to determine entrypoint" (supabase/cli#6254). Only this bootstrap template is daemon-independent: user function sources, import maps, static files, and the multiline-env script directory (present only when an env value contains a newline) still arrive by host bind mounts, so they require a daemon that can see the project directory.
- The worker bootstrap template (`serve.main.ts`) is bundled into a single self-contained module with `jose` and the local path/status helpers inlined, so the edge-runtime worker boots without any network access (supabase/supabase#45570). The bundle is embedded at build time for shipped binaries and produced on demand (esbuild) when running from source. It is delivered into the created (not yet started) container as a `docker cp` stdin tar archive at `/root/index.ts` — never a single-file host bind mount, which materializes as an empty directory on daemons that cannot see the client's filesystem (remote `DOCKER_HOST`/Docker-context daemons, podman machines) and breaks bring-up with edge-runtime's "failed to determine entrypoint" (supabase/cli#6254). The multiline-env script and value files (present only when an env value contains a newline) ride in the same archive under `/root/.supabase/multiline-env/` as root-owned `0644` files that stay in the container until it is removed (supabase/cli#7603). User function sources, import maps, and static files still arrive by host bind mounts, so they require a daemon that can see the project directory.
- The aggregated bind mount list is pruned before `docker create`: a bind is dropped when another bind of the same mode already supplies the same content at the same container path — a file bind nested inside an already-bound read-only package directory would otherwise make the bootstrap `docker cp` fail with `destination "<container>:/" must be a directory` (supabase/supabase#50088). Pruned paths remain visible in the container through their covering parent mounts; the `--workdir` gate and the file-watch set are computed from the unpruned aggregate.
- Existing local values declared under an import map's `scopes` are explicit read-only Docker mounts and may resolve outside the nearest Git root; each distinct out-of-root host path prints one `WARN` during bring-up, deduplicated across Functions sharing an import map. Such out-of-root mounts are excluded from the file-watch set per Function, so a scope target contributes no watch root of its own and cannot enlarge or destabilise the watcher; a path that another Function reaches through its ordinary binds is still watched. Other file-valued binds are watched through their immediate parent non-recursively, while directory binds remain recursive. Missing targets retain serve's existing skip behavior.
- **Spec-strict import-map key matching:** bind mounts are computed by the functions import scanner (`walkImportPaths`/`substituteImportMapValue`, shared with `functions deploy` and `start`'s Edge Runtime bring-up), which matches import-map keys per the import-maps spec Deno/edge-runtime implement — exact match, or prefix match only for a `/`-suffixed key, so a bare-key prefix does not match and bind mounts are limited to spec-matching keys; an unwalkable target (`ENOTDIR` — a value routed through a file) is skipped with a `WARN`.
156 changes: 58 additions & 98 deletions apps/cli/src/commands/functions/serve/serve.integration.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ vi.mock("../../../shared/functions/functions-docker.ts", () =>
import("effect"),
import("@effect/platform-bun"),
import("../../../command-internal/docker-registry.ts"),
]).then(([actual, { Effect, FileSystem, Path }, { BunServices }, { getRegistryImageUrl }]) => ({
]).then(([actual, { Effect, FileSystem }, { BunServices }, { getRegistryImageUrl }]) => ({
...actual,
ensureDockerNetwork: (networkMode: string, projectId: string) =>
Effect.sync(() => {
Expand All @@ -157,41 +157,15 @@ vi.mock("../../../shared/functions/functions-docker.ts", () =>
runChildProcess: (command: string, args: ReadonlyArray<string>, options?: unknown) =>
Effect.gen(function* () {
const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
const envFile = args.flatMap((value, index) =>
args[index - 1] === "--env-file" ? [value] : [],
)[0];
const multilineEnvDir = args
.flatMap((value, index) => (args[index - 1] === "-v" ? [value] : []))
.find((value) => value.endsWith(":/root/.supabase/multiline-env:ro,Z"))
?.slice(0, -":/root/.supabase/multiline-env:ro,Z".length);
const enrichedOptions =
envFile === undefined && multilineEnvDir === undefined
envFile === undefined
? options
: {
...(typeof options === "object" && options !== null ? options : {}),
...(envFile === undefined
? {}
: { envFileContents: yield* fs.readFileString(envFile) }),
...(multilineEnvDir === undefined
? {}
: {
multilineEnvScript: yield* fs.readFileString(
path.join(multilineEnvDir, "multiline-env.sh"),
),
multilineEnvFiles: Object.fromEntries(
yield* Effect.forEach(
(yield* fs.readDirectory(path.join(multilineEnvDir, "values"))).filter(
(name) => name.startsWith("env-"),
),
(name) =>
Effect.map(
fs.readFileString(path.join(multilineEnvDir, "values", name)),
(contents): readonly [string, string] => [name, contents],
),
),
),
}),
envFileContents: yield* fs.readFileString(envFile),
};
deployMockState.runCalls.push({ command, args: [...args], options: enrichedOptions });
const result = deployMockState.runHandler?.(command, args, options) ?? {
Expand Down Expand Up @@ -1098,7 +1072,7 @@ describe("functions serve integration", () => {
},
);

it.live("mounts multiline env values without placing their contents in docker argv", () => {
it.live("copies multiline env values into the container without a bind or argv", () => {
deployMockState.runHandler = (command, args) => {
if (command !== "docker") {
throw new Error(`unexpected process: ${command}`);
Expand All @@ -1118,29 +1092,8 @@ describe("functions serve integration", () => {
throw new Error(`unexpected docker args: ${args.join(" ")}`);
};

let multilineEnvDirWhenLogsStarted: string | undefined;
let multilineEnvDirExistedWhenLogsStarted = false;
const childSpawner = mockDockerLogSpawner([
{
exitCode: 1,
stderr: "error running container: exit 1",
onSpawn: () =>
Effect.gen(function* () {
const fs = yield* FileSystem.FileSystem;
const dockerRun = deployMockState.runCalls.find(
(call) => call.command === "docker" && call.args[0] === "create",
);
if (dockerRun === undefined) {
throw new Error("expected docker create call before docker logs spawn");
}
multilineEnvDirWhenLogsStarted = extractFlagValues(dockerRun.args, "-v")
.find((value) => value.endsWith(":/root/.supabase/multiline-env:ro,Z"))
?.slice(0, -":/root/.supabase/multiline-env:ro,Z".length);
multilineEnvDirExistedWhenLogsStarted =
multilineEnvDirWhenLogsStarted !== undefined &&
(yield* fs.exists(multilineEnvDirWhenLogsStarted));
}).pipe(Effect.provide(BunServices.layer), Effect.orDie),
},
{ exitCode: 1, stderr: "error running container: exit 1" },
]);

const multilineValue = ["-----BEGIN KEY-----", "EOF_ENV_0", "line-3", "-----END KEY-----"].join(
Expand Down Expand Up @@ -1176,40 +1129,52 @@ describe("functions serve integration", () => {
expect(dockerRun.args.join(" ")).not.toContain(multilineValue);
expect(dockerRun.args.join(" ")).not.toContain("EOF_ENV_0");

const multilineBind = extractFlagValues(dockerRun.args, "-v").find((value) =>
value.endsWith(":/root/.supabase/multiline-env:ro,Z"),
);
expect(multilineBind).toBeDefined();
if (multilineBind === undefined) {
throw new Error("expected multiline env bind");
}
expect(
extractFlagValues(dockerRun.args, "-v").some((bind) =>
bind.includes("/root/.supabase/multiline-env"),
),
).toBe(false);

const script =
Predicate.hasProperty(dockerRun.options, "multilineEnvScript") &&
Predicate.isString(dockerRun.options.multilineEnvScript)
? dockerRun.options.multilineEnvScript
: undefined;
const files =
Predicate.hasProperty(dockerRun.options, "multilineEnvFiles") &&
isStringRecord(dockerRun.options.multilineEnvFiles)
? dockerRun.options.multilineEnvFiles
const cp = deployMockState.runCalls.find(
(call) => call.command === "docker" && call.args[0] === "cp",
);
const stdin =
cp !== undefined && Predicate.hasProperty(cp.options, "stdin")
? cp.options.stdin
: undefined;

expect(script).toBeDefined();
expect(files).toBeDefined();
const isCpArchiveStream = (value: unknown): value is Stream.Stream<Uint8Array> =>
Stream.isStream(value);
if (!isCpArchiveStream(stdin)) return yield* Effect.die("docker cp stdin was not a stream");
const archiveBytes = (yield* Stream.runCollect(stdin))[0];
if (!(archiveBytes instanceof Uint8Array)) {
return yield* Effect.die("docker cp stdin did not contain archive bytes");
}
const files = yield* Effect.promise(() => new Bun.Archive(archiveBytes).files());
const readEntry = (name: string) => {
const file = files.get(name);
return file === undefined ? Effect.void : Effect.promise(() => file.text());
};
const script = yield* readEntry("root/.supabase/multiline-env/multiline-env.sh");
expect(script).toContain(
'MULTILINE_SECRET="$(cat /root/.supabase/multiline-env/values/env-0; printf x)"',
);
expect(script).toContain('export MULTILINE_SECRET="${MULTILINE_SECRET%x}"');
expect(script).not.toContain(multilineValue);
expect(script).not.toContain("EOF_ENV_0");
expect(files?.["env-0"]).toBe(multilineValue);
expect(multilineEnvDirWhenLogsStarted).toBeDefined();
if (multilineEnvDirWhenLogsStarted === undefined) {
throw new Error("expected multiline env dir when docker logs started");
}
expect(multilineEnvDirExistedWhenLogsStarted).toBe(true);
expect(yield* fs.exists(multilineEnvDirWhenLogsStarted)).toBe(false);
expect(yield* readEntry("root/.supabase/multiline-env/values/env-0")).toBe(multilineValue);
expect(files.has("root/index.ts")).toBe(true);
expect(dockerRun.args.at(-1)).toContain(". /root/.supabase/multiline-env/multiline-env.sh\n");
expect(
yield* fs.exists(
path.join(
tempRoot.current,
"supabase",
".temp",
"start-secrets",
"supabase_edge_runtime_test-project",
),
),
).toBe(false);
}).pipe(Effect.provide(BunServices.layer));
});

Expand All @@ -1235,13 +1200,6 @@ describe("functions serve integration", () => {
throw new Error(`unexpected docker args: ${args.join(" ")}`);
};

const childSpawner = mockDockerLogSpawner([
{
exitCode: 1,
stderr: "error running container: exit 1",
},
]);

return Effect.gen(function* () {
const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
Expand Down Expand Up @@ -1273,25 +1231,27 @@ describe("functions serve integration", () => {
"stale secret\n",
);

let staleDirExistedAfterBringUp: boolean | undefined;
const childSpawner = mockDockerLogSpawner([
{
exitCode: 1,
stderr: "error running container: exit 1",
onSpawn: () =>
fs.exists(staleMultilineEnvDir).pipe(
Effect.map((exists) => {
staleDirExistedAfterBringUp = exists;
}),
Effect.orDie,
),
},
]);
const { layer } = setupServe({ childSpawner });

const error = yield* functionsServe(baseFlags()).pipe(Effect.provide(layer), Effect.flip);
expect(error).toBeInstanceOf(Error);

expect(staleDirExistedAfterBringUp).toBe(false);
expect(yield* fs.exists(staleMultilineEnvDir)).toBe(false);

const dockerRun = deployMockState.runCalls.find(
(call) => call.command === "docker" && call.args[0] === "create",
);
expect(dockerRun).toBeDefined();
if (dockerRun === undefined) {
throw new Error("expected docker create call");
}
expect(
extractFlagValues(dockerRun.args, "-v").some((value) =>
value.endsWith(":/root/.supabase/multiline-env:ro,Z"),
),
).toBe(false);
}).pipe(Effect.provide(BunServices.layer));
},
);
Expand Down
Loading
Loading