Skip to content

Serve grpc.health.v1.Health on the build session so BuildKit does not drop it - #847

Open
punkpeye wants to merge 1 commit into
apocas:masterfrom
punkpeye:serve-buildkit-session-health-check
Open

Serve grpc.health.v1.Health on the build session so BuildKit does not drop it#847
punkpeye wants to merge 1 commit into
apocas:masterfrom
punkpeye:serve-buildkit-session-health-check

Conversation

@punkpeye

@punkpeye punkpeye commented Sep 6, 2026

Copy link
Copy Markdown

Problem

BuildKit builds started through docker.buildImage(..., { version: '2' }) intermittently stall and never produce an image. Under concurrency it becomes the common case. The build hangs at the first vertex, usually [internal] load metadata for <base image>, and the daemon logs:

level=error msg="healthcheck failed fatally" error="session healthcheck failed fatally: Unimplemented: The server does not implement the method /grpc.health.v1.Health/Check"
...
level=info msg="fetch failed" error="no active session for <uuid>: context canceled"

Because the failure names the base image, it is easy to misread as a registry outage or a bad FROM, but the registry is never reached.

Cause

withSession() opens the gRPC session BuildKit uses for auth and the build context, and registers the moby.filesync.v1.Auth service on it. Recent BuildKit health-checks that session over grpc.health.v1.Health/Check and tears the session down after two consecutive failures (see moby/buildkit session/grpc.go: interval 5s, failureThreshold 2). Since the session serves no Health service, every check returns UNIMPLEMENTED, so BuildKit drops the session ~10s in. A build that clears the session-dependent phase before then succeeds; one that does not stalls, which is why the failure rate rises with the number of in-flight builds.

Fix

Register a minimal grpc.health.v1.Health service on the session that answers SERVING for both Check and Watch. This is what a compliant BuildKit client is expected to provide. No API change; it only adds a service to the session's gRPC server.

Adds lib/proto/health.proto (the standard gRPC health protocol) and registers the service in lib/session.js, mirroring how the Auth service is loaded and added.

Verification

Against a rootless Docker Engine 29.6.2 / BuildKit builder that reproduces the drop:

  • With an unmodified session, the daemon logs healthcheck failed fatally: Unimplemented and then no active session for the session's UUID, and the build hangs.
  • With this change, holding a session open for 40s (8+ health-check intervals) produces no health-check errors for that UUID, and a FROM debian:bookworm-slim + RUN build completes end to end in ~13s, past load metadata, producing an image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant