Skip to content

Docker build caching leading to security scan failures on streak2 - #368

Merged
joshua-white merged 2 commits into
masterfrom
ci-docker-caching
Sep 10, 2026
Merged

joshua-white merged 2 commits into
masterfrom
ci-docker-caching

Conversation

@joshua-white

Copy link
Copy Markdown
Contributor

Even when streak2 is fully patched, security scans find vulnerable packages inside persistent Docker BuildKit cache directories. These files are created while building GEOS base and TPL Docker images. They are not necessarily present in running containers, but they remain on disk after CI finishes and are therefore detected by host-level security scans. Manual pruning removes them temporarily, but subsequent image builds recreate the cache.

The fix proposed here continues building Docker/TPL images on streak2, but using a dedicated, ephemeral Buildx builder for each CI job. The existing workflow already creates such a builder, but the build script currently uses Docker’s persistent default builder instead. The proposed change would explicitly direct builds to the temporary builder and configure it to be removed when the job finishes.

This will likely not completely fix the issue as the core problem has to do with the security scans not recognizing backported security updates because it just naively checks version numbers. This fix here should at least get rid of our huge build cache files being saved and prevent future scans failing on very old caches.

@joshua-white joshua-white self-assigned this Sep 8, 2026
@joshua-white

Copy link
Copy Markdown
Contributor Author

Updated this PR to:

  • Use per-job ephemeral Buildx builders with explicit cleanup.
  • Remove locally loaded images after streak2 jobs.
  • Avoid installing Sphinx—and its Jinja2/certifi dependencies—for ~docs builds.
  • Preserve Sphinx for the Github-hosted +docs build.

This should prevent persistent cache growth and remove the known BCB findings from streak2 builds.

@joshua-white
joshua-white requested review from rrsettgast and victorapm and removed request for victorapm September 10, 2026 00:32
run: |
image="${DOCKER_REPOSITORY}:${DOCKER_TAG}"
if docker image inspect "${image}" >/dev/null 2>&1; then
docker image rm "${image}" || echo "::warning::Failed to remove local image ${image}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If docker image rm fails for some reason, the CI job still succeeds and leaves the image behind. Should we make this a hard failure?

@victorapm victorapm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, Josh!

@joshua-white
joshua-white merged commit 7afcc56 into master Sep 10, 2026
13 checks passed
@victorapm
victorapm deleted the ci-docker-caching branch September 10, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants