Version resolution over unauthenticated git makes feature installs fail hard, with no fallback

[jmkle]

Summary

Most features resolve "latest" by running git ls-remote --tags against github.com through a copy of
find_version_from_git_tags. GitHub rations unauthenticated git over HTTPS per source IP
(https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/ lists
"cloning repositories over HTTPS"). When the request is refused the helper ends up with an empty version
list and calls exit 1, so a rationed network becomes a failed build rather than a slower one.

This is not specific to one feature. The helper is copy pasted into 11 install scripts in 6 slightly
different variants, and every one of them exits 1 on an empty list.

Scope

Features that resolve a version through github.com git:

docker-in-docker, docker-outside-of-docker, github-cli, git-lfs, go, kubectl-helm-minikube,
nix, node, php, powershell, python, rust

oryx and ruby additionally git clone from github.com during install, and copilot-cli resolves its
prerelease channel with a raw git ls-remote. That is 15 of the 28 features in src/ depending on
traffic GitHub rations.

What it looks like

docker-in-docker at defaults, resolving compose:

Finished installing docker / moby!
fatal: could not read Username for 'https://github.com': No such device or address
(!) Invalid compose_version value: latest\nValid values:\n
ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install!

node, resolving nvm, in the same build:

fatal: could not read Username for 'https://github.com': No such device or address
Invalid NVM_VERSION value: latest
ERROR: Feature "Node.js (via nvm), yarn and pnpm." (ghcr.io/devcontainers/features/node) failed to install!

The credentials wording is misleading and costs a lot of debugging time. The refusal arrives in two
shapes. Sometimes git prints the server message, "GitHub is temporarily limiting some unauthenticated
downloads to protect the stability of the platform". Sometimes the answer is 401 with an auth challenge,
git tries to prompt, finds no tty in a build, and reports "could not read Username". Both are the same
limit, and neither is a proxy, a DNS fault or a missing credential.

Measurements

From one container on an ordinary university network, base ubuntu:24.04, git 2.43.0.

Only part of the traffic is rationed. Protocol v2 splits ls-remote into GET /info/refs plus
POST /git-upload-pack, and the POST is the rationed half. Interleaved, same repo, same window:

protocol v2 (git default): ok=4  throttled=10
protocol v0 (legacy):      ok=14 throttled=0

Clones are rationed harder, and the same content over plain HTTPS is not rationed at all:

git clone --depth 1 of a small repo : ok=2 failed=4
codeload.github.com tarball         : ok=6 failed=0

The git endpoints return no rate limit headers, only x-github-request-id, so an install script cannot
detect the condition or back off. A REST reply by contrast carries x-ratelimit-remaining and a reset.

Because a build performs several of these lookups in a row, the failure rate compounds. A default
docker-in-docker plus node build makes four rationed requests, so at 95 percent per request the build
succeeds 81 percent of the time, and at 30 percent per request it succeeds under 1 percent of the time.

Three defects

1. A failed lookup is fatal. The helper exits 1 as soon as the list is empty:

version_list="$(git ls-remote --tags ${repository} | grep -oP "${regex}" | ...)"
...
if [ -z "${!variable_name}" ] || ! echo "${version_list}" | grep "^${!variable_name//./\\.}$" ...; then
    err "Invalid ${variable_name} value: ${requested_version}\nValid values:\n${version_list}"
    exit 1
fi

Fallbacks that do exist sit downstream of this point. In docker-in-docker, fallback_compose and
fallback_buildx only cover a failed artifact download, so the earlier resolution failure never reaches
them.

2. Pinning an exact version does not avoid the network. The guard meant to skip the lookup is

if [ "$(echo "${requested_version}" | grep -o "." | wc -l)" != "2" ]; then

grep -o "." matches any character, so this counts characters rather than dots. Every realistic value,
2.40.0 or 0.40.7 included, is longer than two characters and goes to the network. A two character
value skips the lookup but then fails the validation above, which compares against a list only the
network call can produce. There is no offline safe value except none.

3. Some lookups are redundant. docker-in-docker with moby: true installs moby-compose from apt
unconditionally, and the moby packages provide buildx too, then it resolves and downloads a second copy
from github.com. In an image built here, apt provided moby-buildx 0.36.1 and moby-compose 5.5.0
alongside a feature written /usr/local/bin/docker-compose.

Suggested fix

Fallback logic instead of exit 1. The repo already implements the mirror image of this in
get_previous_version, which calls api.github.com, detects API rate limit exceeded and then falls
back to GitHub tags. The same idea is missing in the other direction.

1. When git ls-remote yields nothing, try another source before failing. The REST API is one option, but
   it is 60 requests per hour unauthenticated and shared per IP, so it belongs in a chain rather than as a
   replacement single point of failure. Tarballs from codeload.github.com and the distro package are
   unrationed alternatives.
2. Degrade to a known good pinned version as the last step, so a rationed network yields a slightly older
   tool instead of a failed build.
3. Fix the character count so an explicitly pinned version skips the network entirely, and do not validate
   a pinned value against a list that requires the network to exist.
4. Skip the resolution when the distro package already provides the tool.
5. Consider centralizing the helper. Eleven copies in six variants means any fix has to be repeated, and
   they have already drifted apart.

Repro

{
  "image": "ubuntu:24.04",
  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:4.1.0": {},
    "ghcr.io/devcontainers/features/node:2": {}
  }
}

Build from a shared or busy egress IP. To force it deterministically, block git smart HTTP to github.com
while leaving curl https://github.com working.

Workaround for anyone hitting this

In the base image, before the features run:

RUN git config --system protocol.version 0
ENV METHOD=script

The first keeps every lookup on the half of the protocol that is not rationed. The second makes nvm's
installer fetch a tarball rather than clone, which node needs on top. Verified with a full
devcontainer build --no-cache of seven features including docker-in-docker at defaults and node.

Versions

ghcr.io/devcontainers/features/docker-in-docker 4.1.0 and node 2.1.0 as the two examples above,
@devcontainers/cli 0.88.0, base ubuntu:24.04, git 2.43.0, podman 5.4.2 with buildah 1.39.3.