Skip to content

Browse: a proxy's or group's package page answers from the cache, with Refresh from upstream - #841

Merged
schmitch merged 1 commit into
mainfrom
claude/bold-curie-f4c2ca
Nov 3, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/bold-curie-f4c2ca

Conversation

@schmitch

@schmitch schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes #836. This is part 1 of #834; the other parts are #837 to #840.

What changes

The web UI's package page (GET /browse/packages/{repo}/{id}) used to go through the read path clients use. On a proxy, and on each proxy member of a group, a listing past metadata_ttl_seconds was fetched from the upstream on the request. Then came the upstream metadata, the tags and the latest manifest, one step and one member after another. The page was slow, and opening it added the package to the browse list.

The page now reads what Forge holds.

  • A proxy, and a group's proxy members, are read with the new ProxyRepo::cached_only (through AppState::cached_read_ops_for). That means the cached version rows, the stored upstream documents whatever their age, the stored tags and the downloaded packages, as an offline proxy serves them.
  • The request asks no upstream and discovers nothing.
  • Stored documents and the persisted index are read without going through the per-instance caches. Otherwise a stale copy read for the page would stay cached there for client requests, and the page could wait on an upstream fetch already under way.

It says how old the data is, and refreshes in the background (the option chosen in #834).

  • Each successful upstream listing records its time in a new table, proxy_package_listing (migration 0018). Repointing the proxy deletes its rows; deleting the repository cascades.
  • The detail now includes cached_at. For a group, this is the oldest listing among the proxies serving the package.
  • When the listing is past the TTL, or older than the repository's last edit or cache invalidation, the page starts a refresh off the request and sets refreshing: true. Limits: one refresh per proxy and package at a time, four per instance, and no new one within a minute of the last. A package the proxy holds nothing of is never refreshed this way.

Refresh from upstream (every reader, as chosen in #834).

  • POST /browse/packages/{repo}/{id}/refresh asks the upstream through the client read path, then returns the refreshed detail. It needs read access on the repository.
  • Data still within the metadata TTL is not fetched again, the same as for a client's install. This keeps reader-triggered refreshes from overloading the upstream.

Web UI

  • A proxy's or group's package page shows "Cached 3 h ago" (the exact time on hover) and a Refresh from upstream button.
  • While a background refresh runs, the page shows "refreshing from the upstream…" and re-fetches every 2 s until it ends.
  • A package the proxy holds nothing of shows as "Package not cached" with the same button. If the upstream doesn't have it either, the page says so.
  • Hosted repositories and offline proxies show neither the button nor the age.

Not in this PR

The other parts of #834 are their own issues:

NuGet proxies still show no description for versions not downloaded yet, because the registration summary carries only dependencies. That gap was already there; #838 covers it.

Tests

  • crates/server/tests/browse.rs, using a mock NuGet upstream that can hold its requests:
    • the page answers from the cache, including dependencies read from the stored registration, while the upstream holds every request;
    • a stale page (after an invalidation) starts exactly one refresh, and the next look shows the new version;
    • an uncached id is neither requested upstream nor recorded;
    • Refresh needs read access (403 without it, 401 anonymous) and discovers an uncached package;
    • a group reads and refreshes its proxy member the same way.
  • Web: PackageDetail.test.tsx (cache age, Refresh, polling during a background refresh, the not-cached state, no Refresh on an offline proxy) and lib/age.test.ts.
  • Run locally, all passing:
    • forge-server: browse, package_delay, proxy_search, npm, plugin_contract, proxy_cache, group_proxy, dispatch;
    • forge-core: lib, group_read, proxy_documents, proxy_read, retention_apply, retention_worker;
    • forge-db: all;
    • web: all 596 tests and tsc;
    • cargo clippy --workspace --all-targets -D warnings, cargo fmt --check, cargo xtask changelog --check.
  • .sqlx is regenerated: three new queries.

Generated by Claude Code

The web UI's package page went through the read path clients use: on a
proxy, and on each proxy member of a group, a listing past the metadata
TTL was fetched from the upstream on the request, then the upstream
metadata, the tags and the latest manifest, one step and one member
after another. The page was slow, and opening it confirmed the package's
rows, so the browse list grew as a side effect of looking.

The page now reads a proxy, and a group's proxy members, cached only
(ProxyRepo::cached_only, AppState::cached_read_ops_for): the cached
version rows, the stored upstream documents whatever their age, the
stored tags and the downloaded packages, as an offline proxy serves
them, without an upstream request and without discovering anything. The
stored copies are read past the per-instance caches, so a stale copy is
never cached there for the reads that ask the upstream, and a fetch
under way there is not waited for.

Each upstream listing records when it was answered, in a new table
proxy_package_listing. The page says how old it is (cached_at), and once
that is past the metadata TTL or older than the repository's last edit
or invalidation, it refreshes the package off the request: one refresh
per proxy and package at a time, four per instance at most, none again
within a minute, and never for a package the proxy holds nothing of.
POST /browse/packages/{repo}/{id}/refresh lets any reader ask the
upstream now, as a client's install does; the UI offers it as Refresh
from upstream, also on the "not cached" state that a package the proxy
holds nothing of now shows instead of being fetched by looking.

The tests drive a mock NuGet upstream that holds its requests: the page
answers while it does, a stale page starts one refresh, an uncached id
is neither asked for nor recorded, Refresh needs read access only, and a
group reads and refreshes its proxy member the same way.

Fixes #836

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJXNBkjivKSznagvTyC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browse: a proxy's or group's package page answers from the cache, with Refresh from upstream

2 participants