Pass upstream 404 and auth failures through, and flag stale npm metadata

[medlabib]

Fixes #499.

With cache_metadata on, npm metadata that fell back to the cache was served through the rewrite path (cachedRewrite → serveRewrittenMetadata) as a plain 200, without the Warning: 110 - "Response is Stale" header that ProxyCached sets on the same fallback. And every upstream error triggered that fallback, including a 404 for a package the registry has unpublished and a 401/403 for credentials that stopped working, so neither answer reached the client.

What changed:

• rewrittenMetadata carries stale, set from the fetch result in handlePackageMetadata, and serveRewrittenMetadata sets Warning: 110 - "Response is Stale" when it is set. A 304 keeps the headers it has today.
• A 401 or 403 is now an upstreamStatusError, and fetchMetadataFromUpstream returns 404/401/403 to the caller instead of falling back to the cached copy. Transport failures and retryable statuses (429, 5xx) still serve the cached copy with the warning, which is what the docs promise for outages and rate-limit holds.
• The npm metadata handler relays a default-registry 401/403 as that status, with a JSON error body, so a broken upstream credential fails loudly. Routed packages keep the existing 502: a route's credentials are the proxy's concern, and relaying its 401 would send npm off to authenticate against the proxy. TestNPMRoutedMetadataDoesNotFallBack still passes.

Tests, in the new internal/handler/npm_metadata_stale_test.go:

• the stale fallback sets the Warning header with the rewrite cache on and off, and returns the cached body (this fails on main: no header, and 404/401/403 are served as a stale 200);
• a fresh response carries no Warning;
• 404, 401 and 403 reach the client as those statuses, with a JSON error body and not the cached packument;
• a 503 still serves the cached copy with the Warning, guarding the rate-limit/hold fallback.

go test ./... -count=1        # all packages pass
golangci-lint run ./...       # 0 issues (golangci-lint v2.13.1)

Not included: a maximum stale age for the fallback, which the issue mentions as optional.

AI disclosure: prepared with AI assistance (Hermes agent); the change was verified with the test suite and lint run above.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[simonchrz]

Thanks for picking up #499. The stale Warning header for rewritten npm metadata looks good.

The no-fallback change affects every ecosystem, though, not just npm. fetchMetadataFromUpstream also serves Composer, PyPI, Alpine, Debian and generic metadata. On main, cached metadata is served on an upstream 404/401/403. With this PR those requests fail. That contradicts the docs (generic upstreams are served stale when the upstream "refuses", and GitHub rate limits are 403s) and the 404 fallback described in #486.

Could you split it? I'd approve the Warning header part as is. The 404/401/403 behaviour either goes to an issue, or becomes opt-in for the npm handler only.

Nit: writeNPMetadataError is missing an "M".

[andrew]

I independently confirmed the shared-fallback regression described in #503 (review). Through the generic HTTP handler with an expired cached document, upstream 404/401/403 produce 404/502/502 on this head. The same test on main returns the cached body with status 200 and Warning 110 for all three. Please keep the stale-warning fix and scope the new fallback policy to npm, or split that policy into a separate change, preserving other ecosystems' documented fallback behavior. Add a regression test through a non-npm handler.