Skip to content

Add per-ecosystem artifact retention (age-based eviction) #306

Closed
#465
Closed
Add per-ecosystem artifact retention (age-based eviction)#306
#465
Labels
enhancementNew feature or request

Description

@pinguinfuss
pinguinfuss
opened on Sep 3, 2026

Summary

Cached artifacts currently stay in storage forever unless storage.max_size is set, and that limit is a single global LRU across all ecosystems. There is no way to say "keep npm tarballs for 30 days, OCI blobs for 14 days, and Maven artifacts indefinitely". This issue proposes a storage.retention block that evicts artifacts per ecosystem (and optionally per package) after they have not been accessed for a configurable duration.

Current behaviour

  • storage.max_size triggers size-based LRU eviction, global for all ecosystems (internal/server/eviction.go). It is DB-driven via artifacts.last_accessed_at.
  • gradle.build_cache.max_age / max_size evict Gradle build-cache entries only, storage-listing-driven (internal/server/gradle_cache_eviction.go).
  • Nothing else is ever evicted. Without max_size, the cache grows without bound.

Proposed configuration

Follows the existing cooldown pattern (default + ecosystems map + packages map keyed by PURL):

storage:
  max_size: "10GB"          # unchanged, size-based LRU still applies
  retention:
    # Evict artifacts that have not been accessed for longer than this.
    # "0" or empty disables age-based eviction (current behaviour).
    default: "0"
    ecosystems:
      npm: "30d"
      oci: "14d"
      alpine: "7d"
      maven: "0"            # explicitly unlimited
    packages:               # optional, keys as in cooldown.packages
      "pkg:npm/lodash": "0"
    sweep_interval: "10m"

Environment overrides: PROXY_STORAGE_RETENTION_DEFAULT, PROXY_STORAGE_RETENTION_SWEEP_INTERVAL (maps have no env override, same as cooldown).

Semantics

  • Retention is measured from the last access, not from the fetch time. Artifacts are immutable, so a fixed max age would only force re-downloads of actively used packages. For artifacts that were never served (for example mirrored ones), fetched_at is used as fallback: COALESCE(last_accessed_at, fetched_at) < cutoff.
  • Resolution order: package override → ecosystem override → default. A value of "0" at any level means "never evict by age" and stops the lookup.
  • Durations accept a d suffix, using the same parser as cooldown.
  • Valid ecosystem keys are the literals the handlers pass to the artifact fetch (also the first segment of the storage path): npm, pypi, cargo, gem, golang, hex, pub, maven, nuget, composer, conan, conda, cran, julia, swift, deb, rpm, alpine, helm, oci. Unknown keys fail validation so typos are not silently ignored.
  • The retention sweep runs before the size-based LRU pass, so max_size operates on the already-pruned set.

Implementation outline

  • internal/config/config.go: add RetentionConfig under StorageConfig, validation of durations and ecosystem keys, parse helpers.
  • internal/database/queries.go: query for expired artifacts per ecosystem (join artifacts → versions → packages.ecosystem, batch limit as in GetLeastRecentlyUsedArtifacts) plus SELECT DISTINCT ecosystem FROM packages.
  • internal/server/retention.go (new): sweeper modelled on evictLRU, reusing store.Delete and ClearArtifactCache. Start it from Server.Start next to startEvictionLoop.
  • Tests: internal/server/retention_test.go using the harness from eviction_test.go; config parsing and validation cases in config_test.go.
  • Docs: config.example.yaml, storage section in docs/configuration.md, README example config.
  • Optional in the same PR: a proxy_artifacts_evicted_total{reason, ecosystem} counter covering both LRU and retention eviction (neither has a metric today).

Notes and edge cases

  • Mirrored artifacts. Packages preloaded with proxy mirror for offline use are evicted once they age out, even if never served. Document that the clock starts at mirror time. A pinned flag would need a schema migration and is out of scope here.
  • Multiple replicas on PostgreSQL. Concurrent sweeps are safe: Delete and ClearArtifactCache are idempotent, matching the existing LRU behaviour.
  • Key naming. Artifact ecosystem keys differ from metadata-cache keys in a few places (deb vs debian, oci vs oci-tags/oci-manifest). Retention uses the artifact keys.
  • Existing bug worth fixing alongside. The comment on gradle.build_cache.max_age promises "7d", but validation uses time.ParseDuration and rejects it. Switching to the day-aware parser fixes both.

Out of scope

  • Retention for the metadata cache (_metadata/ is never evicted today; separate issue).
  • Pinning mirrored artifacts.
  • Retention based on fetch age instead of last access.

Activity

  1. pinguinfuss commented on Sep 3, 2026

    @pinguinfuss
    pinguinfuss
    on Sep 3, 2026
    ContributorAuthor

    @andrew is that something worth implementing?

  2. andrew commented on Sep 3, 2026

    @andrew
    andrew
    on Sep 3, 2026
    Contributor

    Yeah sounds good!

  3. andrew
    added
    enhancementNew feature or request
    on Sep 7, 2026

  4. pinguinfuss commented on Oct 3, 2026

    @pinguinfuss
    pinguinfuss
    on Oct 3, 2026
    ContributorAuthor

    Hey @andrew,

    Before I start, here's how I'd like to build this. I deviate from the issue text in a few places, so I'd like to check with you first.

    I'd split it up: one PR with the infrastructure (config, sweeper, queries, an eviction counter, plus the Gradle "7d" fix), and then one small PR per ecosystem. The first PR doesn't enable retention for any ecosystem, so on its own it changes nothing. Each ecosystem PR registers itself from its handler file and brings tests for its own quirks. Smaller reviews, and if one ecosystem causes trouble it's easy to revert.

    That means default only applies to ecosystems that are already enabled. If someone configures one that isn't enabled yet, they get a warning and the rule is ignored. Typos still fail validation. On startup I'd log which ecosystems are covered and which ones aren't yet, so nobody gets surprised when a new release adds one. Without a default, nothing changes compared to today, and max_size stays as it is.

    I'd also drop the "retention runs before LRU" ordering. A cleared artifact already counts as freed in GetTotalCacheSize, so the order doesn't buy much. A shared loop would also make LRU wait behind a long first retention sweep.

    Speaking of long first sweeps: retention would go through pending_deletes rather than deleting directly, because buffered hits can make an artifact that's being served right now look expired. At 100 deletes per minute, though, an old cache would take days to actually free disk. So I'd add an index on queued_at, let reclaim work through the queue within a time budget per tick, and cap how many artifacts one sweep clears.

    Two things I ran into while reading the code:

    • The COALESCE(last_accessed_at, fetched_at) < cutoff from the issue would evict an artifact again right after it's re-fetched. The clear keeps the old last_accessed_at, and a cache miss doesn't record a hit. I'd check both columns separately instead: fetched_at < cutoff AND (last_accessed_at IS NULL OR last_accessed_at < cutoff).
    • Package overrides: several ecosystems store a different PURL than you'd usually write. For example, library/nginx is stored as pkg:oci/library%2Fnginx, alpine adds the alpine namespace, and swift is stored as pkg:generic/swift-registry/.... Each ecosystem would convert the configured key to its stored form, and the docs would list the accepted form. Julia gets no package overrides, because pkg:julia/X doesn't parse without a uuid.

    One side note: proxy mirror stores rubygems/packagist as the ecosystem, while the handlers use gem/composer. Retention handles both spellings, but should I open a separate issue to make mirror use the handler names?

    I'd leave generic out for now. Release assets can disappear upstream, and then the file would be gone for good.

    Does that work for you?

  5. andrew commented on Oct 8, 2026

    @andrew
    andrew
    on Oct 8, 2026
    Contributor

    That all sounds right.

    • Split into infra + per-ecosystem PRs: yes. The infra PR changing nothing on its own is the right shape.
    • Dropping the "retention before LRU" ordering: fine, the reasoning holds.
    • Routing through pending_deletes with a queued_at index, a time budget per reclaim tick, and a per-sweep clear cap: yes. That matches how reclaim already works after Give each fetch its own storage path #367.
    • fetched_at < cutoff AND (last_accessed_at IS NULL OR last_accessed_at < cutoff): good catch, use that.
    • Per-ecosystem PURL key normalisation with the accepted form documented, and no package overrides for julia: fine.
    • Leaving generic out: agreed.

    On the mirror rubygems/packagist naming: yes, that's a separate bug, I've opened #415 for it. Retention handling both spellings in the meantime is the right call.

    The gradle "7d" parse fix is fine to fold into the infra PR.

    One thing: for the "ecosystem not enabled yet" case I'd rather fail validation than warn-and-ignore, same as unknown keys. Someone configuring oci: 14d before the oci PR lands should find out at startup, not from a log line they might miss. Happy to be talked out of that if it makes the per-ecosystem rollout awkward.

  6. pinguinfuss
    added 9 commits that reference this issue on Oct 10, 2026

    Reject qualifiers and subpaths in retention package keys

    fd25bef

    Say when a package key doesn't match how packages are stored

    4009d48

    Note that re-running the mirror keeps packages cached

    2593ace

    Say when an ecosystem takes no package rules

    450f3ab

    Use helm as the example of an ecosystem without package rules

    f57168d

    Say that each cached file ages on its own

    1b41f61

    Clear an expired artifact and queue its file in one transaction

    878740f

    Show artifact evictions on the analytics page

    2695f5b

    Store artifact times in UTC

    7e76bc3
  7. andrew
    added a commit that references this issue on Oct 12, 2026

    Add age-based retention for cached artifacts (#428)

    b664530

  8. 31 remaining items

  9. pinguinfuss
    added 3 commits that reference this issue on Oct 12, 2026

    Enable storage retention for nuget

    dcfb9a8

    Enable storage retention for pub

    48edbb8

    Enable storage retention for rpm

    42f459e
  10. andrew
    added 4 commits that reference this issue on Oct 12, 2026

    Enable storage retention for cargo

    70ea921

    Enable storage retention for composer

    9c75e59

    Enable storage retention for conan

    ff4d5ca

    Enable storage retention for swift

    f8ca190
  11. pinguinfuss
    added 7 commits that reference this issue on Oct 12, 2026

    Fix gem filenames whose platform ends in a number

    680a2da

    Store mirrored NuGet ids lowercased

    5d7855f

    Enable storage retention for hex

    743fbe5

    Fix hex tarballs whose version contains a hyphen

    c3c1204

    Enable storage retention for conda

    bffba28

    Refuse conda rules with more than a channel and a name

    c58cdae

    Enable storage retention for deb

    3297800
  12. andrew
    closed this as completedin #465on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions