Repository navigation
Add the loading cache core declares and the module that supplies it - #1365
Open
jwrosewell wants to merge 52 commits into
Open
jwrosewell wants to merge 52 commits into
jwrosewell wants to merge 52 commits into
Conversation
`PlatformLoadingCache` in `platform/loading_cache.rs` takes a key and a loader and returns the value, running the loader at most once per key at a time, so however many requests ask for a missing value at the same moment the source is asked once and every one of them continues with its value or its error. A failed load is not stored, so the next caller loads again. `NoLoadingCache` holds nothing and runs every load, and is what `RuntimeServices` carries when no module supplies a cache. Every key is a `StoreKey`, being the publisher's `ServerKeyPrefix`, the cache's name and the caller's key. The prefix is made in one place, `platform/server_key.rs`, from the publisher's domain in the loaded settings with a single leading `www.` folded and `ts.` in front, so two publishers sharing a store never share an entry. `PlatformValueCache` is the host's shared cache of small values with lifetimes, which a host supplies through `RuntimeServices` when it has one, with `lookup_or_reserve` handing the caller a `ValueReservation` where the host can make other callers for a key wait, and `remove`. A loading cache keeps its shared copies there where the host has one and in the key-value store otherwise. An `IntegrationBuilder` offers a loading cache with `with_loading_cache`, and `loading_cache_from` gives an adapter the one its linked builders offer, refusing two, so no adapter names the crate that supplies it. Core's tests get `MemoryLoadingCache`, which keeps what it loads for the lifetime the load gave and records every call.
A contributor's answer is loaded through `RuntimeServices::load_cached`, so the contributor is asked once however many requests want its rules at the same moment, and the copy in the host's shared store answers every instance until the contributor's refresh is due. The answer's lifetime in the cache is the contributor's refresh, and `[cache.robots-txt]` sets how long the copies last with `memory_seconds`, `store_seconds` and `store_idle_seconds`, defaulting to a minute, a week and 30 days. A memory lifetime may be 0, the other two must be at least 1, and none may be more than a year, which `LoadingCacheConfig::check` refuses as the settings load, naming the setting. The last answer a contributor gave is also kept apart from the cache for 30 days, in the host's value cache where it has one and in the key-value store otherwise, and read only when the contributor fails, because yesterday's rules are a better answer than none. Every key starts with the publisher's `ServerKeyPrefix`, so two publishers sharing a store never share an answer. The handler hands the settings through in place of the publisher's domain and the time, and a selection carries the name the contributor was declared under. The tests use `MemoryLoadingCache` and cover the one load for a held answer, the key naming the publisher, changed settings asking again, the kept answer served when the contributor fails, in a second instance and from a value cache, and a failure with nothing kept being an error that is not held.
Each adapter takes the cache from the builders it is given with `loading_cache_from` when it builds its state, and hands it to every request's services, so every request in a process shares the cache's memory copy and waits for one load. Axum, Cloudflare and Spin take it from the stock list followed by the builders a deployment passes, as they take every vendor module, and hold nothing without one. No adapter names the crate that supplies it. On Fastly the finishing steps' services carry the same cache as the request, so a lookup there shares the request's memory copy, and the entry point builds those services once from the state it has, or with the unavailable store and a cache that holds nothing when no state was built. Fastly supplies `FastlyValueCache` over Fastly's cache as the value cache, which needs nothing created or linked on the service. A transaction lookup reserves a key nothing holds for this instance and makes every other instance at the location wait for it to be filled, a stale entry is reserved for a fresh load rather than served, and a reservation dropped unfilled is released so a waiting instance takes it over. Each entry is written with a surrogate key, a hash of its key, which `remove` purges. Its tests run under Viceroy and cover the reservation, its release, a stale entry and the purge.
`crates/loading-cache/51degrees` implements `PlatformLoadingCache` with the published `fiftyone-caching` package, version 4.5.8-49, with its default features off so it builds for every host. Each cache is a least recently used copy in process memory with a short lifetime over the host's shared store, being its value cache where it has one and its key-value store otherwise, written in the package's entry format so a store that deletes late or keeps no lifetimes never returns an entry past its drop time. The memory layer is shared by every request in a process and loads a missing key once, covering the store read and the source together. The store layer is made for each call from that request's stores. A load a cache allows to run apart from its caller runs on a pool of threads on a native host and with the request's `wait_until` on Cloudflare, and every load runs in its caller on Fastly and Spin. The crate is the module `loading-cache.51degrees`, named by its folder, and `builder()` offers the cache whatever the settings say, because a loading cache is how a deployment caches rather than a module a document selects. The stock list carries the builder, so every adapter takes the cache from it with `loading_cache_from` and names no crate, and a test holds the list to offering one cache that keeps what it loads. The crate carries the maintainers table, the workspace lists it, and the Axum aliases lint and test it, where its own tests run natively. The crate's tests ask for one key from 16 callers at once on 16 threads with the pool, on 16 threads with loads in the caller, and on one thread with each load a task of its own, and the source runs once each time. Other tests cover a failure reaching every waiter and not being kept, the store answering a second process, the brief memory copy, idle expiry, a value's own lifetime, a reserving value cache being filled and released, two publishers kept apart, removal, the module's name and the builder's offer.
Spin's key-value store refuses a write with a lifetime, so a robots.txt answer kept for a later failure, and a loading cache entry in the module's key-value layer, are written without one there, as the held answer was before. The entry format carries each entry's own drop time, so a copy in a store that keeps no lifetimes still ends when it should, and a kept answer stays until it is replaced, which is what the configuration guide says of Spin. The robots.txt load carries the name the contributor was declared under as the static text the module context takes, and the handler hands it over from the selection. Tests cover the kept answer's lifetime where the store takes one, the answer kept where it refuses one, and a store that refuses lifetimes still answering a second process through the module.
The configuration guide gains "Loading caches" under the cache section, saying what is held, where the two copies live, how the keys start, which crate a stock build caches with and the `[cache.robots-txt]` lifetimes, and its robots.txt section says each answer is held through it. The integration guide gains "Supplying a loading cache", being how a crate offers one with `with_loading_cache` and how code loads through one with `load_cached`, a `CacheSpec`, a `StoreKey` and a `Loader`, and the builder table lists the offer. The example configuration shows `[cache.robots-txt]`, and the changelog records the cache, the lifetimes, the key prefix and the value cache under Unreleased.
The crate keeps a load running with the request's `wait_until` only where the Worker's handling of the request is wrapped in `scope_request`, and nothing on an adapter wraps it, so the crate's documentation and the changelog say so rather than that the load keeps running after the response on Cloudflare.
The item exists only on the Cloudflare target, so a link to it from the crate's documentation is reported broken on every other target's documentation build. The name is written as plain code instead.
The first build adds the published fiftyone-caching crate, hashlink and tokio-util to the lock, which a build with --locked refuses until they are committed.
A feature that holds answers in a loading cache names, in its own `[cache.<name>]` table, the module whose cache holds them, as the module's short name or its name in full. `loading_caches_from` resolves every feature's selection against the caches the linked builders offer, makes each module's cache once however many features select it, and refuses a name no builder offers a cache from, naming the feature and the modules that do. The services carry the selections as `LoadingCaches`, and a feature that selected none loads through `NoLoadingCache`. Before this the stock build's one offered cache held every feature's copies, and a deployment linking two caches was refused. Now each feature decides, so the asset cache and any later feature can choose a cache of its own. `[cache.robots-txt] module = "51degrees"` selects the stock cache for the robots.txt modules' answers. The guides, the example configuration and the changelog say so, and the adapters pass the selections where they passed the one cache.
# Conflicts: # CHANGELOG.md # crates/trusted-server-core/src/robots_txt/holding.rs # crates/trusted-server-core/src/robots_txt/mod.rs
A held copy that no longer reads as a contribution is removed from the loading cache and the module is asked again, where it used to answer 503 until the copy ended. The 51Degrees crate drops its unused thread pool, local spawner and Cloudflare request scope, so every load runs in its caller. The guides and changelog say holding is opt-in, the four adapters describe the loading cache field in one sentence, and the lock no longer moves prost onto itertools 0.13.
This was referenced Oct 15, 2026
RuntimeServices::load_cached handed the loading cache the request's own services, and the robots.txt holding handed each contributor the same, so the client address, connection details and host signals of whichever reader asked first reached a load whose answer every reader is served. RuntimeServices::without_reader clears those and keeps the stores, and both paths now take it.
…cluded On wasm32-unknown-unknown the factory kept one cache for the whole Worker isolate, so a request could wait on a load another request was running. The published caching crate says such a request fails unless every request runs in its request scope with its wait_until spawner, and nothing here does. The factory now makes a cache for each application state on every host. Cloudflare builds its state for each request, so its memory copy lasts one request and Workers KV holds the copy between requests.
…serves The test written with robots.txt said this adapter holds no answer, so a module that answered once and then failed made the second request a 503. With the loading cache, the last answer a contributor gave is kept and served past its age when the contributor fails, so the second request is answered 200 with that answer, and the test failed in CI. One test now asks for the file twice and expects the kept answer the second time. The 503 is tested with a module that never answers, so nothing is kept. All 275 tests of the Fastly adapter pass under Viceroy.
This was referenced Oct 17, 2026
Neither adapter implemented Hooks::stores, so EdgeZero's empty default applied, run_app built no key-value registry, and every request was given UnavailableKvStore. The loading cache, the kept robots.txt answers and any cache a module declares therefore held nothing between requests on either host, and both build their application state for every request, so the memory copy held nothing past one request either. The Cloudflare adapter now declares TRUSTED_SERVER_KV, which EdgeZero opens as the Workers KV binding of that name because a store's binding is its id unless EDGEZERO__STORES__KV__TRUSTED_SERVER_KV__NAME names another. The Spin adapter declares default, the label spin.toml grants and the store startup already reads the settings from. EdgeZero treats a declared store as required, so a Worker without the binding, or a component that cannot open default, fails every request before any route runs. Tests: every_wrangler_manifest_binds_the_declared_key_value_store checks that both Wrangler manifests bind the namespace EdgeZero opens for the declared id, the_declared_key_value_store_is_a_label_spin_grants checks spin.toml grants the declared label, and on each adapter the_declared_store_is_the_key_value_store_a_request_is_given shows the store reaches a request's services, with the robots.txt tests now putting the store on a request under the declared id as run_app does. The Spin test that asserted edgezero.toml's store declarations is replaced, because the adapter never read that manifest at run time.
The Cloudflare comments said run_app injects a config handle from the TRUSTED_SERVER_CONFIG variable and the KV handle from TRUSTED_SERVER_KV, and the Spin comments said run_app injects config and KV handles. run_app opens only the stores an app declares. Both adapters now declare their key-value store and no config store, so a request gets the declared key-value store and the no-op config store, while startup reads the settings from TRUSTED_SERVER_CONFIG on Cloudflare and from the default store on Spin. The comments say so.
test_cloudflare_without_its_kv_binding_fails_every_request runs the Worker under wrangler dev with the [[kv_namespaces]] table taken out of the CI manifest and expects 500 with the dispatch error on /, the discovery document and /robots.txt, which shows the Worker declares its store and that EdgeZero refuses to route without it. The harness waits for any answer in that case, because the health check never succeeds, and without_kv_namespaces has tests of its own. A test that a robots.txt module's answer is kept across two Worker requests is not possible with this bundle, because the Worker CI builds links the stock modules alone and none of them contributes robots.txt rules.
The changelog said answers were held in the key-value store on Cloudflare and Spin while Fastly and Axum held nothing, that before this change each instance held its own copy, and that the last robots.txt answer is kept in the key-value store on every adapter but Fastly. The guide said the same of the last answer and that the shared store is the host's key-value store wherever there is no value cache. Before this branch no adapter gave a request a working key-value store, and Axum still has none. Now the changelog has an entry for the declared stores and the binding they require, the loading cache entries name the store on each adapter, and the robots.txt entry says the loading cache holds the answer. The configuration guide names Workers KV and Spin's default store as the shared store and says Axum keeps the memory copy alone, and the Cloudflare and Spin guides say each adapter opens its store for every request and fails every request without it.
The Cloudflare README said the adapter has no request-time KV registry and the Spin README that request-time KV services remain unavailable. Each adapter now declares its key-value store, TRUSTED_SERVER_KV and default, which every request is given, and a Worker or component without it fails every request.
The next branch removes the robots.txt route and the route tests that held a module's answer, so a store test inside that module would not outlive this branch. the_declared_store_is_the_key_value_store_a_request_is_given now lives in each adapter's tests/key_value_store.rs with its own store kept in memory, and builds the registry from the declaration the way EdgeZero does. The Spin test that the declared label is one spin.toml grants moves there too. The robots.txt route tests are left as they were.
The sentence said a deployment whose key-value store cannot be opened keeps no last answer, which stays true now that Cloudflare and Spin fail every request without their store, and the Cloudflare and Spin guides say that. The point goes back to its wording so the guide merges cleanly into the next branch, which adds a point after it.
# Conflicts: # CHANGELOG.md # docs/guide/configuration.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-core/srccrates/loading-cache/51degreescrates/trusted-server-adapter-fastly/srccrates/trusted-server-integration-tests/testscrates/trusted-server-adapter-cloudflare/testsdocs/guidecrates/trusted-server-adapter-spin/testscrates/trusted-server-modules/srccrates/trusted-server-adapter-cloudflare/srcCargo.lockcrates/trusted-server-adapter-spin/srctrusted-server.example.tomlcrates/trusted-server-adapter-axum/srcCHANGELOG.mdCargo.tomlcrates/trusted-server-adapter-cloudflare/README.mdcrates/trusted-server-adapter-spin/README.md.cargo/config.tomlcrates/trusted-server-adapter-cloudflare/wrangler.tomlcrates/trusted-server-modules/Cargo.tomlCloses
Closes #1282
Spec
Spec: loading-cache
Test plan
CI on this pull request at e66b955 passed all 32 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)Stacks on a middleware that prepares (#1364).
What this pull request does
Core declares a loading cache,
PlatformLoadingCacheintrusted_server_core::platform, which loads a missing value once per key however many requests ask for it at the same moment, gives every request waiting on that load the value or the error, and holds nothing that failed. Each cache is named by aCacheSpecwith the lifetimes of its copies, a brief one in process memory and one in the host's shared store, and keys under aStoreKeythat starts with the publisher's prefix, so publishers sharing a store never share a copy.Each cache is selected by naming, in its own
[cache.<name>]table, a module that offers a loading cache. A table that names none takes the one loading cache module the build links, so a stock build holds answers without a line in the document. With none linked, or several and none named, the table getsNoLoadingCache, which holds nothing and runs every load. A name no linked crate offers a cache from is refused when the service starts, with the names that could be written. On Cloudflare, which builds the state for every request, that refusal is per request. Each robots.txt module's answer is the first thing held this way, so a module is asked once per refresh however many instances and requests want its rules.The implementation is a module crate,
crates/loading-cache/51degrees, maintained by 51Degrees, written with the publishedfiftyone-cachingcrate over the stores each adapter opens. On Fastly those are its cache and its key-value store. The Axum, Cloudflare and Spin adapters open no key-value store yet (#1139), so there a copy is held in memory for the life of the application state, which on Cloudflare and Spin is one request. Every load runs in its caller. It offers the cache from its builder, the stock list carries it, and an adapter takes the selections from the settings and the builders it is given withloading_caches_from, so no adapter names the crate.A load that several requests share is handed nothing of the reader whose request started it, through
RuntimeServices::without_reader, each application state has a loading cache of its own, and a[cache.<name>]table whosemoduleno linked builder offers is refused at startup and byts config validate.Why
The same answer is asked for by every instance and every request, and a cache that lets every miss load at once asks a module's service as many times as there are requests. One load per key at a time, with the result shared by every waiter, is the pattern the other 51Degrees language packages use, and the crate that implements it is published for every 51Degrees service to use. Core keeps only the interface, so core names no vendor and depends on no package.
What breaks
A robots.txt module's answer is now held by the loading cache of the one loading cache module the build links, or of the module
[cache.robots-txt] modulenames. The robots.txt change (#1355) held each answer in the key-value store on Cloudflare and Spin. A stock build links one loading cache module, so it holds answers with no line in the document. A build that links none, or several with none named in[cache.robots-txt], holds nothing, and every request for the file asks each module's service. A deployment on such a build keeps the holding by adding the table.How it was verified
No load has run against a real host store. Fastly's cache is checked on Viceroy. The Cloudflare and Spin adapters build their application state for each request and open no key-value store, so on those hosts a copy lasts one request until the adapters open one (#1139). The page template does not load through the cache. Locally on Windows the fixes of 16 October (a1931fd to 0d49c46) passed their tests, clippy and
cargo fmt --all -- --check, and at 3564019 all 275 tests of the Fastly adapter passed under Viceroy.References #1282.