Repository navigation
Remove administration from the publisher's domain - #1354
Open
jwrosewell wants to merge 39 commits into
Open
jwrosewell wants to merge 39 commits into
jwrosewell wants to merge 39 commits into
Conversation
`ts keys rotate` and `ts keys deactivate` do from an operator's machine what `POST /_ts/admin/keys/rotate` and `POST /_ts/admin/keys/deactivate` do from inside the running service. They drive the same rotation code in core, `KeyRotationManager`, over two store implementations that write through the `fastly` CLI with the operator's own token from `FASTLY_API_TOKEN`. The token reaches `fastly` in its environment and a private key reaches it on standard input, so neither is in an argument list another process can read. Every value is joined to its flag with `=`, so a key id beginning with `-` is never read as a flag. A key id is checked before anything is asked of `fastly`. A new key takes the rule the rotate route applies, `kid_is_creatable`. A key being retired takes the looser rule the deactivate route applies, exposed here as `kid_is_well_formed`, so that a key created under an earlier rule can still be removed. Neither command takes `current-kid` or `active-kids` for a key. Named as a key, either entry would be written over by a rotation or deleted with the key. Two things differ from the stores the service reads, and the command's stores allow for both. - A read is a call to the platform and can fail for a reason other than a missing entry. The rotation library takes any failed read for an entry that does not exist, which would write a key list without the keys already active, or let the current key leave the active set. So the stores tell a `404` from any other failure and write nothing once a read has failed. - The service's management client takes a `404` on a delete for success, which is how a delete tried again after a partial failure finishes. The stores do the same, after asking for the store, because the API answers `404` for a missing store too. `fastly_cli` holds the runner, so the next command that administers a service through the host uses the same one. Its report of a failed `fastly` names the command by its words and says what `fastly` printed, including when the process stopped before it read its input. The command comes before the removal of the two routes, so that no commit leaves an operator without a way to rotate a key.
The command called `POST /_ts/admin/cache/purge` on the publisher's domain with a shared password. It now runs `fastly service purge --key` with the operator's own token from `FASTLY_API_TOKEN`, so the credential that can empty the cache stays with the platform's access control and off the domain every reader reaches. A token with the `purge_select` scope is enough, which allows a purge by surrogate key and does not allow a purge of everything. `fastly service purge` is the Fastly CLI's name for the command since its version 14.0.0, where `fastly purge` became a deprecated alias. The repository pins 15.1.0. The surrogate key is derived in the command by `reader_url_surrogate_key`, the function the service tags a cached page with, and `--all` names `ts-template`, the key every cached object carries. Both scopes purge what they purged before. Neither runs `fastly service purge --all`, which would also flush objects the service cached without one of those keys. `--service <url>`, `--username` and `TRUSTED_SERVER_ADMIN_PASSWORD` go and `--service-id <id>` arrives. A service id that is not letters and digits is refused, because it becomes an argument of the `fastly` process. The tests that drove the command against a fixture service are replaced by tests against a fake `fastly`. The command changes before the route is removed, for the reason `ts keys` comes first. The guide's command table gains the two `ts cache` rows it lacked.
`GET /_ts/admin/ec` and `GET /_ts/admin/ec/{id}` read any reader's identity
graph record, and `GET /_ts/admin/eids` decoded the caller's EID cookies,
on the publisher's own domain behind one shared password. `GET /_ts/data`
shows a reader what is held against that reader's own Edge Cookie and asks
for no password, so the operator lookups go.
What goes with them: `ec/admin.rs`, `KvIdentityGraph::lookup_raw`, the
decoder's diagnostic view of an EID cookie, the two probes that established
handler coverage for the lookup by id, the adapters' routes and their
not-supported stand-ins, and the tests and guide sections for all of it.
What stays is the check that keeps administration paths away from the
origin. It moves to `closed_paths.rs` as `closed_path_response`, with the
matcher unchanged: every path beneath `/_ts/admin`, and the `/admin/keys`
alias beside it, as sent and after each of up to four percent-decoding
rounds. A request there that no route claims is answered `404` by the
server, so a caller's `Authorization` header and body are never forwarded.
The three removed paths are answered that way for every method, where a
method other than GET was answered `405`.
`POST /_ts/admin/keys/rotate` and `POST /_ts/admin/keys/deactivate` let anyone holding one shared password replace the keys the service signs with, over the domain every reader reaches. `ts keys rotate` and `ts keys deactivate` do the same work from an operator's machine with the operator's own token, so the two routes and their handlers go. Only Fastly served them. Axum, Cloudflare and Spin registered them to answer `501`. The `/admin/keys/rotate` and `/admin/keys/deactivate` routes go with them. They existed to refuse those paths, so that a call to one was not forwarded to the origin with its credentials. `closed_path_response` already answers every path beneath `/admin/keys` and `/_ts/admin` before the publisher fallback, on each adapter, so the four paths are still answered `404` here for every fallback method. Each adapter's `key_administration_paths_are_answered_here_and_never_proxied` holds that with the credentials a caller would send. `kid_is_creatable` and the two validators behind it stay in `request_signing/endpoints.rs`, because `ts keys` refuses a key id with them before it writes. The tests of the password challenge ask at `/_ts/admin/anything`, since the path they asked at is not a route any more, and the two probe tests take `/first-party/proxy` as their named route for the same reason. The key rotation guide's diagram had `current-kid` written before `active-kids`. The code writes `active-kids` first, and the diagram now agrees with the text beside it. `[request_signing]` still requires `config_store_id` and `secret_store_id`, which only the removed handlers read. They go in the next commit.
`[request_signing]` required `config_store_id` and `secret_store_id`, the platform ids of the two stores the keys live in. Only the removed rotate and deactivate handlers read them, to write those stores. The service finds its keys by the names the stores are linked under, `jwks_store` and `signing_keys`. With the handlers gone, the two ids were settings an operator had to supply with real values for nothing to read. They go, with the placeholder and padding checks on them in startup and deploy validation, and the auction plan's rule that enabled signing needs both. `ts keys` takes the ids on its command line. A configuration that still carries either key is refused with directions, through the refusal the removed tables use, when it is validated and when the service starts. Serializing the settings writes `enabled` alone, so a configuration the service writes is one it reads back. Fixtures whose `[request_signing]` section held only the two ids now say `enabled = false`, which is what an absent `enabled` meant.
`management_api.rs` was a client for the Fastly management API. The deployed service read an API token from a secret store named `api-keys` and used it to write its own config store and secret store, which is how the key rotation routes stored a new key. Nothing calls it now that keys are rotated with `ts keys`, so the client goes, with the `api-keys` entries in `fastly.toml` and the Viceroy template. The Fastly platform's store writes, `put`, `delete` and `create`, refuse with a message saying the service does not write its stores. They remain on the traits because `ts keys` implements them over the `fastly` CLI. `BackendConfig::from_url` and its two variants built the backend the client reached the API through. Nothing else called them, so they go as well, and the one test that parsed a URL through them states the parsed parts directly. The adapter's `base64` and `url` dependencies go with their last users. A deployment that linked an `api-keys` secret store can unlink it and revoke the token it held.
`POST /_ts/admin/cache/purge` let anyone holding one shared password empty the service's cache over the publisher's own domain, with nothing limiting how often. `ts cache purge` asks the hosting platform to purge the same surrogate keys with the operator's own token, so the route goes, with its handler in core, the stand-ins that answered `501` on Axum, Cloudflare and Spin, the harness mode and workflow step that drove it, and the parity tests of it. `auth::authenticated_username`, which named the caller in the purge log, goes too. Nothing in the service purges by surrogate key or purges everything now, so `purge_url_surrogate_key` and `purge_all` leave the template cache interface and every implementation of it. `purge_url`, which core uses to drop an entry it finds invalid, stays. The harness's purge mode was the end to end check that purging a key empties the cache. Two Fastly tests keep that evidence where it can still be had without the route: purging `ts-template` through the platform clears a stored template, and purging one page's reader key clears that page and leaves another. No admin endpoint is left, so the rule that every one must be covered by a `[[handlers]]` entry goes: `ADMIN_ENDPOINTS`, the coverage check at startup and at deploy validation, and the test that held the list against the Fastly route table. A configuration with no handler starts. Handler passwords are still checked. A request to the removed path is still answered `404` by the server and never forwarded, as for any path beneath `/_ts/admin`.
HTTP Basic authentication existed to gate the endpoints beneath `/_ts/admin`. None is left, so the gate goes: `auth.rs` in core, the `AuthMiddleware` each of the four adapters registered, the `Handler` type with its regex and its placeholder and weak password checks, and `handlers[*].password` as a declared secret. A configuration that still carries `[[handlers]]` is refused with directions when it is validated and when the service starts, through the refusal the removed tables use. Loading it and ignoring the list would leave an operator who gated a path of their own, a staging site for instance, believing it was still gated. An empty list is refused too, so is a handler given by an environment variable, and the settings are never written with one. The upgrade has an order to it. A build before this series refuses a configuration without a handler covering `/_ts/admin`, and this build refuses one with any handler, so no one configuration starts both. The new configuration and the new build go live together. The publisher path treats an `Authorization` header as bound for the origin in every case. It shared a cached page across readers when the edge had checked the header itself, and with no check at the edge that case is gone, so a request carrying the header always bypasses the shared template. The closed path check is unchanged and still answers `/_ts/admin` and `/admin/keys` with `404` before the publisher fallback, so a caller that sends a password to either never has it forwarded. Each adapter's tests of that send the credentials a caller would. What goes with the gate: - the tests of the challenge in the four adapters and the parity suite, and the tests that proved other routes were not challenged; - `[[handlers]]` in every test fixture, 52 entries; - the `handler_password` secret in `fastly.toml`, the Spin manifest, the Viceroy template, the integration fixtures, the three harnesses and the four smoke scripts, whose missing-secret cases cover the two secrets that remain; - the handlers section of the configuration guide, the Basic Authentication section of the API reference, the `Handler.password` row of the configuration guide's table of exceptional fields, and "unless an operator adds a handler" in the contracts of `/first-party/sign` and `/_ts/debug/ja4`. The configuration page's tests take their example list from `proxy.asset_routes`, having taken it from `handlers`. The page-bids alias comments named a `[[handlers]]` pattern covering `/_ts` as one reason a browser falls back to the alias. They name a gate in front of the service, which is the case that remains.
The DataDome module skipped its protection API for every path beneath `/admin/` and `/_ts/admin/`, listing them with the server's own routes. Nothing of the server's is served beneath either. `/_ts/admin` and `/admin/keys` are answered by the closed path check before any request filter runs, and everything else beneath `/admin/` is the publisher's own site, which the list left out of protection. `only_the_servers_own_paths_are_internal` holds the list both ways.
# Conflicts: # CHANGELOG.md # crates/trusted-server-adapter-cloudflare/src/app.rs # crates/trusted-server-adapter-fastly/src/app.rs
The purge command passes --key and --service-id joined with =, as ts keys does, with a test that no value stands alone. The guide no longer says a Basic credential is validated and the ts cache row names both caches. A duplicate percent check goes from closed_paths.rs, and comments that argued for alternatives not taken are cut or shortened.
# Conflicts: # CHANGELOG.md
This was referenced Oct 15, 2026
`ts config push --adapter fastly --key <key>` without `--local` now writes the key itself to the live config store that EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME selects, or to the store named `trusted_server_config` when that is unset, because EdgeZero writes no Fastly key but the logical store ID. The store's id is found by name with `fastly config-store list --json`, and each entry is written with `fastly config-store-entry update --stdin --upsert`, a long document's chunks first and its pointer last, through the same runner `ts keys` uses, with the token taken from FASTLY_API_TOKEN. A failed write stops the push and its message leaves out what `fastly` printed, because that can quote the value. `--dry-run` lists the keys and asks `fastly` nothing. Until now a push of such a key without `--local` was refused, because the runner arrived only with the key commands of this branch. Tests: a_remote_push_writes_chunks_then_the_pointer runs the writer against a recording runner and checks every update goes to the store the selector names, chunk by chunk and then the pointer, with each value on standard input. Two more cover the store name without a selector and a listing where the name is absent or carried twice. The integration test that expected the refusal is replaced by one that runs a remote dry run with no `fastly` on the path and checks it names the live store and writes nothing locally.
The guide, the CLI page and the changelog said a key other than trusted_server_config could be pushed only with `--local` and copied into a live store by hand. Now that `ts config push --key <host>` writes the live store the __NAME selector picks, they say so, and the guide's commands for a document per host drop `--local`.
…secret The test fake replaced a secret that already had the name. The API refuses to create a secret under a name the store holds and replaces one only when asked with --recreate or --recreate-allow, which the command never passes. The fake now refuses too, and a test checks that a rotation onto a secret a half finished rotation left behind fails, writes nothing to the config store and never asks to replace. The fake's 404 is now the text the fastly CLI prints, as 15.1.0 and 16.1.0 print it for a missing config store or entry and a missing secret store or secret. A test checks that both carry the wording the stores read a missing entry from, and another reads a value from the fields config-store-entry describe --json prints. The comment that said no run of the CLI had shown the wording now says which versions print it.
…efused Every adapter closes /_ts/admin before the publisher fallback, and the attestation endpoint is the one route the settings can place, so an endpoint there passed validation and was served beneath the prefix. validate_endpoint now refuses a page or JSON address the closed path matcher closes, as reserved_endpoint naming the address, with a test of /_ts/admin, /_ts/admin/attestation and /_ts/adminx/attestation.
Fastly, Cloudflare and Spin check for a closed path first in their fallback, and Axum ran the preparers first and checked inside dispatch_fallback, so a preparer saw, and could answer, a request to a closed path. The check moves into fallback_handler, ahead of execute_handler. A test sends /_ts/admin/keys/rotate with the seam probe's counting header and expects 404 with no preparer run.
…ble encoded paths C:/Program Files/Git/_ts/admin/cache/purge was in the core test alone. It joins the key administration paths each adapter sends with every publisher fallback method. The Fastly closed path test gains /admin%252Fkeys/rotate and /_ts%252Fadmin/ec, which the Axum, Cloudflare and Spin tests already send.
…int and store settings The comments on reader_url_surrogate_key and the shared cache key named the purge endpoint, and the request signing module said the store ids come from the settings, which no longer take them. The changelog said only the removed routes read the two store ids, which the settings checks did too, and now says what GET /_ts/data does not cover.
A configuration that still carries `[[handlers]]` is refused since HTTP Basic authentication was removed, so the settings of `attestation_routes_skip_the_request_preparers`, written with the list before that removal, could not be read and the test stopped at its first step.
This was referenced Oct 17, 2026
# Conflicts: # docs/guide/cloudflare.md # docs/guide/fastly.md # scripts/smoke-axum.sh # scripts/smoke-cloudflare.sh # scripts/smoke-fastly.sh # scripts/smoke-spin.sh
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-cli/srccrates/trusted-server-core/srcdocs/guidecrates/trusted-server-adapter-fastly/srccrates/trusted-server-integration-tests/testscrates/trusted-server-adapter-axum/testscrates/trusted-server-adapter-spin/testscrates/trusted-server-adapter-cloudflare/testscrates/bot-protection/datadomeCHANGELOG.mdcrates/trusted-server-adapter-axum/srccrates/trusted-server-cli/testsscripts/smoke-axum.shcrates/trusted-server-js/libscripts/template-cache-local-test.shcrates/trusted-server-adapter-spin/srccrates/trusted-server-adapter-cloudflare/srctrusted-server.example.tomlscripts/smoke-cloudflare.shscripts/smoke-fastly.shscripts/smoke-spin.shcrates/trusted-server-adapter-spin/spin.tomlfastly.toml.env.dev.env.example.github/workflowsCargo.lockcrates/auction/prebid-servercrates/auction/prebidcrates/tag/google-tag-managercrates/testing/seam-probecrates/trusted-server-adapter-cloudflare/wrangler.ci.tomlcrates/trusted-server-adapter-fastly/Cargo.tomlcrates/trusted-server-integration-tests/fixturescrates/trusted-server-integration-tests/srcscripts/attestation-local-test.shscripts/config-by-host-local-test.shscripts/smoke-common.shCloses
Closes #1271
Spec
Spec: administration-off-the-publisher-domain
Test plan
CI on this pull request at 24a6c79 passed all 32 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)Stacks on attestation (#1353).
This takes out everything that administered a deployment through the domain its readers reach. Each thing goes only after the command or the page that does its job another way is in place, so no commit leaves an operator without a way to do the work.
What this pull request does
POST /_ts/admin/keys/rotateandPOST /_ts/admin/keys/deactivatets keys rotateandts keys deactivate, which run the same rotation code in core and write the two stores through thefastlyCLI with the token inFASTLY_API_TOKENPOST /_ts/admin/cache/purgets cache purge --service-id <id>with--allor--page <url>, which runsfastly service purge --keywith the surrogate key core tags a cached page withGET /_ts/admin/ec,GET /_ts/admin/ec/{id}andGET /_ts/admin/eidsGET /_ts/data(#1350), which shows a reader what is held against that reader's own Edge Cookieapi-keyssecret[request_signing] config_store_idandsecret_store_id--config-store-idand--secret-store-idonts keys. Only the removed handlers read the two settings[[handlers]]and HTTP Basic authenticationThe two commands reach the stores through the platform's API, so they handle three things the routes did not have to.
404from any other failure and write nothing once a read has failed. The404is told apart by the wording thefastlyCLI prints. Only the test fake is known to print it. No recorded run of the CLI has, and the live check below saw401alone.404on a delete counts as success, after asking for the store, because the API also answers404for a store that does not exist.=, in both commands. A key being retired may begin with-, since the rule for retiring is the looser one the deactivate route applied.Neither command takes
current-kidoractive-kidsfor a key. They are the two entries the config store keeps its key list in.closed_path_response, in core's newclosed_paths.rs, stays. Every path beneath/_ts/admin, and the/admin/keysalias beside it, is still answered404by the server before the publisher fallback, on all four adapters. An operator's script that was not updated sends its password and its key payload to a path that answers404, and never to the origin.Each command line the two commands build was given to Fastly CLI 16.1.0, where
.tool-versionspins 15.1.0, with a token that is not one and with store and service ids that do not exist. The CLI accepted every one, and the API refused each with401.Why
Four things were true of a deployment before this change.
api-keysand wrote its own config store and secret store with it.GET /_ts/admin/ec/{id}returned the stored identity graph record for any Edge Cookie ID.[[handlers]]pattern broad enough to cover/_tsanswered/_ts/page-bidswith401for every reader. Bot protection skipped every path beneath/admin/on the publisher's site.A Fastly API token belongs to the user who made it, can be limited to purging by key (the
purge_selectscope) and can be revoked. So administration goes through the platform's own tool with the operator's own token, and the publisher's domain serves readers.What breaks
[[handlers]]is refused when it is validated and when the service starts, with directions. Loading it and ignoring the list would leave an operator who gated a staging site believing it was still gated.[[handlers]]entry could also gate one of this server's own routes,GET /_ts/debug/ja4for instance. Nothing here replaces that, so a route of the server's that should not be public stays switched off.config_store_idorsecret_store_idunder[request_signing]is refused the same way./_ts/admin, and one with[request_signing]and no store ids. So the new configuration and the new build go live together.ts deploy --stagingandts config push --staginglet the pair be checked together first.ts cache purgetakes--service-idandFASTLY_API_TOKENwhere it took--service <url>andTRUSTED_SERVER_ADMIN_PASSWORD, and no longer takes--username. It runsfastly service purge, which the Fastly CLI has had since version 14.0.0.If refusing is too abrupt for a live deployment, the refused keys could load, be ignored and log a warning for one release, and be refused in the next. This pull request refuses, because an ignored
[[handlers]]protects nothing.Closes #1271.