Repository navigation
Make robots.txt a middleware core supplies - #1366
Open
jwrosewell wants to merge 56 commits into
Open
jwrosewell wants to merge 56 commits into
jwrosewell wants to merge 56 commits into
Conversation
Core's own middleware `robots_txt`, in `robots_txt/middleware.rs`, makes the file `[robots-txt]` describes where a `[[fetch]]` or `[[serve]]` entry for `text/plain` on `/robots.txt` names it. Its preparation does what the route's handler did, a refusal of every crawler from nothing and otherwise every selected contributor's held answer assembled, and answers a refusal with `503` and `Retry-After` when a contributor fails with nothing held, so nothing is stored and no empty file is served. Its action replaces whatever the origin sent with the file, and the creating rule makes the file where the origin has none. The registry makes the middleware itself once every registration is in, because the contributors it asks are known only then and a middleware cannot reach the registry. The settings refuse a `[robots-txt]` section that no entry places, because the file would never be made. The route goes from the four adapters, with `handle_robots_txt`, the response builder and the route's own cache lifetime. The file takes the deployment's cache policy for the path, as every file does. The adapters' route tests go with it, and each now proves the path is no route of this server; the file itself is proven through the publisher path in core. The stock list carries the middleware on `text/plain`, and the example configuration places it.
The configuration guide's robots.txt section says the file is made by the `robots_txt` middleware where an entry names it, in place of the origin's file or where the origin has none, that a section no entry places is refused, and that the response takes the deployment's cache policy for the path. Its example carries the entry, and the placing table has the middleware's row. The changelog records the middleware and the route's removal.
…leware # Conflicts: # crates/trusted-server-adapter-axum/src/app.rs # crates/trusted-server-adapter-cloudflare/src/app.rs # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-adapter-spin/src/app.rs # crates/trusted-server-core/src/publisher.rs # crates/trusted-server-core/src/robots_txt/mod.rs # docs/guide/configuration.md # docs/guide/integration-guide.md # trusted-server.example.toml
The settings refuse robots_txt named on any other path or media type and require a fetch entry on /robots.txt. An origin page answered at that path is replaced by the file, as a single-page application's catch-all would otherwise be served to crawlers. Axum again proves a refusal tags every response, robots.txt names resolve through the shared module name rule, the unused registry lookup is gone and the API reference drops the removed route.
The middleware reads its prepared file with MiddlewareContext::take_prepared, and its tests keep and read what they prepare under the phase they run in, as the preparation now keys what it keeps by phase.
# Conflicts: # crates/trusted-server-core/src/integrations/registry.rs # crates/trusted-server-modules/src/lib.rs # docs/guide/configuration.md
# Conflicts: # docs/guide/configuration.md
This was referenced Oct 15, 2026
The middleware's action passed the origin's file on when its preparation had left nothing, so a fault that skipped the preparation would serve whatever the origin holds, or no file at all, which a crawler reads as permission to crawl everything. The action now writes the refusal, with the `always_allow` paths kept open, whatever `modules` selects. No request reaches the action without its preparation today, because core prepares every response it streams, so this is the answer to a fault and not a path in use. The tests write the file with nothing prepared under an `allow_all` selection, and with a value of another type under the middleware's name.
With `refuse_all` selected the origin's answer still reached a crawler wherever the middleware was not handed it as a `200` of `text/plain`. A `401`, `403` or `410` reads to a crawler as permission to crawl everything, a redirect sends it to rules of the origin's choosing, and a file in another charset, in an encoding core cannot read or of another type kept the origin's rules, so the refusal could be loosened. A middleware now says with `writes_whole_document` that it writes the whole document whatever the origin sent. Where a `[[fetch]]` entry with a `path` names one, core makes the document on an empty one in place of any answer it would not hand the middleware as a `200` of the entry's media type, as it already did for a `404`. The robots.txt middleware says so, and the neutral rule replaces the rule for a page at the file's path, which named the middleware in core. A request that names no host is left as it was, because core runs no middleware for it. The module comment and the comment on `refuse_all` say that the origin is still asked and that the file takes the place of whatever it answers. The test sends eleven answers, among them `401`, `403`, `301`, `204`, `206` and files core cannot replace, and each becomes the refusal with status `200`.
… its host What a contributor reported when it failed was kept whole and written to the log, by the holding when it serves a kept answer past its age and by the middleware when it refuses the request, so an address the contributor named with a licence key in its path, query or user part would have put the key in the log. The text now keeps only the scheme and host of each address, and drops the rest of it, before the load error is built from it. The body of the `503` was and is a constant, so nothing of the text reaches a crawler. The test fails a contributor with an address carrying a secret in its user part, path, query and fragment, and a second address in quotes, and reads the error the middleware logs.
An entry's `path` is a prefix, so the entry with `path = "/robots.txt"` also covered `/robots.txt.bak`, `/robots.txtx` and `/robots.txt/rules`. On each of them a `text/plain` answer was replaced by the robots.txt file and a `404` was made into the file with status `200`, where the route this change removes answered `/robots.txt` alone. A middleware now says with `handles_path` which request paths it works on, every path by default, and the registry leaves it out of the chain for any other path the entry covers. Core makes no document where no middleware the entry names works on the path. The robots.txt middleware works on `/robots.txt` alone, where RFC 9309 places the file. The test sends a `200`, a `404` and a `403` for three longer paths, each of which keeps the origin's answer, and a `404` for the file's path with a query, which is made into the file.
Core removes a reader's conditional and range headers before it asks the origin only for a page the ad stack answers and for a personalized HTML document, so a request for `/robots.txt` carried a crawler's `If-None-Match`, `If-Modified-Since`, `Range` and `If-Range` to the origin. The origin could then answer `304` for a copy the crawler holds, which may be the origin's own rules, or `206` with part of its file. A middleware now says with `needs_whole_body` that it needs the origin's whole body, and where an entry of either phase names one for the request's path, core removes the four headers as it does for those pages. The robots.txt middleware says so. The test sends the four headers with a request for the file, which reaches the origin without them, and with requests for two paths no such middleware works on, which keep them.
No adapter test asked for `/robots.txt` once the route was removed, so the file's own `X-Robots-Tag`, a `HEAD` and a `POST` were not tested through an adapter. Each adapter's tests now ask a stub origin that answers each request in turn. A `GET` has the origin's file replaced by the refusal, a `HEAD` has the file made where the origin has none and gets no body, a `POST` keeps the origin's `404`, and each carries the tag. Axum, Cloudflare and Spin run their router with the stub as the origin and buffer the answer. Fastly runs its publisher fallback with its streaming finalizer, then writes the headers its finalizing middleware writes.
The entry for the removed route did not carry the mark the other entries for removed configuration carry in the same block, yet a configuration with `[robots-txt]` and no `[[fetch]]` entry for the file is refused when it loads, and a build from before this change refuses the entry. The entry is marked and says that the two go live together.
The configuration guide said that an answer other than a `404` or a page, such as a `401` from a staging site behind a login, is passed on as the origin gave it. It now says that the file is made in place of any such answer to a `GET` or `HEAD`, that the origin is asked for its whole file and that the middleware works on `/robots.txt` alone, as the code now does. The changelog entry for the middleware says the same, and a new entry and the integration guide describe `handles_path`, `writes_whole_document` and `needs_whole_body` for a module's own middleware.
An origin can answer `304` even when it is asked for its whole file. The test now sends one with the origin's `ETag`, which becomes the refusal with status `200` like the other answers the middleware is not handed whole.
This was referenced Oct 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-core/srccrates/trusted-server-adapter-fastly/srccrates/trusted-server-adapter-axum/testscrates/trusted-server-adapter-cloudflare/testscrates/trusted-server-adapter-spin/testscrates/trusted-server-modules/srccrates/trusted-server-adapter-axum/srcdocs/guidetrusted-server.example.tomlCHANGELOG.mdcrates/trusted-server-adapter-cloudflare/srccrates/trusted-server-adapter-spin/srcCloses
Closes #1283
Spec
Spec: robots-txt-middleware
Test plan
CI on this pull request at 95a9634 passed all 32 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)Stacks on the loading cache (#1365).
What this pull request does
/robots.txtwas answered by a handler of core's own, registered as a route on each of the four adapters with a cache policy of its own, while every other change to what a deployment serves is a middleware named by an entry. It is nowrobots_txt, a middleware of core's own, which runs ontext/plainand prepares.Its preparation does what the route did. A refusal of every crawler needs nothing and gives the refusal document, and otherwise each selected module is asked through its holding, for the host the request was addressed to and never one that
ForwardedorX-Forwarded-Hostnames, and the file is assembled. The action replaces the origin'stext/plainfile with it. Where the origin answers404, or200with a page, as a single-page application's catch-all does, the creating rule (#1363) makes the file. Any other answer passes as the origin gave it, so a staging site behind a login that answers401sends the crawler that401and not the refusal, though every response still carriesX-Robots-Tag. With no answer held for a module that fails, the preparation refuses with503andRetry-After, and nothing is stored.The registry makes the middleware itself, once every registration is in, because the modules it asks are known only then and a middleware cannot reach the registry.
[robots-txt]is refused when the settings load, and so byts config validate, unless a[[fetch]]entry fortext/plainwithpath = "/robots.txt"namesrobots_txt, because only a[[fetch]]entry makes the file where the origin has none. An entry naming it on another path or media type is refused too, because it would replace that file with robots.txt.The name is
robots_txtwith an underscore, unlikerobots-txtfor the section and the module folder andads-txt.sellersfor the stock file middleware, because it is core's own middleware and not a module under a type folder, and core's own names are written with underscores, asjs_asset_proxy,refuse_allandallow_allare.On Fastly a
GETorHEADof/robots.txtis answered once the modules have prepared the request, before the Edge Cookie steps and the request filters, as the route was (#1355), so no filter's answer stands in for the file and nothing is read from the crawler's cookies or written back. The Axum, Cloudflare and Spin adapters run no request filters.Why
#1355 added the route. This undoes its commit 3701fbc in full, which is about 600 of the 692 lines removed here: the route, the handler and eleven adapter tests. The two are separate pull requests because the middleware needs the preparation step (#1364) and the creating rule (#1363), which came after #1355, and #1355 is kept as the change that made robots.txt modules, whatever serves the file.
What breaks
handle_robots_txt, the response builder and the route's ownCache-Control: public, max-age=3600go. The file takes the deployment's cache policy for the path, as every file does, so where the response is stored a module is asked once per stored copy's life, not once per refresh.The route goes from the four adapters, with its path constant in each named-route table and the served-check that existed for it alone. Each adapter's tests now prove the path is no route of this server and that a refusal still tags every response.
A
[robots-txt]section needs the[[fetch]]entry above.How it was verified
No file has been served through an adapter against a real origin. The file is proven through the publisher path with a stub origin, the Axum router is driven with a forwarded host that never reaches a module, the Fastly adapter's tests prove a request filter never stands in for the file, and each adapter's tests prove the path is no route of this server. Locally on Windows the core robots.txt tests (77), the Axum seam probe tests (21), the Fastly adapter tests under Viceroy (272), clippy with warnings refused on core, Axum and Fastly, and
cargo fmt --all -- --checkpassed at 12e6686 and 76e0ec6, which differ in one line of a Fastly test.References #1283.