Repository navigation
Conversation
Both the Axum and Fastly adapters call the module-aware compiler now, and the only bare calls left are in test modules that import it themselves.
Renaming the integration table shortened a cell, so two rows needed re-padding. The Fastly test module re-imported the plain plan compiler through the app module and calls it by its full path, so it stops.
The Axum and Fastly test fixtures supply no integration builders, so they compile the plan without them. They were reaching the plain compiler through the adapter's own import, which the adapter no longer needs.
Main wrote a disabled JS asset proxy block into the starter template and then replaced it. Here the template documents every integration as a commented example and the audit appends the block it fills in, so its tests count only entries at the start of a line and expect the commented example to still be a comment. One of them was also still reading the removed `[integrations]` table out of the parsed draft. The ad template diagnostics report `[demand] provider`. The two fixtures that isolated "no providers configured" remove the demand table rather than a key inside `[auction]`, and the one that wanted two sources adds the second to the existing table rather than writing a second `[demand]` header, which is a duplicate key. The report sorts the names it prints, which the old map gave for free and a selection list does not. The integration guide's documented geo fixture implements an asynchronous `lookup` that takes the request's services, and the crate its source is compiled in gains the async-trait dependency that needs, resolved from the workspace the same way error-stack already was.
The Fastly test module's import of the plain plan compiler went in at the top of its group rather than in sorted order.
Takes the identity graph changes merged upstream on 28 September (conditional writes, tombstones from a snapshot, grouped batch sync, the pull sync marker and the EID sync source) and the reusable sandbox in the Fastly entry point, and keeps the provider seam on top of them. Conflicts, settled file by file: - crates/trusted-server-adapter-fastly/src/app.rs: upstream's sandbox lifecycle. The seam's settings_with_missing_consent_store test helper had no caller after the merge and is dropped. - crates/trusted-server-core/src/ec/batch_sync.rs: mappings are validated inside upstream's grouping loop, keyed by the owning provider's canonical form through accepted_providers.canonical_kv_key. The seam's three provider tests stay, followed by upstream's renamed fan-out test. - crates/trusted-server-core/src/ec/pull_sync.rs: build_pull_sync_context keeps the provider dispatch for the identifier and its key, then applies upstream's gate (pull-enabled partners, a consented row still missing a partner UID). PullSyncContext carries the key, and dispatch returns early with no pull partners. - crates/trusted-server-core/src/ec/finalize.rs: provider response headers are applied first, then the marker is validated and the gate read. The returning-user, generated and recovery paths key the snapshot, EID ingestion and sync by the provider's canonical key, and withdrawal tombstones each provider-derived key through tombstone_existing_from_snapshot, warning on a failed tombstone. Upstream's new code and tests assumed ec.passphrase and helpers that take no provider, which is not this branch's shape: - The pull sync marker key derives from the selected provider's HMAC passphrase, or from the proxy secret for a provider that has none, since ec.passphrase is only the deprecated location here. - Upstream's tests pass the accepted providers to process_mappings, the provider to handle_batch_sync_with_writer, the gate flag to the finalize context helpers, and a registry with a pull partner plus a seeded snapshot to build_pull_sync_context. Tests that set ec.passphrase select the hmac block instead, and the redaction canary reads ec.hmac.passphrase. The seam's EID ingestion test opens upstream's new EID sync source gate, as upstream's own returning-user tests do.
EcContext captured the request headers, path and query only when no usable identifier arrived, while orphan recovery runs only when one did. So a provider asked for a replacement identifier during recovery saw a request with no headers, although the docs said recovery reads the evidence captured at read time. A provider that derives identity from client hints then ran with no client hints on every recovery attempt. The snapshot is now also captured for a document navigation, the only request that can recover, so a returning visitor's subresource requests still clone nothing. The field doc says when it is captured. Test recovery_on_a_navigation_passes_request_parameters_and_cookies_to_the_provider fails without the change.
a_provider_reads_request_cookies_from_the_request_info built its own OwnedRequestInfo with a Cookie header and called a test double's generate with it, so it tested OwnedRequestInfo::header and nothing in core. The test and its CookieCapturingProvider double are gone, and the tests module no longer imports OwnedRequestInfo. The recovery test added in the previous commit covers a request's cookies reaching a provider through the request path.
remove_labeled_provider_secret_errors strips the error that judged a labeled block's secret key name as a passphrase. Nothing checked that it strips only that one, so it could drop the Edge Cookie section's other errors and no test would fail. push_validation_keeps_the_sections_other_errors_for_a_labeled_block selects [ec] provider = "primary" with a primary block for hmac holding the key name ec_key, adds a partner whose source domain carries a scheme, and expects the section's errors to hold the partner error and no passphrase error. They are read from the section rather than from the message, because deploy validation reports the same bad partner.
Comments and docs say what the code does. History and plans come out of them: the legacy reader retirement plan on provider_owns_id, the plan that the built-in provider becomes a module, the reader list seam on AcceptedProviders, the design-document references, the pull request narrative around the test doubles, and the "still built into core" wording. The header-appending rule on apply_provider_response_headers is stated in three sentences, the resolution docs say what resolves where, and edge_cookie.rs says it reads an inbound identifier and that its generation helpers are test-only. The ec module summary lists admin, finalize and provider. Tests that could not fail now can, and repeated tests are folded into tables. One OpaqueProvider double in ec::tests serves the admin lookup, batch sync and pull sync tests in place of four copies. The batch sync canonicalization test asserts the key the writer was given. The request path reuse test drives a built-in selection through an unthreaded services value, since the threaded helper already threads the provider. The uninjected provider test covers build_provider and the startup check together. The reserved response effect tests are one table over each header with and without an identifier, and the three discarded-candidate tests are one table. The withdrawal key cases and the HMAC grammar cases are tables. The consent gate test has an open-gate control and a backed context, so only the gate can withhold the cookie. New tests: a provider that creates nothing still has its headers applied; an orphan recovery that cannot rotate leaves a failed snapshot and no cookie, for a provider with no client IP, a store whose write fails and a replacement that always collides; an orphan rotation applies the replacement provider's response headers; both RequestInfo views list their header names; and the request path hashes an IPv6 client by its /64 prefix, which replaces the edge_cookie test that compared two test helpers. Tests say create where they said mint. noop_services_with_resolved_ec_provider duplicated noop_services_with_ec_provider and is gone.
What an operator selects to supply a capability is a module, and provider keeps the one meaning it has on main, an auction provider instance. This renames the Edge Cookie identity seam this series adds, and only the words it adds: every provider word was compared with upstream main, and a word that exists there, in the auction code or the EID extensions, is untouched. The retired [ec.providers] table keeps its literal in the refusal that names its replacement. - ec/provider.rs is ec/module.rs, so ec::provider is ec::module. - EdgeCookieProvider is EdgeCookieModule and HmacProvider is HmacModule. - EcProviderSelection, EcProviderBlock, EcProviderBlocks and EcProviderSettings are EcModuleSelection, EcModuleBlock, EcModuleBlocks and EcModuleSettings, HmacProviderConfig is HmacModuleConfig, and Ec's provider and provider_blocks fields are module and module_blocks, so the selector reads [ec] module and the environment override is TRUSTED_SERVER__EC__MODULE. - ProviderCode and the provider_code! macro are ModuleCode and module_code!, and HMAC_PROVIDER_KEY, BUILTIN_PROVIDER_KEYS, HMAC_PROVIDER_CODE, PROVIDER_CODE_SEPARATOR and PROVIDER_IMPLEMENTATION_KEY say MODULE. - build_provider, build_shared_provider, request_provider, provider_owns_id, apply_provider_code, provider_kv_key, apply_provider_response_headers, ensure_provider_available, split_provider_code and resolve_named_provider say module, as do AcceptedProviders and accepted_providers, selected_provider, validate_provider_selection and validate_provider_name, and remove_labeled_provider_secret_errors. - RuntimeServices' resolved_ec_provider and with_resolved_ec_provider, on the services and their builder, say module. - Test helpers, test doubles and test names in the Edge Cookie code say module. - The plain word, where it means an Edge Cookie module, in the Edge Cookie code, settings.rs, config.rs, config_payload.rs, test_support.rs, proxy.rs, testlight.rs, the adapters' entry points, the example settings, the integration fixture, the API reference and the edgecookie crate directory's README.
Brings the refreshed Edge Cookie seam, and through it upstream main at 7a0ecb4, into the device and geo branch. The seam's names say module, so every line here that used one of its names says module too, and the conflicts settled as that rename applied to this branch's own text: - The Fastly entry point keeps this branch's device and geo wiring with upstream's cache directive helper and PlatformCacheIntent, and drops the RuntimeServices and PlatformKvStore imports upstream no longer needs. - The cargo aliases keep this branch's device and geo crates and gain upstream's test-fastly-reuse alias. - In the seam, the host-signal implementation keeps its resolution arm, its key constant and its two tests, and the uninjected-module test covers build_module and the startup check together as the seam's does, with this branch's host_signals argument. - build_reusable_module keeps its three arguments, and the composition roots call it by that name. This branch's own device and geo names still say provider, and are renamed in a commit of their own.
The Edge Cookie settings refused the old host-signals spelling with a message naming the spelling to write. No configuration in this repository uses it, so the constant, the refusal in validate_module_selection, names_retired_module, the test of the refusal, the doc paragraphs describing it and its row in the error reference are gone. An old spelling now fails as any unknown implementation does when the module is built at startup, with a message listing the implementations the deployment has, which an_unknown_implementation_fails_naming_the_known_ones covers. The settings test kept the one assertion that was not about the old spelling, that a host_signals block is read as the built-in module's own settings, as a test of its own.
The fixed secret list already holds ec.hmac.passphrase and ec.host_signals.passphrase, but labeled_provider_block_names also listed a block named after one built-in that configures the other. The same path was then resolved twice, the second time reading the secret as a key name, and the load failed with a message that read as a missing store entry. A block named after either built-in is now left off the labeled list, because the fixed list covers it whichever it configures. Test a_built_in_block_naming_the_other_built_in_resolves_its_passphrase_once fails without the change.
What an operator selects to supply a capability is a module, and provider keeps the one meaning it has on main, an auction provider instance. This renames the words this branch adds on top of the Edge Cookie seam, which already says module, and only those words: every provider word was compared with upstream main, and a word that exists there is untouched. - DeviceProvider is DeviceModule, and BuiltinDeviceProvider, FastlyDeviceProvider and the test stand-ins are BuiltinDeviceModule, FastlyDeviceModule and so on. DeviceConfig's provider_key is module_key, and build_device_provider and build_geo_provider are build_device_module and build_geo_module. - HostSignalProvider is HostSignalModule, HostSignalsProviderConfig is HostSignalsModuleConfig, and HOST_SIGNALS_PROVIDER_KEY is HOST_SIGNALS_MODULE_KEY. - The [device] and [geo] selectors read module, as [ec] does, so the environment overrides read TRUSTED_SERVER__device__module and TRUSTED_SERVER__geo__module, and labeled_provider_block_names is labeled_module_block_names. - Test helpers and test names say module, and the plain word says module wherever it meant a device, geo or Edge Cookie module, in the core code, the adapters, the device and geo crates, the example settings, the fixtures and the guides.
Brings the refreshed device and geo branch, and through it the Edge Cookie seam and upstream main at 7a0ecb4, into the permission model branch. The lower branches' names say module, so every line here that used one of those names says module too, and most conflicts settled as that rename applied to this branch's own text. The rest, settled by hand: - Upstream removed the legacy consent store, so the consent pipeline's ec_id and kv_store inputs, its read fallback, its KV pipeline tests and the Fastly test helper that named a missing consent store are gone, and this branch's default jurisdiction input stays. The consent-era tests of allows_ec_creation and gate_eids_by_consent do not return, because the permission model replaced those functions here. - Withdrawal is the permission signal modules' answer on this branch, so the pull sync marker expiry upstream added reads storage_withdrawn, and the consent-derived flag upstream computes above the gate goes. - The generated-cookie gate test is upstream's two-case table, worded for the permission gate. - build_pull_sync_context keeps this branch's sharing gate inside upstream's new signature, which takes the partner registry and returns nothing when no partner is pull-enabled. - The test services keep this branch's geo helper beside the renamed Edge Cookie helper, and the publisher keeps gate_eids_by_permissions with upstream's sharedid cookie import.
The one-time warning told the operator that the jurisdiction would be unknown and Edge Cookie creation would fail closed. On this branch neither is true. With no location the caller supplies the declared jurisdiction, which is the permission policy's top node, so identity and the auction both keep running. The warning described behavior from before the policy tree existed. That matters because it is the line an operator reads when advertising stops. Being told identity is blocked when it is allowed sends them looking in the wrong place, and a deployment that selects no geo module sees this message on every start. It now says what happens rather than only what is missing. The jurisdiction is the declared one rather than one resolved from the request, country and region rules cannot match, and a lookup that fails, as opposed to resolving nothing, is the case that does fail closed.
A permission whose baseline requires a signal, and for which every configured module answered neutral, was simply not in the resolved set, so a page could not tell "nobody has answered yet" from "refused". A prompt that never loaded then looked like a visitor who said no. PermissionState gains awaiting, the permissions in exactly that position, recorded by resolution alongside the set. It reaches the page as a third key of window.tsjs.permissions beside set and tdls, and the page core normalises it like the other two. Nothing yet acts on it. That is the next step, the page core holding what depends on an awaited permission.
The awaiting list said every permission that requires a signal and got none. Some of those no module in the deployment could ever grant, the marketing channel opt-ins in the shipped sample policy for one, and a page told to wait for them would wait for ever, which is the defect the list exists to remove. Each module now declares, through grants on the trait, which permissions it can answer Grant for under the policy it is given. An opt-out declares nothing. TCF declares every Data Use a purpose maps to, and nothing when the policy silences the record. The assembly narrows awaiting to the union of those declarations, so a permission nothing here could grant is simply unset. Start-up compares the policy against the declarations and warns, naming every permission the policy requires a signal for somewhere that no configured module can grant. It warns rather than refuses, because the shipped policy itself carries two such permissions and a deployment that never uses those channels is not misconfigured. PermissionSet gains intersect, union and without. PermissionMaps gains requires_signal_anywhere, the union over the whole tree. The guide and the seam's README describe awaiting.
…alid ones Core no longer answers for any scheme. The override that revoked everything ahead of the modules when a record arrived unreadable is gone, and with it ConsentContext::has_malformed_record. What an absent, unreadable or expired signal means for the permissions is the decision of the module for that scheme, taken in its own answer and taken silently. The four shipped modules read their own unreadable record as the refusal it may have carried, on the permissions their scheme covers, and say nothing about any other scheme, so the configured order decides a readable record from one scheme beside an unreadable one from another, as it decides everything else. The permission state gains signals, one ValidSignal for each signal a configured module read and found valid, as received, with the module and the scheme. The page reads it as the fourth key of window.tsjs.permissions. A signal nobody vouched for goes no further, because after assembly the consent context keeps only the raw strings the valid signals name, so a corrupt or expired record never reaches a bid request. The start-up comparison of the policy against the modules, added in the previous commit, is removed. Nothing is logged about a signal.
…ookie A fifth permission signal module, crates/permission-signal/mtm, identifier mtm. It reads the answer a preference platform recorded, one of the three words standard, personalized and non-marketing, from the first party cookie __mtm_pref. The name carries no vendor in it, so any platform that asked the visitor may set it, and Trusted Server reads the cookie and never writes it. With no word present the module says nothing, so the country and region rules stand, which is clause 5.3 of the Model Terms. standard and personalized both grant storage, contextual advertising, measurement, content performance, market research, product improvement and basic content selection. Only personalized grants the four Data Uses that depend on browsing history or interactions, and standard refuses them rather than leaving them alone, because the visitor chose against them. non-marketing changes nothing: the visitor named neither kind of marketing, so the module has no marketing choice to act on and the country and region rules decide. Under a group where every marketing Data Use requires a signal nothing is granted, and under one where they are granted the baseline applies. A deployment that wants a decline to bind there sets that in its own permissions policy. No word withdraws. The word is the valid signal, as received, and whenever one is present the module declares the versioned Model Terms document as the terms the data is available under. Any other value is treated as absent. Every adapter offers it last, after the four IAB schemes. The TCF module's declaration of what it can grant is computed once rather than on every request. A test over a platform's real Set-Cookie strings pins that the cookie is never HttpOnly, because a browser lets a script neither read nor replace such a cookie and the platform that owns the question would stop seeing its own answer.
What an operator selects to supply a capability is a module, and provider keeps the one meaning it has on main, an auction provider instance. This renames the words this branch adds on top of the device and geo seams, which already say module, and only those words: every provider word was compared with upstream main, and a word that exists there is untouched, as is the icu_provider crate the permissions inspector's lock file names. - PermissionSignalProvider is PermissionSignalModule, and GpcProvider, GppSaleOptOutProvider, UsPrivacyProvider, TcfProvider and MtmProvider are GpcModule, GppSaleOptOutModule, UsPrivacyModule, TcfModule and MtmModule. - build_permission_signal_providers is build_permission_signal_modules, the adapters' shipped_signal_providers is shipped_signal_modules, and the permission_signal_providers field and arguments say modules. - The [permission_signal] selector reads module, as the seams' selectors do, so the refusal of the removed sources key names module. - Test doubles, test helpers and test names say module, and the plain word says module wherever it meant a permission signal, geo, device or Edge Cookie module, in the core code, the five module crates, the adapters, the sample policy, the fixtures and the guides.
The five permission signal crates carry their own unit tests, and no CI job ran them, because the test-axum alias covered the adapter crate alone. The test-axum and clippy-axum aliases now name the five crates as well, so the Axum job tests and lints them.
Comments say what the code does. The "today" qualifiers go, along with the claim that no shipped geo module reports a lookup failure and the counts of shipped schemes, which change as schemes are added. Device storage gates every shipped Edge Cookie module, not only the built-in one. The fifteen tests of how from_yaml refuses a malformed policy are one table, each case named with the refusal it must report, so a new refusal is one row rather than a new test shaped like the last one.
Brings the refreshed permission model branch, and through it the device and geo seams, the Edge Cookie seam and upstream main at 7a0ecb4, into the client-resolve branch. The lower branches' names say module, so every line here that used one of those names says module too, and most conflicts settled as that rename applied to this branch's own text. The rest, settled by hand: - The Fastly entry point keeps this branch's resolve route import beside upstream's EcContext and EidSyncSource imports, and constants keeps the resolved-cookie marker beside upstream's pull-complete cookie. - The seam keeps this branch's client_fixed implementation id and its retired spelling, with the built-in list naming all three ids, under the trimmed docs the seam now carries. - The settings keep this branch's refusal of the client-fixed spelling and its startup compilation check, with the host-signals refusal gone as the device and geo branch removed it. This branch's own names still say provider, and are renamed in a commit of their own.
The sandbox tests load settings that select the HMAC Edge Cookie module through the deprecated passphrase form and no geo module. On this branch that pair needs the single-jurisdiction acknowledgment, as every other test settings document in the adapter carries, so without it the settings refuse to load and the test traps under Viceroy before it starts.
An [ec] resolve_allowed_origins entry such as "https://www.example.com/" loaded cleanly, then matched no request, because origins_match treats any value that is not a serialized origin as matching nothing. Every POST to the resolve endpoint from that origin answered 403, with only a warn line saying why. The parse inside origins_match is now parse_serialized_origin in ec/resolve.rs, and it returns the reason a value is not a serialized origin. origins_match reads the request's Origin and each entry through it, and Ec::validate_resolve_allowed_origins refuses an entry it rejects, naming the entry and the reason. The check runs in Settings::finalize_deserialized beside the module selection check, which covers from_toml, from_json_value and the runtime config store load, and in validate_settings_for_deploy_with, so ts config validate and ts config push refuse the entry before it can stop an instance loading its settings. parse_serialized_origin is stricter than the old closure on both sides. It requires http:// or https:// and then nothing from a slash, a backslash, a question mark, a hash, an at sign, whitespace or a control character before handing the value to the url crate for the host and port. The old closure let through userinfo, the ftp, ws and wss schemes, a path that normalizes to a slash, a backslash, a missing double slash and surrounding spaces, each of which matched itself. A browser sends only http or https as the Origin of a page's POST, so no real request that matched before stops matching. The field doc, docs/guide/configuration.md and trusted-server.example.toml say each entry must be a bare origin and is refused otherwise.
handle_ec_resolve rebuilt its module with build_module, passing the module the composition root had threaded into the request as the injected one. On Fastly that is the built-in module itself when hmac is selected, so build_module found the built-in name claimed twice and the endpoint answered 500 for a valid selection, where it should answer 204 for a module that creates nothing from a client post. The same happened to client_fixed in a build that compiles it, which is the one built-in module whose page script posts here. The endpoint now uses EcContext::selected_module, the module the request's own context resolved. The context's host signal and injected module fields existed only for the rebuild, so they and their accessors are removed. The resolve tests built contexts with no module and relied on the rebuild, so their helper now resolves the selected module the way a request's context does. Tests resolve_answers_204_for_a_threaded_module_that_creates_nothing and resolve_sets_the_cookie_for_a_threaded_client_fixed_module read a context with the module threaded through the services, as Fastly does, and fail without the change.
The Edge Cookie settings refused the old client-fixed spelling with a message naming the spelling to write. No configuration in this repository uses it, so the constant, the refusal in validate_module_selection, names_retired_provider, the test of the refusal and the doc paragraphs describing it are gone. An old spelling now fails as any unknown implementation does when the module is built at startup, with a message listing the implementations the deployment has, which an_unknown_implementation_fails_naming_the_known_ones covers.
set_provider_ec_cookie validated and wrote the same cookie as set_ec_cookie, which no longer checks the HMAC identifier shape either, so the resolve endpoint had a second copy of one function. The copy is deleted, handle_ec_resolve calls set_ec_cookie, and the client_fixed module's doc links to it. The three tests of the copy fold into the set_ec_cookie tests, as two tables. One writes an HMAC identifier, an opaque value and a base64url value verbatim with the security attributes, and the other refuses a semicolon, a carriage return and line feed, a line feed and a space with no cookie at all. The second fails when set_ec_cookie stops validating.
# Conflicts: # crates/trusted-server-adapter-fastly/src/main.rs # crates/trusted-server-core/src/test_support.rs
main's new EID persistence gate is the permission model's sharing check. EIDs are written to the identity graph only when the storage and personalised-ad permissions are both set, the pair that already gates the bid request, so allows_eid_persistence and its parity test are removed and the tests that described Purpose 4 denial state the permissions instead.
main's rule that partner EIDs are written to the identity graph only with consent for personalised ads is stated in the guides as the storage and personalised-ad permissions, the pair the permission model already uses for sharing.
…d run the Cloudflare auction test with no demand modules The build script's TRUSTED_SERVER__GIT_VERSION reaches test processes and the strict loader refused it as an unknown field. Cloudflare runs one demand source at a time, so the fixture's two demand modules stopped worker startup.
The wasm clock lint reaches core's build script, which runs on the host.
jwrosewell
marked this pull request as ready for review
October 14, 2026 12:35
Collaborator
|
Moved into this repository as #1355, one branch of the stack, so the review continues there. This pull request is closed in its favor and its comments stay here for the record. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on the removal of administration from the publisher's domain (#1298). The additions that follow the seven pull requests open here are listed in #777.
Compare
split/13-removalswithsplit/14-robots-txtto see only this pull request's change.
With
[robots-txt]in the settings a deployment answers/robots.txtitself, from rules that two names of core's own and any selected module give. Without the section nothing changes, and the path reaches the publisher's origin like any other.What this pull request does
modulesalways_allowsitemapSitemap:linetop_text,bottom_text[robots-txt.<name>]<name>, which that module reads and core does notrefuse_allandallow_allare core's own names and need no module. Any other name is a module declared with the newwith_robots_txt_contributor, and core names none of them. The code calls such a module a contributor, after the extension point below, which #1313 replaces.A module returns a
Contribution, which is groups of records, and core writes every byte. That is what lets the following hold whatever a module returns.refuse_allselected the file is the refusal and nothing else, whatever else is selected.always_allowstays open to every crawler a module refuses the whole site to,*included, whether the refusal is writtenDisallow: /or with wildcards such asDisallow: /*. Core adds a group of its own below the modules' rules and never edits them.modulesnames them, between the publisher'stop_textabove and theSitemapline andbottom_textbelow.A module is asked through the module context (#1293), with the request's method, path, host and scheme. It is not given the query or anything about who asked, because its answer is served to every crawler. Its answer is held in the key-value store and served until the module's own
refreshis due. The key is a SHA-256 hash of the publisher's domain, the module's name and itsfingerprint.refresh503withRetry-After: 600andCache-Control: no-storeGETandHEAD /robots.txtare registered on all four adapters, only when the settings carry the section. On Fastly and Axum the path is a row of the adapter's named-route table.HEADis answered with the headers of theGETand no body. Every other method on the path goes to the publisher fallback as before.The settings refuse
modulesabsent, empty or naming one thing twice, a key that is not a setting, a block for a modulemodulesdoes not name, and analways_allowentry that is not a path on one line. The registry refuses, when it is built, a name that nothing the deployment runs supplies.Why
modules = "refuse_all"refuses every crawler in the file, and putsX-Robots-Tag: noindex, nofollowon every response, which tells a crawler that fetched anyway not to index. The header is written last when a response is finalized and replaces any tag already there.404or an empty file reads to a crawler as permission to crawl everything. Failing closed is easier to get right once, in core, than in every module that wants to add a rule.How it was verified
CI passed on
c66316b4aon the fork. Not exercised: no vendor module has contributed rules through the extension point yet, so the seam probe's contributor is the one proof of it, and the key-value store's lifetime on Fastly and Cloudflare is set but not read back.References #1272.