Skip to content

Add the sellers a deployment appends to the publisher's ads.txt - #1307

Closed
Jamesr51d wants to merge 546 commits into
IABTechLab:mainfrom
jwrosewell:split/20-ads-txt
Closed

Jamesr51d wants to merge 546 commits into
IABTechLab:mainfrom
jwrosewell:split/20-ads-txt

Conversation

@Jamesr51d

@Jamesr51d Jamesr51d commented Oct 12, 2026 •

Copy link
Copy Markdown

Stacks on middleware on other media types (#1306). The additions that follow the seven pull requests open here are listed in #777.
Compare split/19-media-types with split/20-ads-txt
to see only this pull request's change.

What this pull request does

crates/ads-txt/sellers, a module that writes the lines the settings give at the end of the publisher's ads.txt.

[ads-txt]
modules = ["sellers"]

[ads-txt.sellers]
lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0", "CONTACT=ads@publisher.example"]

[[fetch]]
media_type = "text/plain"
path = "/ads.txt"
middleware = ["ads-txt.sellers"]

The module is a middleware for text/plain, the first to use the entries for a media type other than HTML (#1306). Named in a [[fetch]] entry on the file's path, it passes the origin's file through and writes the lines after its last one, on a line of their own. The entry's path is a prefix, as for every entry, so /ads.txt also covers /ads.txt.bak and /ads.txt/anything. The refusal and the guide say so rather than this pull request giving one entry an exact path, because PhaseEntry::covers is shared by every entry.

StockMiddleware gains media_type, and middleware_for now leaves a middleware for a media type other than HTML out of the page entry ts audit writes.

Why

A publisher who sells through this server has sellers to declare that their own ads.txt does not know about, and a buyer's crawler that does not find a seller in the file refuses that seller's inventory. The publisher's file stays theirs: it is read from the origin as it is, and the deployment adds to it. A publisher whose origin has no file gets the origin's answer, because the module adds to the publisher's file and never invents one.

What is checked

Every buyer's crawler reads the file and a typo in it costs the publisher revenue silently, so each line is checked as the settings load, and by ts config validate, against the file's grammar.

Line Accepted
domain, account id, DIRECT or RESELLER with an optional fourth field, the certification authority id Yes, the relationship in either case
CONTACT=, SUBDOMAIN=, INVENTORYPARTNERDOMAIN=, OWNERDOMAIN= or MANAGERDOMAIN= with a value Yes, the name in either case
A comment, an empty line, a record with two or five fields, no domain, a domain with a space, no account id, another relationship, a trailing comma, a variable with no value or another name Refused, naming the line and what is wrong
An empty list Refused
A document that selects the module and names it in no [[fetch]] entry for text/plain Refused, because the lines would never be written

How it was verified

CI passed on 6fdc503e8 on the fork. Not exercised: no file has been fetched from a real origin through an adapter. The middleware is run on a body read in pieces, and its place in a registry's chain for the file's path is checked.

References #1278.

Comments on the adapters and two test notes in core described what the code
did before it changed. Each now says what the code does and why.

- The resolved Edge Cookie module held on the Cloudflare, Fastly and Spin
  application state is resolved when the state is built, because resolving
  reads no request data.
- The three tests that a selected module the adapter cannot build fails the
  request say what a default context would do.
- The Spin state builder reads its settings from the config store, and its
  test says why the failure has to be the absence of a config store.
- The test that keeps the internal header list in step with the Edge Cookie
  response headers, and the test for an unknown key in a module block, say
  what they guard.

No code changes.
- The Cloudflare region test says why the region header has to reach the
  privacy outcome.
- The Fastly geo module's re-export and the geo selector's default are
  described as they are.
- The integration fixture's note on the geo selector no longer promises a
  setting that a later change does not add.

No code changes.
- The integration fixture carried two notes on its geo selector that
  contradicted each other. The one that stays says that Viceroy maps the
  loopback client to US/CA, so the platform geo module resolves a real place.
- That fixture, the Viceroy template, the template cache harness and the
  permission model guide no longer mention a default country setting, which
  no released version had.
- Five code comments say what the code does without saying what it did
  before.

No code changes.
Core builds the identity-graph key from the module's code and whatever
`normalize_id_for_kv` returns, so the method decides what two visits must
have in common to be treated as the same visitor. Its documentation
described normalization only and advised a module with a case-sensitive
identifier to return the value unchanged. That is right for a stable
identifier and wrong for one that carries a signature, a nonce or a
timestamp, because each reissue then gets a row of its own and the identity
does not survive it.

The trait documentation now says that the returned value is the identity two
visits must share, and that a module whose identifier has a part reissued
each time must return the stable part. It also says that core asks
`accepts_id` about the returned string as well as about the identifier as
issued, so a module must accept its own canonical form, or no row is read or
written for it.

`two_issues_of_one_identity_share_one_identity_graph_key` drives a fixture
module whose identifier is a stable part and a part that changes on each
issue through `AcceptedModules::canonical_kv_key`. Two identifiers that
differ only in the reissued part reach one key, and a different stable part
reaches another. The test fails for a module that returns the value
unchanged.

No behavior changes.
The Edge Cookie guide still said that an unknown country falls back to a
configured default country, and that only the storage permission is resolved
from a TCF signal. Neither is how the permission model works.

- When no country is identified, or neither the region nor the country has a
  rule, the baseline on the top node of the rules tree applies.
- Every permission is resolved from that baseline and the signals, each
  signal read by its own permission signal module in the configured order,
  with the Model Terms for Marketing preference among them.
- A location lookup that fails resolves at the requires-signal floor.

The setup guide describes the bare identifier form by what it is, a value
with no module prefix, rather than by when it was issued.
- The context query parameter example names `[ad-server.mock]`, the table
  the mock ad server's settings are read from.
- A deploy validation test, the jurisdiction acknowledgment, two selector
  tests, the registry's geo test, the settings fixture and the integration
  fixture say what the code does without saying what it did before.

No code changes.
The Fastly adapter was a binary only, and the function that builds its
state with a deployment's integration builders was private to the crate. A
deployment that shipped a vendor crate had to edit the adapter, and the
round trip that proves the integration seam ran on the Axum dev server
alone, where the seam design asks for it on the primary deployment target.

The adapter is now a library with a thin binary over it. `run` serves the
way the binary does. `run_with` records the integration builders a
deployment offers, before any request is served, and they are composed with
the built-in ones when the application state is built. A deployment's binary
is one line, and offering a builder does not switch it on, because a builder
runs only when the settings select its module.

    fn main() {
        trusted_server_adapter_fastly::run_with(vec![a_vendor_crate::builder()]);
    }

The round trip runs under Viceroy on wasm32-wasip1, in
`src/app/seam_probe_tests.rs`, driven by the seam probe, a crate core does
not know about. Through this adapter's own router:

- the probe's carried browser module is served in the unified bundle under
  the hash composed from its content,
- its proxy route reports the country its own geo module resolved and that
  its request preparer ran exactly once, on a named route and on the
  fallback,
- a geo selector naming a module that declares no geo module fails at
  startup, naming the module and the capability,
- two builders claiming one id are refused, naming both sources,
- the Edge Cookie and device selectors resolve the modules the probe
  declares, and its Edge Cookie module creates an identifier the built-in
  module cannot make,
- the builders registered through `run_with` reach the state the application
  hooks build.

The seam probe is a test-only dependency of the adapter, as it is of the
Axum adapter.
The guide described an integration inside core only. A section on a vendor
crate was written for this branch in August and was lost when the guide was
rewritten on `main`, so the seam had no developer documentation beyond its
tests. The section is back, written for the selection model the branch has
now.

- What the crate provides: an `IntegrationBuilder` with an id, a source, a
  build function, a validate function, a module name taken from the crate's
  folder, and optionally a request preparer and auction implementations.
- What a registration can declare, including a carried browser script and the
  Edge Cookie, geo and device modules the three selectors can choose.
- How an adapter composes it in, with `routes_with_registrations` on Axum,
  Cloudflare and Spin, and what a Fastly deployment has to do instead.
- Two traps: the carried script's hash literal against line endings, and a
  vendor's deploy rules not running through the CLI.

The integrations overview points at the section, and the registration
checklist no longer asks for a disabled default, because an integration runs
when a section selects its module.
… crate

The integration guide's section on integrations outside core said the Fastly adapter could not take a vendor crate. It now shows the one-line binary a deployment writes around `run_with`, and points at the round-trip tests that run on the Fastly adapter. The changelog records that the adapter is a library with a thin binary over it and that the Wasm artifact keeps its name.
…point

The Fastly adapter's entry point moved from `main.rs` to `lib.rs`, so the comments in `app.rs` and `middleware.rs` that point at it, and the two onboarding tables, name `lib.rs`. No code changes.
`[ec] module`, `[geo] module` and `[device] module` picked a module a
registration declared by that registration's integration id, a hand-written
name where every other selection reads a module name.

A registration now declares each of the three under a name, the path under
`crates/` of the crate the module lives in, and the selectors read a written
name the way every section does, as written or with the type folder
(`edgecookie`, `geo` or `device`) in front. A geo module from
`crates/geo/example` is `[geo] module = "example"`, and one registration can
supply a module of each type, each under the name of its own crate.

- `with_ec_module`, `with_geo_module` and `with_device_module` take the name.
  An Edge Cookie module is declared under the name its `id` returns, and the
  registry refuses a registration where the two differ, because core checks
  the selected module by its id. Two registrations that supply a module of one
  type under one name are refused, naming both.
- `[ec] module` resolves the implementation its block names, as core does, so
  a label of the operator's own reaches a registered module. It looked for a
  registration under the label itself before, so a labelled block never
  reached one.
- A `[geo] module` or `[device] module` that names nothing running lists the
  modules of that type the deployment runs, and says when the name is a
  module no section selects or one that supplies no module of that type.
- The seam probe declares its three modules under `testing.seam-probe` and
  reports that name from its route. Its integration id stays the name of its
  routes and its browser module.

The module that supplies one of these still has to be selected in its
section, because the registry builds only what a section selects.

The configuration reference loses a paragraph and a heading that appeared
twice above `[ec.hmac]` and says that `[device]` holds no module settings
table where it implied one, and the auction README names the `modules` and
`module` keys where it still said `provider`.
…ition branch

The merged-in registry reads a registration's geo, Edge Cookie and device
modules by module name, so the Fastly round trip selects the probe's three as
`testing.seam-probe` and `seam_probe_tests.rs` changes with the merge.
…ings load

The settings are validated as they load, and that validation compiled the
auction plan with the built-in implementations alone. A deployment whose
`[demand]` or `[ad-server]` named an implementation one of its own builders
supplies was therefore refused while its settings loaded, before the adapter
built the state that compiles the plan with those builders. Deploy validation
had the same split, taking the builders for the block checks and compiling
the plan without them.

- `validate_settings_for_runtime_with`, `settings_from_config_blob_with` and
  `get_settings_from_config_store_with` carry a deployment's builders through
  the load, and the functions without the suffix call them with none. Each
  builder's validate function runs as part of that load, so a vendor's own
  rules hold at startup for a deployment that composes it.
- `validate_settings_for_deploy_with` compiles the plan with the builders it
  was given, so it and the running service agree on which names are valid.
- Four items an auction crate outside core needs are public:
  `UPSTREAM_RTB_MAX_RESPONSE_BYTES`, `collect_response_bounded`,
  `ensure_integration_backend_with_timeout` and
  `predict_integration_backend_name`. A crate rolling its own would bound a
  response at another size and derive another backend name for the same URL.
- The seam probe supplies an ad server, so that seam is reached from a crate
  core does not know. Two Axum tests start the adapter and run deploy
  validation with the probe's ad server selected, and both refuse the same
  settings without the probe's builder.

Four tests in core assert both halves, through the plan, runtime validation,
deploy validation and the real settings load, so each fails if the built-in
path stops refusing the name or the path with the builder stops accepting it.
The deploy validation test failed before this change. A fifth shows a
builder's own rejection surfacing from runtime validation.

The integration guide shows the load with builders, and a doc comment names
`[demand] modules` where it said `provider`.
…ition branch

The merged-in settings load takes a deployment's builders, so the Fastly
adapter hands it the ones `run_with` registered, as it hands them to the
state build. A round trip test builds this adapter's state with the probe's
ad server selected and shows the same settings refused without the probe's
builder. The guide keeps both accounts, the load with builders and the
Fastly library.
The Fastly entry point built each request's device signals with
`build_device_module`, which knows only `builtin` and `fastly`. With
`[device] module` naming a module a registration supplies, the registry
resolved that module and the adapter put it in the request's services, and
nothing ever asked it to classify a request, so the built-in User-Agent
module answered in its place.

The entry point now hands `derive_device_signals` the module the registry
resolved. A registered module is shown the User-Agent and the cookies, which
carry what a page gathers in the browser. With no registered module the
built-in or the Fastly module answers as before, shown the User-Agent alone.

Two tests in the adapter show a registered module classifying the request
and seeing its cookies, and the built-in module answering when nothing is
registered. `build_device_module`'s doc comment said a name it does not know
is refused by the settings, which is the registry's job, and says so.

Fastly is the one adapter that classifies a request. The Axum, Cloudflare
and Spin adapters resolve the selected device module and set no device
signals, on `main` as here.
…ition branch

The merged-in entry point hands the device module the registry resolved to
`derive_device_signals`. That function lives in `lib.rs` on this branch,
so the change is made there and `main.rs` stays the thin binary. A ninth
round trip test selects the probe's device module and shows it classifying a
request through the function the entry point calls, with the built-in module
answering when the selector is unset.
The probe supplied an ad server, so one half of the auction seam was reached
from a crate core does not know and the other was held by tests inside core
alone. Its builder now registers a demand source as well, one that sends the
standard request and reads every answer as no bid, which is enough to show
that everything a demand source needs is reachable from outside core.

An Axum test starts the adapter and runs deploy validation with the probe's
demand source selected, and both refuse the same settings without the
probe's builder. The probe's own tests pin both implementation names to its
module name and show the demand source refusing a setting it does not read.
…ition branch

The merged-in probe supplies a demand source, so a tenth round trip test
builds this adapter's state with it selected and shows the same settings
refused without the probe's builder, as the ad server's test does.
The Fastly adapter becomes a library with a thin binary, so a deployment
that ships an integration crate writes its own binary and passes that
crate's builder to `run_with`. The seam's round trip then runs on Fastly
under Viceroy from a crate core does not know, as it already does on Axum.

The branch was built and checked apart from this one so the seam could be
read without it. It is merged here because the seam's design asks for a
crate outside core to register on every adapter, and Fastly was the one
adapter where that could not be shown.
…ords

`KvNetwork` said a low cluster count "indicates an individual or
household". A count of connections is a fact about a network, and reading
a low one as an individual describes a device record as a record about a
person. It now says a small network, such as a home connection.

`KvEntry::ids` now says what it holds, which is each partner's identifier
for the same browser on the same device, never an identifier for a person.

Comments only, with no change in behavior.
Jamesr51d and others added 24 commits October 13, 2026 14:28
…d run the Cloudflare auction test with no demand modules

The build script's TRUSTED_SERVER__GIT_VERSION reaches test processes and the strict loader refused it as an unknown field. Cloudflare runs one demand source at a time, so the fixture's two demand modules stopped worker startup.
The wasm clock lint reaches core's build script, which runs on the host.
Core's version_header module is removed, with its call from the finalize
step of the Axum, Cloudflare and Spin adapters and from the Fastly adapter's
health probe and its settings fallback, so no response carries x-ts-version
unless a [[serve]] entry names the service_version middleware. That
middleware now writes the git version the build compiled in from
TRUSTED_SERVER__GIT_VERSION or local git as X-TS-Version, when the build
knows one, and the version the adapter reports for the service as
X-TS-Service-Version, when it reports one. HEADER_X_TS_FASTLY_VERSION, which
nothing wrote, goes with the module.

The health tests of the three adapters prove the probe carries no version,
the middleware's own tests prove each header is written when its version is
known and nothing when neither is, and the publisher tests prove both
headers reach a buffered page and neither reaches a page no entry names the
middleware for. The configuration guide, the getting started guide, the
EdgeZero guide and the changelog say the same.
# Conflicts:
#	docs/guide/configuration.md
@jwrosewell

Copy link
Copy Markdown
Collaborator

Moved into this repository as #1361, one branch of the stack, so the review continues there. This pull request is closed in its favor and its comments stay here for the record.

@jwrosewell jwrosewell closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants