Repository navigation
Add page changes as middleware - #1351
Open
jwrosewell wants to merge 56 commits into
Open
jwrosewell wants to merge 56 commits into
jwrosewell wants to merge 56 commits into
Conversation
…form A module changes a page through four hook traits today. The integration seam design replaces them with one middleware contract (section 3.7), and asks that each converted implementation reproduces its recorded output. This is the recording, made by the hooks as they are. `html_processor::test_support::assert_page_is_recorded`, behind `test-utils`, runs a page through the document pipeline of a deployment that writes a module's settings and links its builder, and compares what a reader receives with a file, byte for byte. With `TS_RECORD_PAGES` set it writes the file instead. The script bundle's `?v=` hash is replaced by a fixed word, so a recording changes when a module's page change does and not when a browser module is rebuilt. Eleven module crates and core's own script proxy each gain one test and three files under `src/fixtures`, being the settings a deployment writes, the page the origin sends and the page a reader receives. A `.gitattributes` line keeps the two pages' line endings as committed, as the seam probe does for its script. Between them the recordings pass through all twenty hook implementations in this tree, being nine attribute rewriters, three script rewriters, one stream processor and seven head injectors. Two of the head injectors act on a request their module marked, the Google Publisher Tag diagnostics and DataDome's tag, and the recordings are of a request with no mark, so the first writes nothing and the second writes its tag. Two things the recordings show as they are. The Google Tag Manager page holds `'&l='+l` where the origin's snippet holds `'&l='+l`, because a rewritten script is written back escaped. And Prebid's script removal takes out a link whose address matches a script pattern, as it takes out a script.
When a script rewriter replaces a script's text, the HTML processor wrote the replacement with `ContentType::Text`, which escapes `&`, `<` and `>`. A browser decodes no entity inside a `<script>`, so the page ran the escaped text as it stood. `main` does the same. It reached every inline script the Google Tag Manager module rewrites. The standard snippet's `'&l='+l` came out as `'&l='+l`, so a page that names its own data layer asked for a parameter called `amp;l` and its container read the default data layer. A loader of the page's own holding `&&`, `<` or `>` became a syntax error. It reached a `script#__NEXT_DATA__` the Next.js module rewrites as well, wherever the payload holds one of the three characters unescaped. The replacement is written with `ContentType::Html`, which is inserted as it is. A rewriter is handed the text as the page wrote it, so that is the right content type in any element. The recording of the Google Tag Manager page changes by the one line that showed the fault, from `'&l='+l` to `'&l='+l`, and no other recording changes. Four tests, each failing before the change. One in the HTML processor swaps a host inside a script holding `&&`, `<`, `>` and a query string. Two in the Google Tag Manager module run the standard snippet with a data layer name of its own, and a loader holding operators, through the page pipeline. One in the Next.js module runs a payload holding a query string and a value with all three characters. This comes before the page hooks convert, because the middleware contract hands each handler a script's text as the one before it left it and writes it back as it is, and converting a hook must leave its recorded page as it was.
The Google Tag Manager module holds a script's text back while a marker
might still arrive in a later part, and writes it with the script's last
part. A part that arrived after the held text had stopped looking like a
marker was kept as it came, so it went out at once, ahead of the text
held before it, and the script reached the page out of order. `main`
does the same.
The parser hands a script's text over in parts split where a `<` might
close the script, so this needs no help from the network. Before this
change,
footer.innerHTML = '<b>Search with google</b> <i>or</i> <u>another</u> engine';
came out as
footer.innerHTML = '<> <i>or</i> <u>another</u> engine';b>Search with google</b
because the part ending on `google` is held and the parts after it are
not. It also happened when the origin's bytes broke after `google`, or
after a longer start of either marker such as `googletag`, in any inline
script that carried on past the next break.
When nothing held can become a marker any more, what was held is now
written ahead of the part that showed it, and the rest of the script
passes through as before. No recording changes.
Three tests, each failing before the change. One page arrives whole and
writes markup, one arrives in parts with no character that needs
escaping, and one is a script that is held, never rewritten, and holds
`&&` and `<`, which needed the change before this one as well.
A module changes a page through four hook traits, and every hook of a
selected module runs on every page. This adds the other way to make a
page change, which the modules convert to in the commits that follow. A
middleware is a page change a module supplies and the operator's
settings place. It runs only on the pages a `[[fetch]]` entry names it
for, in the order that entry gives.
[[fetch]]
media_type = "text/html"
path = "/news/"
middleware = ["tag.example.cleanup", "tag.example"]
[[fetch]]
media_type = "text/html"
middleware = ["tag.example"]
A page takes the first entry that covers it. An entry places and orders,
and switches nothing on. The section that selects a module still decides
whether it runs, and a middleware reads its settings from its module's
own table.
`trusted_server_core::middleware` holds the contract. A middleware is
asked once for each document, and answers with any of head markup,
markup for straight after the script bundle, element handlers, text
handlers and a stream processor.
What core does with it.
1. The registry takes `.with_middleware(...)` from a registration. At
startup it refuses two registrations that supply one name, a name no
entry could write, and an entry naming a middleware that no running
module supplies, listing the names that could be written. A
middleware no entry names is logged as a warning, because a module
may be selected for what else it does.
2. The settings read `[[fetch]]` and refuse an entry that could not do
what it says, when they load and at deploy validation. That is a
media type other than `text/html`, a path that is not a request path
prefix, an entry that names nothing or names one middleware twice, a
key an entry does not read, and an entry an earlier one already
covers.
3. The publisher path carries the request path to where a document's
processor is made, and the chain for that path is planned there.
4. The HTML processor carries the plan out after the hooks of the same
kind. Every text handler, a hook's or a middleware's, is now handed a
script as the one before it left it. Before, a second rewriter that
matched the same script was handed the origin's text, and its
replacement overwrote the first's.
A fetch middleware is told nothing about the reader, because what it
leaves is what a shared template stores. The entries are part of the
settings a template's fingerprint covers, so changing them selects a
template of its own.
A middleware chooses its selectors for each document, so one that does
not parse cannot be refused at startup. The response fails with a 500
that names the middleware, through both finalizers, where the parser
would otherwise panic.
The seam probe supplies a middleware from outside core, with one of each
handler, and the Axum and Fastly adapters' probe tests cover an entry at
startup.
Tests cover the entries, the registry's checks, what the document
pipeline does with each part of a plan, a whole request through both
finalizers at a covered path and at another, and a stored shared
template. Eleven of them were watched failing with the code broken in
six places, being the chain dropped from the publisher path, the chain
planned for the root path whatever was asked for, text handlers handed
the origin's text, the registry's check removed, and each of the two
settings checks removed.
The guides describe `[[fetch]]` and the contract, and the documentation
test compiles the guide's middleware fixture as it compiles the existing
one.
No stock module supplies a middleware yet, so no page changes.
The commit before this one places a middleware on the page as the origin
sent it, where what it writes is stored with a shared template. Some
page changes belong to one reader. This adds the second phase. A
`[[serve]]` entry names the middleware that run on each reader's copy,
whether the page came from the store or from the origin, and nothing
they write is stored.
[[serve]]
media_type = "text/html"
middleware = ["tag.example.reader"]
A middleware says which phase it runs in. Startup refuses an entry that
names one in the other phase, and the message says which phase it runs
in.
A serve middleware is handed what its module's request hooks left on the
reader's request. A request that carries any was already kept off the
shared template path, so a page from the store has none to hand over.
A page leaves from four places, and the serve entry runs at each.
1. A page that is not stored runs it on the same pass as the fetch
entry, after it, so a streamed page is still streamed.
2. A page transformed as a shared template runs it once the template is
stored and this reader's copy is assembled.
3. A page from the store runs it after assembly in the buffered
finalizer.
4. A page from the store runs it piece by piece in the streaming
finalizer, so the head still goes out before the auction is awaited.
A serve middleware works on a page core has already rewritten, and
nothing of core's own is written twice. Its head markup goes straight
before the script bundle's tag, after the fetch entry's, and its markup
for after the bundle goes straight after the tag.
A page from the store keeps no record of its origin's host. A serve
middleware is told the host from the settings, so it is told the same
on a hit as on a miss.
A serve entry that cannot run fails the response with a 500 at each of
the four places, before a byte of the page is sent.
The seam probe supplies a serve middleware from outside core, and the
Axum adapter's probe tests cover a `[[serve]]` entry at startup, one
that names a serve middleware and one that names a fetch middleware.
Tests drive a serve middleware through a whole request at each of the
four places and through both finalizers, with an auction and without,
and read the stored template to see it holds what the fetch entry wrote
and nothing the serve entry wrote. Each of the four places was then
broken in turn, and tests failed for each. Tests also failed with the
host read from the page's own record, and with the serve entry run
inside the transform of a page that is stored.
The guides describe both phases.
The Axum adapter's probe tests already build an adapter with a `[[serve]]` entry that names the probe's serve middleware, and are refused one that names its fetch middleware. The Fastly adapter is the primary deployment target, so its probe tests now make the same two checks where that adapter builds its state.
A page takes one entry, so a deployment that runs several modules names all their middleware in the one entry that covers those pages. An operator who instead writes an entry for each module was refused with advice to put the entry with the longer path first, which does not help when both entries cover the same pages. That case now says that the second entry covers the same pages as the first, and that the middleware for those pages belong in one entry's list.
The modules convert their page hooks to middleware in the commits that follow, and each module's tests then need the same four things. `AttributeRewrite::each` gives one handler for each of several attributes, all asking one decision, which is the shape nine of the attribute hooks have, a script's address judged in `src` and in `href`. `middleware::test_support::context` is what a test hands a middleware it calls directly. `html_processor::test_support::place_on_every_page` writes the entry that covers every page, and `create_page_processor` builds the processor of a test page with the middleware both phases' entries name. The recording helper builds its processor through it, so a test and a recording run a page the same way.
The GPT module embeds `gpt_bootstrap.js` as it is on disk and writes it into the head of every page. A checkout that rewrites line endings, as a Windows checkout does under `core.autocrlf`, changed the bytes of the script that is served, so a deployment built there served a different page from one built anywhere else. The recorded page showed it. The Axum tests failed on the Windows runner in `the_page_a_reader_receives_is_the_recorded_one`, and giving the file CRLF line endings on a machine where the test passes makes the same test fail. The file is pinned to LF in the crate's `.gitattributes`, as the seam probe pins the script it embeds.
Testlight pointed a `src` or `href` that names `testlight.js` at its shim through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `testing.testlight`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_scripts` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware. This is the first module to convert, so the changelog gains the entry for the whole change, which is breaking for a deployment that selects a converted module and writes no entry. The configuration guide gains the table of the middleware the stock modules supply, with this one row.
Permutive pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `audience.permutive`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware.
lockr pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `identity.lockr`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Didomi wrote `window.__tsjs_didomi` into the head through a head injector, on every page of a deployment that selected it. The script hands the browser module the path Didomi is served under. It is now the middleware `cmp.didomi`, which runs on the pages a `[[fetch]]` entry names it for, and writes the same script. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Sourcepoint changed a page through two hooks, on every page of a deployment that selected it. A head injector wrote the browser module's settings and, with `rewrite_sdk` set, the trap on `window._sp_`. An attribute hook pointed a `src` or `href` on Sourcepoint's CDN at the first-party path, also only with `rewrite_sdk` set. Both are now the one middleware `cmp.sourcepoint`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes under the same setting. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Google Tag Manager changed a page through two hooks, on every page of a deployment that selected it. An attribute hook pointed Google's tag addresses in `src` and `href` at the first-party path, and a script hook did the same in the text of inline scripts. Both are now the one middleware `tag.google-tag-manager`, which runs on the pages a `[[fetch]]` entry names it for and makes the same decisions. The script hook kept the text it was holding in the document's state, under its own id, so that one document's held text could not reach the next. The handler that replaces it is made for one document and holds the text itself. The test that an interrupted document leaves nothing for the next one now asks the registered middleware for a handler twice. The recorded page is unchanged. Its settings gain the entry that places the middleware.
DataDome changed a page through two hooks, on every page of a deployment that selected it. An attribute hook pointed a `src` or `href` that is a DataDome script's address at the first-party path, when `rewrite_sdk` is set. That is now the fetch middleware `bot-protection.datadome`. It makes the same decision under the same setting, on the pages a `[[fetch]]` entry names it for. A head injector wrote the client tag, and left it out for a request the request filter had marked. A change that depends on the request cannot be stored for every reader, so it is now the serve middleware `bot-protection.datadome.tag`, which runs on each reader's copy of the pages a `[[serve]]` entry names it for. It reads the same mark from the document's state and writes the same tag. The design text expected the tag to move to the start of the head. It does not move. A serve middleware's head markup goes straight before the script bundle, which is where the hook's markup went, so the recorded page is unchanged. Its settings gain the two entries.
Prebid changed a page through two hooks, on every page of a deployment that ran it. A head injector wrote the configuration Prebid.js reads and the tag that loads the bundle. An attribute hook removed an element whose `src` or `href` matches `script_patterns`, which is the publisher's own Prebid script. Both are now the one middleware `auction.prebid`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes. Neither depends on the request, so both stay in the fetch phase and what they write is stored with a shared template, as it was. The recorded page is unchanged. Its settings gain the entry that places the middleware.
A browser module can read a setting from the tag the script bundle was loaded by. Two modules use that today, and each sets its attribute through a method of the head injector trait, which goes when the page hooks have converted. The attribute is not a change to a page that an entry places. It belongs to the module's browser code and is the same on every page the bundle is written into. A registration now states it with `.with_bundle_tag_attribute(name, value)`. The registry puts those after the attributes the head injectors still ask for, in registration order, and keeps the first value a name is given as it did. The tag is written as markup with no escaping, so a name or a value that could end the attribute refuses startup, where the tag's writer only asserted it in a debug build. Core's tagging stand-in states its attribute this way, so the test that a page's bundle tag carries it runs through the new path.
APS implemented the head injector trait, wrote no markup through it, and used it only to put `data-ts-aps-rendering-mode="publisher_native"` on the script bundle's tag where a demand source asks for native rendering. It now states that attribute on its registration and supplies no middleware, so no entry names it and no page is changed in its name. The attribute is on the tag in the same case as before, and the recorded page is unchanged.
GPT changed a page through two hooks, on every page of a deployment that selected it. A head injector wrote the `tsjs.adInit` bootstrap, and an attribute hook pointed the GPT script's address at the first-party path when `rewrite_script` is set. Both are now the one middleware `ad-tag.google`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes. The head injector also put `data-ts-gam-attribution="true"` on the script bundle's tag when `gam_attribution_enabled` is set. The registration states that attribute now, so it is on the tag of every page the bundle is written into, as it was. The recorded page is unchanged. Its settings gain the entry that places the middleware.
GPT diagnostics wrote its bootstrap and the tag that loads its module through a head injector, which ran on every page of a deployment that selected it. What it writes depends on the request, because a reader turns diagnostics on with `?ts_console=1` and off again, so it has to run for each reader and never once for a page that is stored. It is now the serve middleware `ad-tag.google.diagnostics`, which runs on each reader's copy of the pages a `[[serve]]` entry names it for. The bootstrap still goes ahead of the script bundle and the module's tag straight after it, and a request that did not activate diagnostics still gets neither. The recorded page is unchanged. Its settings gain the entry that places the middleware, and so do the settings the browser tests run with, which select this module for the diagnostics test.
Next.js changed a page through three hooks, on every page of a deployment that selected it. Two script rewriters worked on the text of the `__NEXT_DATA__` script and of the React Server Components payload scripts, and a stream processor substituted the placeholders the second of them left. All three are now the one middleware `framework.nextjs`, which runs on the pages a `[[fetch]]` entry names it for. Its action carries two text handlers, in the order the rewriters were registered in, and the stream processor. They share the state the module keeps for a document as they did, because all three are made for the document together. The rewriters keep their logic. What each is told about a chunk of text is now a type of this crate with the same fields, less the selector, which neither rewriter read. Their tests change by that type's name alone. The recorded page is unchanged. Its settings gain the entry that places the middleware, and so do core's settings for the Next.js auction tests that the adapters and the parity suite share. The two Google Tag Manager tests that run Next.js alongside it name both.
Core's JavaScript asset proxy pointed a configured script at its first-party path, or removed one that is blocked, through an attribute hook. The hook ran on every page of a deployment that selected the proxy, and ahead of every module's own. It is now the middleware `js_asset_proxy`, which runs on the pages a `[[fetch]]` entry names it for. Its selector asks about the `src` of `script` elements alone, which the hook checked for itself. Its place among the page changes is the place the entry gives it. A middleware is asked about an address as the one named before it left it, so the proxy is named first where it has to see the address the page wrote. One test names it ahead of a module and another names the module ahead of it, and the guide says to name it first. The tag stand-in that core's tests run beside the proxy becomes a fetch middleware in the same change, because the two were tested together through the order of their hooks. The test that asked the hook about an element that is not a script is removed. The selector decides that now, and the page test that holds an image with a configured address covers it. A test is added for a script tag written in upper case. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Core's tests stand two modules in for a vendor's. One carries a deferred browser module and writes its settings into the head, and the other rewrites script payloads in two passes. Both made their page change through the hooks. Each now supplies one fetch middleware under its own name, and a test that runs one on a page places it with an entry. The payload stand-in kept its captured payloads and its held script text in the document's state, because one rewriter served every document. Its text handler and its stream processor are now made for one document, and share what the handler captures between the two of them. Two tests checked that the payload stand-in had registered a stream processor before going on. They check that its middleware runs on the page. The test that two configurations of a module never share a template now also checks that each page carries the markup its own configuration writes. Nothing looked at what the deferred stand-in wrote before.
…ware Core's tests stand one module in for a vendor's that acts on one request. Its preparer or its filter marks the request, and its head injector read the mark and wrote a script ahead of the bundle and its module's tag after it. What it writes is one reader's, which the page path kept out of a shared template by handing a shared mode no request state. It now supplies one serve middleware under its own name, which reads the mark from the state the request left. A serve middleware runs on a reader's copy and nothing it writes is stored, so the same output is kept out of a shared template by where it runs. The tests of a request a module acts on place the middleware with a `[[serve]]` entry. The tests that render whole documents compose the serve entry as the production function does, which runs none on a template every reader shares. They also run a serve middleware that writes for every reader, marked or not, and refuse its output in a shared template. Without it a template that ran the serve entry would still pass, because a shared mode is handed no request state and the stand-in writes nothing without a mark.
Core moves the origin's address in `srcset` and `imagesrcset`, as it does in `href`, `src` and `action`, and no test said so. Taking either rewrite out left every test passing. This test runs both through the document processor, with a bare host among the candidates of the `srcset`, and a `srcset` that names another host, which is left as it is.
Every module that changed a page through a hook now supplies a middleware, core's JavaScript asset proxy and its test stand-ins included, so nothing implements the hooks any longer. Removed from `trusted-server-core`: - `IntegrationAttributeRewriter`, `IntegrationScriptRewriter`, `IntegrationHtmlStreamProcessorFactory` and `IntegrationHeadInjector` - their context types, `IntegrationAttributeContext`, `IntegrationScriptContext`, `IntegrationHtmlContext` and `IntegrationHtmlStreamContext` - the registration methods `with_attribute_rewriter`, `with_script_rewriter`, `with_html_stream_processor` and `with_head_injector`, and the registry's four lists behind them - `AttributeRewriteOutcome`, which was the registry's answer after asking every attribute rewriter - `ScriptTextAccumulator`, which kept a script's text in the document's state because one rewriter served every document. A text handler is made for one document and keeps its own. The document processor asks nothing of the registry about a page any more. It moves the origin's address in the five attributes it always has, and carries out the plan of the middleware an entry names. A request's state reached a document through the hooks, and a function handed a shared mode none of it, so that nothing made for one request could be stored. That function is removed with its reader. What a request left now reaches a document through the `[[serve]]` entry alone, which does not run on the pass that builds a template every reader shares, and the fetch chain is handed none of it. The tests that render whole documents fail if the serve entry is run on a shared template. Tests of a hook are removed where a middleware test already asserts the same thing. The test that a registration from the auction plan goes ahead of a section's module reads the order from the middleware names the registry lists. The guides describe a middleware as the one way a module changes a page, and the changelog lists what is removed.
A tool that writes a configuration, as `ts audit` does, selects the modules it finds on a page. A module's page changes run only where an entry names its middleware, and a tool had no way to learn those names without naming a module itself. `trusted-server-modules` now lists the middleware the stock modules supply, core's script proxy among them, each with the integration that supplies it and the phase it runs in, in the order the registry registers the modules. `middleware_for` gives the names for a set of integration ids and a phase. The list is held to what the modules register. One test builds a registry from the settings each module's recorded page is made with and compares what it registers for each phase, and another holds the order to the stock builders'. The registry gains `middleware_in`, which gives the names that run in a phase.
`ts audit generate` writes a draft configuration that selects the modules it can configure from what it finds on a page. A module changes a page only where an entry names its middleware, so a draft with no entry selected modules that changed nothing. The draft now ends with one `[[fetch]]` entry, and one `[[serve]]` entry where a selected module supplies a serve middleware, that run those modules' middleware on every HTML page. The names and their order come from the stock list, so the tool names no middleware itself. The script proxy is placed with the rest, so an asset an operator enables after review is rewritten as well as served. `trusted-server.example.toml` documents the two entries with every middleware the stock modules supply. The stock list's tests hold the template's entries and the configuration guide's table to the list, in order, so neither can fall behind a module. The guide's table is put in that order, and the changelog says how a deployment keeps a page as it was when it upgrades. The lockr guide and the getting started guide say a page change needs an entry, as the other module guides do.
The tests of a module's page change kept the names of the hooks they once called, such as `head_injector_emits_config_script` and `attribute_rewriter_rewrites_gpt_urls`, which name traits that no longer exist. They are renamed for the parts of a middleware they now test, `head_markup_`, `element_handler_` and `text_handler`. Thirty-nine names and section comments change in eight module crates, and no test's body does. Four comments in core still spoke of script rewriters. They speak of text handlers, and the description of the state a document's middleware share says what it holds in each phase. The test of the payload stand-in's placeholders is renamed, because the stand-in no longer keeps them in that state.
The integration guide said a text handler's replacement is written with no character escaped. It now says that a `<` that would end the script, or begin `<script` after `<!--`, is written as `\u003c`. The configuration guide's field table says `/` covers every path, as no path does, and the changelog entry for page changes as middleware states both.
…again Core moved the origin's address in `href`, `src` and `action` with three replacements in a row, for the `https://`, `http://` and protocol-relative spellings, in `imagesrcset` with the same three, and in `srcset` with a fourth for the bare host. Each read the previous one's output. When the request host begins with the origin host, a later spelling matched inside an earlier replacement and the request host's tail was written again. With `publisher.example` served as `publisher.example.localhost` over `http`, every `href` came out as `http://publisher.example.localhost.localhost.localhost/`, and over `https` with one extra `.localhost`. The bare-host replacement in `srcset` did the same to a `www.` host in front of an apex origin, writing `https://example.com/a.png` as `https://www.www.example.com/a.png`. `replace_each_once` scans a value once from the start. It takes the earliest match of any spelling, the longer of two that start at the same place, writes that spelling's replacement and carries on after the matched text, so no replacement is read again. `UrlPatterns` builds the spellings once per document. A value that begins with the bare host is still moved only when the host ends there. The tests check that a request host which begins with the origin host is written once in `href`, `src`, `action`, `srcset` and `imagesrcset` over both schemes, that a hyphenated host and a `www.` host over an apex origin are written as before, and `replace_each_once` on its own.
This was referenced Oct 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-core/srccrates/ad-tag/googledocs/guidecrates/tag/google-tag-managercrates/framework/nextjscrates/trusted-server-modules/srccrates/testing/seam-probecrates/bot-protection/datadomecrates/trusted-server-cli/srccrates/auction/prebidcrates/cmp/sourcepointcrates/testing/testlightcrates/audience/permutivecrates/identity/lockrcrates/cmp/didomicrates/trusted-server-adapter-axum/testscrates/auction/apscrates/trusted-server-adapter-fastly/srctrusted-server.example.tomlcrates/trusted-server-integration-tests/testsCHANGELOG.mdcrates/trusted-server-integration-tests/fixturescrates/trusted-server-core/.gitattributescrates/cmp/osanocrates/trusted-server-js/libAGENTS.mdcrates/trusted-server-integration-tests/README.mdCloses
Closes #1269
Spec
Spec: page-middleware
Test plan
CI on this pull request at 0157d13 passed all 32 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)Stacks on the inspection endpoints (#1350).
It builds section 3.7 of the seam design (#1342).
A module changes a page through a middleware and in no other way, and a middleware runs only on the pages where an ordered entry of the settings names it. Every module that changed a page is converted, core's own script proxy included, and the four page traits are removed.
What this pull request does
Each phase has an ordered list of entries. An entry covers
text/html, optionally only the requests under a path prefix, and names the middleware to run in order. A page takes the first entry that covers it. An entry places and orders, and switches nothing on. The section that selects a module still decides whether it runs.[[fetch]][[serve]]The contract is
trusted_server_core::middleware. A middleware is asked once for each document and answers with any of five things.<head>ahead of the script bundleA registration offers one with
.with_middleware(...). The guide has two fixtures a vendor can copy, and the documentation test compiles both as written.What converted
Each conversion is one commit. The first commit of this pull request recorded the page each module produced through its hooks, as a file in the module's crate, and every conversion leaves its recording byte for byte as it was.
auction.prebid, fetchjs_asset_proxy, fetchtesting.testlight, fetchframework.nextjs, fetchaudience.permutive, fetchidentity.lockr, fetchcmp.didomi, fetchcmp.sourcepoint, fetchtag.google-tag-manager, fetchbot-protection.datadome, fetch, andbot-protection.datadome.tag, servead-tag.google, fetchad-tag.google.diagnostics, servewith_bundle_tag_attributeTwenty implementations of the four traits became thirteen middleware, because a module's hooks that work on one page are one page change.
Comments in
prebid/index.tsand the Osano test that named a head injector name the module's middleware.Where the serve entry runs
A page leaves from four places, and the serve entry runs at each.
A page from the store keeps no record of its origin's host. A serve middleware is told the host from the settings, so it is told the same on a hit as on a miss.
What is refused, and when
text/html, a path that is not a request path prefix, an entry that names nothing or names one middleware twice, a key an entry does not read, an entry an earlier one already covers, and two entries for the same pages, which are told to share one list500that names the middleware, before a byte of the page is sent, where the parser would otherwise panicA name is checked when the service starts and not at deploy validation, which is where a section's module names are checked and for the same reason. A crate the
tstool never links may supply the name.Three faults found on the way
Recording the pages showed two faults in how a rewritten inline script is written back, both on
mainas well, and CI on Windows showed a third. Each has tests that fail without its fix.&,<and>. A browser decodes no entity inside a<script>, so the standard Google Tag Manager snippet's'&l='+lreached the page as'&l='+l, and a loader holding&&or<became a syntax error. It reached a rewritten__NEXT_DATA__holding those characters too.footer.innerHTML = '<b>Search with google</b> <i>or</i> <u>another</u> engine';reached the page out of order.include_str!and written into the page. A checkout that rewrites line endings changed the bytes of the page, which a Windows runner caught. The file is pinned to one set of line endings.The first two commits stand alone and can be taken on their own. Core moves the origin's address in
srcsetandimagesrcset, and no test said so. A test now does, added before the hooks were removed and passing on both sides of the removal.Why
A module changed a page through four hook traits,
IntegrationAttributeRewriter,IntegrationScriptRewriter,IntegrationHtmlStreamProcessorFactoryandIntegrationHeadInjector. Every hook of a selected module ran on every page, in the one order the modules were registered in. A publisher could not run a change on part of a site, could not choose the order, and could not place two changes of one module apart. Whether a hook's output could be stored in a shared template rested on what the hook declared about itself.What breaks
A deployment that selects a module and writes no entry gets no page change from it, and startup logs a warning that names each middleware no entry names.
trusted-server.example.tomldocuments one[[fetch]]and one[[serve]]entry naming every middleware the stock modules supply, in the order the modules' hooks ran in. A deployment keeps the names of the modules it selects.ts audit generatewrites the entries for the modules it selects, in that order.crates/trusted-server-moduleslists the stock middleware (middleware(),middleware_for(...)). Its tests hold the list to what the modules register, and hold the example configuration and the configuration guide's table to the list.One order is worth knowing. The script proxy knows a script by the address it is handed. It ran ahead of every module a section selects, and it now runs where the entry puts it, so it is named ahead of a middleware that moves the address of a script it is to proxy or block.
Removed:
IntegrationAttributeRewriter,IntegrationScriptRewriter,IntegrationHtmlStreamProcessorFactoryandIntegrationHeadInjector, with their four context types.AttributeRewriteOutcomeandScriptTextAccumulator. A text handler is made for one document and keeps its own text, so the per-document buffer that guarded against one document's text reaching the next has nothing left to guard.[[serve]]entry alone, which does not run on the pass that builds a template every reader shares, and a fetch middleware is handed none of it. This is the rule section 3.7 said would become structural.IntegrationMetadatacarries amiddlewarelist, the names of the middleware the integration supplies, in place ofattribute_rewriters,script_selectorsandhead_injectors.The document processor no longer asks the registry anything about a page. It moves the origin's address in the five attributes it always has, writes the bundle, and carries out the plan of the middleware an entry names.
Closes #1269.