Repository navigation
Open the integration seam so a vendor module can live outside core - #1094
jwrosewell wants to merge 377 commits into
Conversation
|
Sequencing note on the While checking today's activity I found that A merge simulation of
Reproduce: The request is the one we made on #940. Please land this stack (#1043 to #1047, #1084 and this PR, all based on |
35584db to
b64b45c
Compare
c6dd589 to
54337f0
Compare
1bfb2e6 to
95be9fb
Compare
…r needs Every provider trait method was synchronous while every platform I/O service is asynchronous, and a provider was handed no services at all. So a provider could not make a backend call, read a key-value store, or fetch a secret. That made every real vendor provider impossible to write, which defeats the point of having a pluggable seam. `EdgeCookieProvider::generate`, `DeviceProvider::detect` and `PlatformGeo::lookup` are now async and take `&RuntimeServices`. `EdgeCookieProvider::resolve_from_client` gets the same treatment, because it is the client-side counterpart of `generate` and its own documentation requires an implementation to verify what the browser posted, which generally means reading a secret or calling the vendor's backend. Leaving that one synchronous would have recreated the defect on the single method that most needs the services. The three traits use `#[async_trait(?Send)]` while keeping their `Send + Sync` bound, which is what `PlatformHttpClient` already does and for the same reason: the provider object stays safe to share and run multi-threaded, and only the future it returns is pinned to one thread, because the host SDKs produce `!Send` futures on wasm32. No provider keeps a synchronous method, the built-in HMAC, client-fixed and host-signal providers included. One that does no I/O declares an async method that returns immediately rather than taking an escape hatch, so the runtime decides how the work is driven and the implementer only has to make the type safe to share. Geo turned out not to be the problem it looked like. It is called just ahead of permission resolution rather than from inside it, and its result is passed in as a `GeoStatus`, so making the lookup async stops at `EcContext::read_from_request_resolving_geo` and the permission assembly stays synchronous. Two places had no services to hand a provider. The Fastly finalize middleware and the entry-point finalize path both run after the handler, where the per-request services are out of scope, so they now build the graph once through `build_finalize_services` and apply the request's client metadata with the new `RuntimeServices::with_client_info`. The entry point itself is synchronous host code, so it drives the two async calls with `futures::executor::block_on`, at the same boundary it already drives the router. `resolve_geo_for_response` took a synchronous closure, which an async lookup cannot satisfy, so it is now the predicate `geo_allowed_for_response` and each caller awaits the lookup itself. The 401 skip rule it exists for is unchanged. Two tests prove the services arriving at a provider are the caller's real ones, rather than the parameter being inert. Each drives a provider that resolves a value out of the config store it is handed, through the production path, and asserts the value reaches the output. Substituting a different services graph anywhere in either chain fails them. Addresses: crates/trusted-server-core/src/ec/provider.rs, crates/trusted-server-core/src/ec/device.rs and crates/trusted-server-core/src/platform/traits.rs, where the three provider seams were synchronous and blind to the platform services.
…mplementation Fix the seam-probe fixture crate's own unit tests so they compile and pass (A14): add tokio to dev-dependencies, build a valid RuntimeServices from self-contained stub stores plus core's unavailable implementations, make the geo test async, and give the fixture settings the now-required `[geo] default_country`. Add a host-target seam-probe test step to the CI job so the fault cannot recur unseen. Add a round-trip test (A13) that drives the module's Edge Cookie provider through `generate` and asserts the `seam-probe-` prefix the built-in HMAC provider can never produce, and narrow the selection test's doc to point at it. Document the identity and device capabilities (A7): add `.with_ec_provider(...)` and `.with_device_provider(...)` rows to the integration guide, identity and device rows to the CLAUDE.md capabilities table, and correct the Validate row and the geo default wording. Correct doc comments that contradict the code (B4): the geo resolver run-on and its "unset means host lookup" claims, the Fastly service-graph doc attachment, the "module-supplied geo provider" docs that omit identity and device, the router-level `# Errors` that omit the Edge Cookie provider failure, the AppContext and prepare-request docs, the registered-builder-ids and 256 KB base64 claims, and the seam-probe module and validation docs. House-style over lines this branch added: fingerprint to signals or hash by sense, mint to create or derive, several to a plain quantity, initialise to initialize.
Adapters applied a module's provider through with_ec_provider, which wrote a second RuntimeServices slot beside the one the composition root threads. Section 3.6 specifies one identity injection path, so the registry's answer now lands on the resolved seam the request path already reads, and the second slot is gone. The registry matched the selector against the registered modules before this point, so the module's provider is the resolved answer rather than an input the selector still has to weigh.
DeviceProvider::required_permissions is public but has no enforcement point. Device classification runs before the permission set for a request is assembled, so there is no per-request gate to check a declaration against, and selecting a provider is an operator decision rather than a per-request permission decision. Before the module seam that was a documented no-op, because core and the host supplied the only two providers and neither declares anything. With the seam a vendor crate can implement the trait, so the method reads as a promise that is not kept. Refuse the selection at registry resolution, loudly and at startup, rather than honoring the declaration by ignoring it. The refusal lifts when a real per-request device gate exists. Raised by Christian Pavilonis in review of IABTechLab#1084.
…-refusal message Prettier realigns table columns in the two guide pages the seam change widened, which the format-docs workflow checks. The startup refusal for a module device provider that declares permissions used a string literal continued across lines without a trailing backslash, so the message carried runs of spaces. It now reads as one sentence.
# Conflicts: # crates/trusted-server-core/src/auction/endpoints.rs # crates/trusted-server-core/src/ec/finalize.rs # crates/trusted-server-core/src/test_support.rs
# Conflicts: # crates/trusted-server-adapter-fastly/src/main.rs # crates/trusted-server-core/src/test_support.rs
main's new EID persistence gate is the permission model's sharing check. EIDs are written to the identity graph only when the storage and personalised-ad permissions are both set, the pair that already gates the bid request, so allows_eid_persistence and its parity test are removed and the tests that described Purpose 4 denial state the permissions instead.
main's rule that partner EIDs are written to the identity graph only with consent for personalised ads is stated in the guides as the storage and personalised-ad permissions, the pair the permission model already uses for sharing.
…d run the Cloudflare auction test with no demand modules The build script's TRUSTED_SERVER__GIT_VERSION reaches test processes and the strict loader refused it as an unknown field. Cloudflare runs one demand source at a time, so the fixture's two demand modules stopped worker startup.
|
Moved into this repository as #1348, one branch of the stack, so the review continues there. This pull request is closed in its favor and its comments stay here for the record. |
Add the integration seam and one selection convention for every type
Stacks on the module documentation set (#1047), the last of the five pull
requests that decompose #838, so this pull request sits on the whole module
series. It also depends directly on the
EdgeCookieModuleandDeviceModuletraits from the Edge Cookie module seam (#1043) and the device and location
seam (#1044), whose modules a registration can now carry, and on the
permission model (#1045), where an unset
[geo] modulemakes no host geocall. The stack has six pull requests (#1043, #1044, #1045, #1046, #1047,
#1094), each targeting
main, with this one sixth.Compare
split/5-response-hook-docswithsplit/7-integration-seam-implto see only this pull request's change.
The design specs for the series are carried by the spec pull request (#1084),
which has no code. This pull request implements
2026-08-27-integration-provider-seam-design.md,where sections 3.1 to 3.6 define the seam and section 8 records what
implementing it found, so read #1084 first and this pull request as the answer
to it.
What this does and why
On
main,trusted-server-corecarries nine vendor integrations as core code because every place an
integration plugs in is closed. The builder table is private, browser
JavaScript is fixed at compile time, deploy validation names each vendor's
configuration type, the auction names its vendors in core, and two vendors
reach into core through named types. Every new vendor is therefore another core
change, most recently the LiveRamp module (#1054).
This pull request opens those places and then moves every integration
module through them. An integration ships in its own crate with its Rust, its
configuration type, its deploy rules and its tests, it can carry its browser
script too, and an adapter composes that crate into a deployment at startup
without core naming the vendor. No existing integration changes behavior, the
same modules compose to the same served bytes and
?v=hash, and the modulesa stock build ships register through the same path a vendor crate uses. One browser module's own bytes change, because the Prebid
module's console error for a bidder with no adapter names
[auction.prebid.bundle.modules].What is now composable:
IntegrationBuilder::new(id, source, build, validate), passed to an adapter'sroutes_with_registrations, which builds the registry withIntegrationRegistry::with_plan_and_registrations.with_js_module(CarriedJsModule { source, sha256 })on the registrationvalidate_settings_for_deploy_withget_settings_from_config_store_withandsettings_from_config_blob_with, which validate against the builders a deployment composes and run each builder's validate function.with_request_preparer(...), run byIntegrationRegistry::prepare_request.with_response_finalizer(...), run byIntegrationRegistry::finalize_responsewith what the module's request hooks left for the request.with_secret_settings(...), looked up as the settings load when a section selects the module.with_plan_registration(...)and.with_plan_validator(...), for a module whose page support follows what the plan selects.with_ec_module(name, ...),.with_geo_module(name, ...)and.with_device_module(name, ...), each selected by that name in[ec] module,[geo] moduleand[device] module.with_module_name("<type>.<name>")on the builder, selected in[<type>]withmoduleormodules.with_demand(...), selected by[demand] modules.with_adserver(...), selected by[ad-server] moduleOne convention for every type
Everything a deployment can switch on is a module, selected the same way: the
Edge Cookie identity, the location and device lookups, the permission signals,
the demand sources in an auction, the ad server that picks the winner, and the
page integrations.
The types are
ec,geo,device,permission-signal,demandandad-server, and the section of each page integration's type,cmp,tag,ad-tag,bot-protection,identity,audienceandframework, withauctionandproxyselecting modules beside their own settings. A typethat runs one takes
module, a string, a type that runs several takesmodules, a list, and a name with nothing to set needs no table at all. A table's name is the implementation it configures,unless the table carries an
implementationline naming the implementationby its module path. A demand table always carries one, because
demandisnot the type its implementations are named under, which is how two Prebid
Servers run side by side under names a deployment chooses for itself. A name
is parts joined by
., each of lower case letters, digits,_or-. Amodule from a crate is named by its folder below
crates/, so it may bewritten in full, as
permission-signal.gpc, or with its section's type leftoff. Core's own modules such as
hmactake bare names, anda
[demand]or[ad-server]name is snake_case because it may be a labelof the operator's own.
Five things stop a deployment rather than being tolerated:
[<type>.<name>]table that its type's selector does not name.misspelt table of Trusted Server's own meets.
implementationline, naming an implementation thisbuild does not have. The message lists the ones it does have.
it does not recognize.
demandorad-serverendpoint that is not HTTPS. Plain HTTP is allowedonly to
127.0.0.1,::1orlocalhost, so a local test stack runs withoutcertificates and nothing leaves the machine unencrypted.
A configuration written for the previous spelling is told the new one rather
than half read: a
providerkey in[demand]or[ad-server]is refused withthe key to write, an
[adserver]table is refused with its new name, and an[integration]table is refused with where modules are selected now.The rules the auction tables and the module sections share live in one
place,
crates/trusted-server-core/src/provider_table.rs,whose
SectionModulesreads a section'smoduleormodulesand thetables of the modules it selects, and
TypeSectionsinsettings.rsreads every top-level table core does not read itself as one. The whole
convention, the checks and the move from the previous layout are written up for
operators in
docs/guide/configuration-rules.md.The auction side selects modules the same way
The ad server, plain OpenRTB, Prebid Server and APS are implementations that
a crate registers through
.with_demand(...)and.with_adserver(...)on itsbuilder, selected by
[demand] modulesand[ad-server] module. The fourthat ship here are
auction-protocol.openrtb,auction.prebid-server,auction.apsandad-server.mock, incrates/auction-protocol/openrtb,crates/auction/prebid-server,crates/auction/apsand
crates/ad-server/mock,and core's own list offers none.
What an implementation may decide is stated once, in
crates/trusted-server-core/src/auction/demand.rs.A shared OpenRTB driver builds every standard request field, sends the request
and enforces the privacy rules, and an implementation chooses only its field
policy, its own extensions, its outbound headers and how its responses are
read. The compiled plan in
crates/trusted-server-core/src/auction/plan.rsresolves each selected name to a registered implementation, so
crates/trusted-server-core/src/auction/profile.rsand its fixed list of threeprofiles are deleted.
main's Prebid Server changes since the fork sit inside that split. A storedrequest is sent only when the slot's intent allows the fallback, an
implementation that has no demand for an impression marks it omitted, the
driver leaves those impressions out and makes no request when none is left,
and the parse state keeps only the slots that were sent, so a response cannot
be read against an impression that never went.
[auction]keeps the settings that belong to the auction itself, beingenabled, the whole-auction timeout and creative handling, with[auction.bidders.<code>] modulenaming the demand source a browser biddercode is sent to.
The module interfaces are asynchronous, and modules get the services
Before this change every module trait method was synchronous while every
platform service was asynchronous, and a module was handed no services at
all, so a module that needed to call a backend, read a key-value store or
fetch a secret could not be written. The methods that do the work on all three
module interfaces (
EdgeCookieModule::generate,DeviceModule::detectandPlatformGeo::lookup) are now asynchronous and take&RuntimeServices, andso is
resolve_from_client, which has to verify what the browser posted.Every implementation was converted, the built-in ones included. Finalizing the
Edge Cookie response is asynchronous too and takes the services, because
main's orphaned identifier recovery asks the selected module for areplacement identifier.
The traits keep their
Send + Syncbound and use#[async_trait(?Send)], asPlatformHttpClientin this repository already does, so a module stays safeto share while its future stays on one thread.
Two tests in
crates/trusted-server-core/src/ec/mod.rs,a_module_reads_a_platform_service_through_the_services_it_is_givenanda_geo_module_reads_a_platform_service_through_the_services_it_is_given,drive a module that reads a value from the configuration store it is handed,
through the production path, and assert the value reaches the output. They
exist because once everything compiled and passed, nothing anywhere read the
services parameter, and a parameter no code exercises is not a working seam.
Two supporting changes
The bid renderer is no longer an enum with a single APS variant. It is a type
tag plus the payload the demand implementation supplies, serialized flat so the
response a page receives is byte for byte what it was, and
ApsRendererV1moves into APS's crate. DataDome's cache and origin-path marker becomes
a neutral
PersonalizedResponserequest extension that any integration mayset, so core acts on the marker without knowing which integration asked for it.
main's split between the reader's origin requirement, decided before theresponse is stripped, and the request's, decided after, is carried into that
neutral marker rather than keyed on one vendor's tag.
One design decision made while stacking this on #1045
Two designs for
[geo]met when this branch moved onto the stack, and theydisagreed about what an unset selector means. The permission model (#1045)
treats unset as resolving nothing and making no host geo call, and this branch
had treated unset as leaving the host's own lookup in place. The permission
model's meaning is kept, because a deployment that has not asked for a host geo
lookup should not be making one. Unset and
"none"therefore both resolve toDisabledGeo,"platform"is the explicit opt-in to the adapter's own lookup,and any other value names a module that declared a geo module.
Where to start
crates/trusted-server-adapter-axum/tests/seam_probe.rs.Every seam is driven end to end from a crate core does not know about,
through the Axum adapter's real router. The tests turn on observable
outcomes, being the bytes served, the JSON a route returns and the error a
startup or deploy check produces, rather than on a function having been
called. Start here, because a seam is only proven by an implementation that
is not the built-in one.
ec_module_generates_an_identifier_with_the_modules_prefixshows themodule's own Edge Cookie module ran, because its identifier starts
seam-probe-, which the built-in HMAC module cannot produce, andec_selector_naming_a_module_starts_the_adapterstarts the Axum router withthat module selected.
crates/testing/seam-probe/src/lib.rs,the fixture those tests drive. It is a workspace member and a dev-dependency
of the Axum and Fastly adapters only, so no production build reaches it. It names its
module from its folder,
testing.seam-probe, and one registration carriesa browser module, a proxy route, a request preparer, geo, Edge Cookie and
device modules and its own settings table. Its builder also registers a
demand source and an ad server, which
[demand]and[ad-server]select.crates/trusted-server-core/src/provider_table.rs,the table
[demand]and[ad-server]share, which takesmodulesormodulefrom its selection type, andauction/plan.rs,which resolves
[demand]and[ad-server]names to registeredimplementations and reports an unknown one by listing the ones this build
has.
crates/trusted-server-core/src/integrations/registry.rs,with_plan_and_registrations. Building the registry refuses a duplicate id,naming both sources, runs a builder when a section selects its module and
refuses a selection no module answers to, listing what the section could
select. It checks the SHA-256 of a carried browser module, and resolves
[ec] module,[geo] moduleand[device] moduleagainst the namesthe running registrations declared their modules under.
crates/trusted-server-core/src/tsjs_bundle.rs,new. Composition of the served script moves out of
trusted-server-jsandinto core, keyed on content rather than on ids, because a module a vendor
crate carries is not in the compile-time map. The byte rule is unchanged,
being core first then each part joined by
;\n. The tests in that filecompare the composed bytes and hash against
trusted_server_js::concatenate_modulesandtrusted_server_js::concatenated_hash, which is the evidence thatcomposing in core moves no
?v=value.crates/trusted-server-core/src/config.rs.The hand-written list of vendor configuration types is replaced by a loop
over the builders, and the auction plan is compiled and checked on the same
path, so a bidder route naming a demand source
[demand] modulesdoes notselect is caught before a push.
crates/trusted-server-core/src/auction/types.rs,BidRenderer. This is the riskiest single change, because the wire shape{"type":"aps", ...}has to survive byte for byte.aps_renderer_serializes_to_versioned_camel_case_contractpins theserialized form and
renderer_bid_id_key_matches_the_serialized_formpinsthe one field the publisher reads by key, both in APS's crate,
crates/auction/aps/src/lib.rs.crates/trusted-server-core/build.rscarries two jobs. The migration guard used to list core source files by hand
with
include_str!, so a vendor moving out of core would break the buildrather than a test. The list is now generated from the source tree, so a file
that leaves core leaves the guard with nobody editing anything. The second job
is
main's, writing the manifest's default config store id into the build.The integration modules move out of
trusted-server-coreThe sixteen integration modules that sat in
crates/trusted-server-core/src/integrations/now live in fifteen crates of their own undercrates/<type>/<vendor>/, which is the layout #784 asks for. Each is selected and configured exactly as before. Each carries its own Rust, settings type, deploy rules and tests, and names its maintainers in its manifest (#1100). Core names none of them.The move is 31 commits, one per module, per extension point, per removal or per fault found, so each can be read on its own.
What an operator sees
Nothing changes in a configuration.
[cmp] module = "didomi",[auction] modules = ["prebid"], a[demand.<name>]table withimplementation = "auction.prebid-server"and every module's own table read as they did, and the routes, the browser module names and the wire names (aps,adserver_mock,gpt) are the same.The modules
cmp.osanocrates/cmp/osanoidentity.lockrcrates/identity/lockraudience.permutivecrates/audience/permutivecmp.sourcepointcrates/cmp/sourcepointcmp.didomicrates/cmp/didomitag.google-tag-managercrates/tag/google-tag-managertesting.testlightcrates/testing/testlightframework.nextjscrates/framework/nextjsad-tag.googleandad-tag.google.diagnosticscrates/ad-tag/googlebot-protection.datadomecrates/bot-protection/datadomead-server.mockcrates/ad-server/mockauction.apscrates/auction/apsauction.prebidcrates/auction/prebidauction.prebid-servercrates/auction/prebid-serverauction-protocol.openrtbcrates/auction-protocol/openrtbA module is named by its crate folder, and a test in each crate holds its
MODULEconstant to that folder.crates/trusted-server-moduleslists the modules a stock build ships, in the order their hooks run. Every adapter builds its state and loads its settings with that list followed by the builders a deployment added, and thetstool registers it for deploy validation. Offering a module does not run it, because the registry builds a module only when a section of the settings selects it.What stays in core's
integrationsfolder is the registry, the builder a module registers with and the JavaScript asset proxy, which is core's own. Core's built-in list offers no auction implementation.Maintainers
Every crate outside core names who maintains it in its manifest, the way Prebid.js requires a named maintainer of every adapter.
vendor ownedpermission-signal/mtm(51Degrees)seeking vendor ownerproject ownedA test in
crates/trusted-server-modulesreads all 24 manifests and fails for one with no declaration, no owner or an unknown status. The integration guide states the convention.What core gained to let them go
Three extension points, each vendor neutral, each its own commit with its own tests.
Core also offers its auction test helpers under its
test-utilsfeature, so an implementation's crate tests itself against core's request builder, response reader and orchestrator. About fifteen small items of core's auction code are public for an implementation outside core (bid admission, the notification policy, a slot's stored request intent, the plan's bidder routes), and the reader of an ordinary OpenRTB response sits in core's own OpenRTB driver for any implementation to use.Two places where core read a vendor's own data are closed by the moves. The
page's bid map read the APS renderer's bid id by APS's names, and a renderer
descriptor now carries the name of the field that holds its bid id, stated by
the implementation that builds it, which the bid map asks for. DataDome's
marker for leaving its browser tag out of a page was a named type in the page
path, and it is DataDome's own request state now.
Tests
A module's tests went with it. Where core held a test of a vendor's own behavior, the test moved to the vendor's crate and runs through core's public interface. Where core used a module as its example of a kind of module, the test stayed and runs on a stand-in compiled for core's tests only. There are seven, being four page modules (one that streams, one that tags a page, one that acts on a single request, one with a deferred browser module), two demand implementations (a plain one, and one that serves stored requests) and an ad server.
Core's shared test settings selected Prebid, so every test of core and of every other module ran with Prebid in it. They select no auction module now.
integrations/prebid.rswas 9,994 lines holding the page integration, the Prebid Server request and response code and a provider kept for the test build. Its shipped code falls into two parts with nothing shared, so 27648c0 divides the file inside core without changing behavior, and the two halves then move in 39f1240 and dd7f232. The test-only provider and its combined configuration are kept and divided the same way. The Prebid Server, APS and plain OpenRTB request goldens, two each, are carried byte for byte.One test is not carried.
register_rejects_invalid_bid_param_override_rulechecked that the test-only page registration refused a bad override rule, and that registration no longer holds override rules (the provider's own test of the refusal stays). The ten tests of the unused ad server model inmodels.rswent with it. Compared by name, every other test core had before the move is in core or in a module's crate after it, 60 of them under a name that says what the test covers now, such asa_bot_protection_block_is_refused_by_the_status_checkfora_datadome_block_is_refused_by_the_status_check.Size
Every
.rsfile undercrates/trusted-server-core/src, tests included, because a module's tests leave core with the module:mainAgainst
main, core is 9.5 percent smaller with comments and 12.8 percent smaller without, and that is with the module seams and the permission model thatmaindoes not have. A comment line is a line that holds only a comment, and blank lines are counted in both columns. The sixteen modules are 41,581 lines in their crates, 38,735 without comments.What the moves found
tsaudit generator named moved modules through core. It asks the stock list where a module is selected now (e08ef73).Cookieheader of every request. A stock build still does that, and a build without the Google crate forwards cookies as they arrived.<head>for a tag to be written into, so it could not fail. It asserts something that can.module = "bot-protection.datadome") did not have its secrets looked up, because the lookup read the short name alone. The second extension point reads both.tstool's integration tests validate settings in a process of their own and registered none, so the first CI run of the move refused the fixture's Prebid Server source in the integration workflow and in the tool's browser tests. Each registers the stock list first, as the tool does (9ce9e3e). One test of the generator's own loads settings from the blob it built, as an adapter does at startup, and it passes the stock list to the loader (190ea05).Why this rather than what
maindoes todayCore names vendors it has no business knowing.
crates/trusted-server-core/src/auction/plan.rsonmainholds
MOCK_MEDIATOR_ID = "adserver_mock"and refuses any other ad server. Itspecial-cases the profile ids
prebid-serverandapsby name when itcompiles a plan, and
crates/trusted-server-core/src/auction/profile.rsonmainholds the three profiles a deployment may choose, being
standard,prebid-serverandaps, as a fixed list with a closed enum behind it. Avendor cannot add a demand source or an ad server without a pull request
against core, which is the cost this series exists to remove.
The same idea is spelled four different ways. An integration is switched on
by
enabled = trueinside its own block. A demand source is switched on by akey appearing in
[auction.providers]. A permission signal is switched on byits name appearing in a
sourceslist. An Edge Cookie module is switched onby a selector naming a nested
[ec.providers.<name>]block. Two of those fourare on
maintoday. The other two arrived inside this stack, in the EdgeCookie module seam (#1043) and the permission model (#1045), which shows how
quickly a fifth spelling appears when no rule says what a spelling looks like.
One convention now covers all of them, and the next type gets the same one
without anyone deciding again.
For the people who run a deployment, one convention plus refusal at startup
changes what a mistake costs:
quietly ignored. On
mainan[integrations.datadome]block that outlivesits integration keeps every setting it had, and nothing tells the next reader
whether those settings are live. A leftover
[auction.providers.pbs-old]entry is worse, because on
mainthe entry is what switches the demandsource on, so a stale one is a live bidder nobody meant to keep. Now a table
has to be named by its type's selector line or the deployment does not
start.
keys it does not know, so
timeout_milliswhere the setting istimeout_msstops the deployment rather than silently leaving the default in place,
which is the kind of fault that otherwise turns up much later in a latency
graph.
modules = ["pbs_main"]becomingmodules = ["pbs_main", "aps_main"]sayswhat happened. On
mainthe same intent is a new nested block, aprofilestring, a
profile_configobject and aprotocolkey, and a reviewer has toreconstruct the outcome from four places.
be HTTPS, or HTTP to a loopback host for a local test stack.
[auction.providers],[auction] mediator,[adserver],[integration]or a
providerkey is refused, and the error names the table or key thesetting moved to
instead of leaving the operator to search for it.
For the people who write modules, a module plugs in through one
registration and inherits every check above without writing any of them. Core
names nothing a crate outside it supplies, so shipping a demand source, an ad
server, an identity module, a geo module or a device module means publishing a
crate, not editing core and waiting for a core release. The seam probe shows
this from the outside, because it is a crate core does not know about. Through
the real router or the real startup path it reaches a proxy route, a carried
browser module, a request preparer, a geo module, an Edge Cookie module, a
demand source and an ad server. Its device module is resolved by
[device] module, and the Fastly adapter's own tests show a registered devicemodule classifying a request.
What this costs. These are breaking changes, and the auction keys get no
deprecation period. That is a deliberate choice for a major release, taken
because the alternative is carrying two spellings of every selector
indefinitely, and because a configuration refused with a message naming the fix
is cheaper to migrate than one that half works.
Moving from the previous layout
[integrations.<id>]withenabled = true, and[integration] module[cmp] module = "didomi",[ad-tag] modules = ["google"]and so on, with[<type>.<name>]only for settings[ec.providers.<name>][ec.<name>][permission_signal] sources[permission-signal] moduleshost-signals,client-fixed,gpp-sale-opt-out,gpp_sale_opt_out,us_privacyhost_signals,client_fixed,gpp,us-privacy[auction.providers.<id>]withprotocol,profileandprofile_config[demand] modulesand[demand.<name>], withimplementationand the settings flat in the tableprofile = "standard"implementation = "auction-protocol.openrtb"profile = "prebid-server"andprofile = "aps"implementation = "auction.prebid-server"andimplementation = "auction.aps"[auction] mediator = "adserver_mock"and[integrations.adserver_mock][ad-server] module = "mock"and[ad-server.mock][integrations.aps] rendering_moderendering_modein the[demand.<name>]table of theauction.apsimplementation[debug.auction_html_comment_options] include_mediator_responseinclude_adserver_responseThe word mediator goes with those keys. It is "ad server" in prose and
ad-serverin configuration, and the auction response metadata that readparallel_mediationnow readsparallel_adserver.A file in the old shape is not read on a best-effort basis.
[auction] providersand[auction] mediatorare both refused with a message namingwhere the setting moved to, and so are
[adserver],[integration]and aproviderkey.ts prebid server inspectfollows the file. It reads the[demand.<name>]tables whose
implementationline isauction.prebid-server, and the[auction.prebid]table. Each demand sourceit reports carries
selected, which is whether[demand] modulesnames it,and the report's own
selectedis whether[auction] modulesselectsprebid, in place of the removedenabled_explicit.How it was verified
The head this pull request shows now is
49bdf21b8, which merges the move ofthe integration modules, 31 commits ending at
f5a2858fa, intoa90a141bc,the head that carries
mainat 182fdf4. The tree of49bdf21b8is the tree off5a2858fa.On
f5a2858fa, on Windows,cargo fmt --all --check, the core suite (2,557tests and doctests), the Fastly alias under Viceroy (3,565 across its test
binaries, the Fastly adapter's and every module crate's included), the Axum
alias (897 tests, which runs every module crate's tests natively), the
Cloudflare (56 tests) and Spin (90 tests) adapter suites, the cross-adapter
parity suite (17 tests), the seam-probe fixture's own tests (11), the Viceroy
configuration generator's own tests (8) and the 29 tests of the integration
target that need no container, Clippy with warnings denied on all four
adapters, the permissions inspector's wasm build, and the docs Prettier
check, ESLint and VitePress build with the pinned Prettier all pass. The CLI
crate does not build on a Windows host, because a dependency uses an unstable
Windows feature, so the audit generator, the bundle command,
ts prebid server inspect, the environment overlay test and the tests of deploy validation werechanged by reading and are verified by CI alone, where the host-target Clippy
of the CLI and the CLI's own tests on Ubuntu and macOS pass. CI caught two
faults that way, a type error in the audit generator on the section change,
fixed in ad734fd, and the settings validated without the stock modules that
the section on the move records, fixed in 9ce9e3e and 190ea05. Four
browser script files changed with the move, being two comments and a test
fixture path that named a Rust file or the Google Publisher Tags bootstrap
script by its place in core. CI runs the browser script's lint, format check
and tests, and they were not run again locally.
On CI, every workflow passes on
49bdf21b8: Run Tests, Run Format, Integration Tests, Permissions Inspector and CodeQL Advanced. Run Tests passed on its second attempt. The first failed in one test of thetstool,collects_gpt_slot_from_local_fixture, on a browser launch timeout, which #1256 describes. GitHub's codescanning check is the one check that shows as failing, because CodeQL carries one open
alert on this branch and on no other in the stack,
alert 192
(
rust/cleartext-logging, high, raised on 15 September), on the orchestrator'slog line that names the ad server, its logical budget and its transport
timeout. The move of the modules added no alert. The query reaches that line
by tracing a test helper's secret store into the runtime services the
transport timeout is canonicalized through, and the line logs an identifier
and three millisecond values, so the alert stands as reported, with the
reason in a comment at the line, rather than being worked around in the code.
The same log line is on
main, with the word mediator where this branch saysad server. Dismissing the alert needs security write access on this
repository.
The CI workflow runs the seam-probe fixture crate's own tests, but no clippy
alias or workflow step names that crate, so its source is not linted in CI.
What is not in this change
The design in #1084 is wider than this pull request. Every part of it this pull
request does not deliver is listed here.
Core's auction engine still builds and reads OpenRTB itself. Only the
plain demand implementation moved, so OpenRTB as a translation at the edge of
the auction is a later change.
Vendor logic in core's own files is mostly untouched. What sat in an
integration module has moved. What sits in core's own files has not, for
example Prebid's identifier cookie handling, the Prebid slot parameters in the
creative opportunities settings and Google Ad Manager slot paths. Four small
pieces of it are removed, being the
Prebiderror variant (Prebid's cratesraise
Integration, which answers the same), the unusedmodels.rsandGamerror variant, Prebid's typed request extensions, and two Prebid names in the
routing code.
Each vendor's browser script is still in
crates/trusted-server-js, andcore TypeScript still imports APS directly (section 8, item 8).
auction.tsand
types.tsimport and name the APS renderer, so moving APS's Rust did not move its
browser code.
The consent decoders are still in core. The TCF, GPP and US Privacy
decoders sit in
crates/trusted-server-core/src/consent, and the permissionsignal crates read the consent context core decoded.
The stock list is written by hand (section 3.1). The design discovers the
built-in integrations at build time from their directories. Core's own list
holds the JavaScript asset proxy alone, and the modules a stock build ships
are listed in
crates/trusted-server-modules,written in hook order, because the order of that list is the order the page
hooks run in, and a generated list needs that order stated another way
first.
Page changes are still the four hook traits (section 3.7). The middleware
contract and the
[[fetch]]and[[serve]]entries are designed in #1084and are not in this pull request. They follow in a pull request of their
own.
A crate that supplies only an Edge Cookie, geo or device module does not
run from its selector alone (section 3.6).
[ec] module,[geo] moduleand
[device] moduleselect a module by the name its registration declaredit under, and the module that supplies it has to be selected in a section as
well, because the registry builds only what a section selects.
[ec],[geo]and[device]are read by core and are not sections that run abuilder, so a crate that supplies only one of these modules cannot yet be run
from its selector alone.
[geo]and[device]also hold no settings tablefor a module. The first vendor identity, geo or device crate needs the
first, and a geo or device crate with settings of its own needs the second.
Nothing is refused for a missing host signal (section 3.6). A registration
declares no host-signal requirements, so a module that needs a signal the
running adapter does not expose is not turned back at startup.
No test checks the order of several hooks on one registration (section 6,
item 1). The acceptance round trips run through the Axum adapter and, under
Viceroy, through the Fastly adapter, which is where items 1 and 2 ask for
them. Item 1 also asks that a module's hooks run in the right order, and the
tests show the preparer runs before routing and exactly once, but no test
checks the order of several hooks on one registration.
A deployment's own module is not known to the stock CLI (section 8, items 1,
6 and 15).
ts config validateandts config pushvalidate against themodules a stock build ships, so each of those modules' own rules runs on the
path an operator uses. A module from a crate a deployment added is not on that
list. Its validate function runs at startup for a deployment that loads its
settings with its builders, and wherever the deployment's own code calls
validate_settings_for_deploy_with, which is why the seam probe still repeatsits check inside its build function.
docs/guide/integration-guide.mdrecords the gap, and the operator path needs a decision, which is whether the
CLI is built per deployment with its vendor crates.
No auction runs through the probe's demand source or ad server (section 8,
item 10). Both reach the plan, the settings load and deploy validation from
outside core, and neither is driven through a request. What an implementation
outside core does during an auction is held by the tests in the
implementations' own crates, which drive APS and Prebid Server through core's
request driver, response reader and orchestrator, and the plain OpenRTB demand
through the request driver.
No test loads settings from a config store with registered builders on
Fastly (section 8, item 7). The Fastly adapter takes a module crate through
run_with, so no adapter is edited to compose one. The load with adeployment's builders is tested in core, and the Fastly adapter passes the
registered builders to it in one call that no test drives. The test that
registers through the process-wide list also leaves the probe on offer for
the tests that run after it in the same binary, where it runs only if the
settings select its module, which none of them do.
A module can still be resolved more than once per request (section 8, item
3). On
POST /auctionlocation is looked up to build the Edge Cookie contextand again in
handle_auction, and the seam probe's proxy route calls its geomodule as well. This pull request adds no per-request module context, so a
module sharing one backend across identity, location and device has nowhere to
hang a single call per request.
Request preparers still cover different routes on each host (section 8, item
5). Replacing the vendor-named call with the registry call did not make the
covered routes the same. Axum runs preparers once before routing, Cloudflare in
its shared handler wrapper and its fallback dispatch, Fastly after its early
admin dispatch and in its fallback, and Spin only in the auction handler, the
page-bids handler and the fallback. A module that strips its own reserved query
or cookie is therefore protected on a different set of routes depending on the
host.
The
ts auditdetection patterns stay in the CLI. Thets auditcommand keeps its own vendor detection patterns outside the registry, and how
the CLI learns a vendor's detection pattern from a crate is still open. What
the audit writes for a module it detects, being the section that selects the
module and its name there, comes from the stock list.
What the round trip does not exercise. The seam probe declares a proxy, a
carried browser module, a request preparer, and geo, Edge Cookie and device
modules. It declares no head injector, attribute rewriter, script rewriter,
HTML post processor or request filter, and it uses neither the deferred nor
the standalone script delivery flag, so the round trip does not exercise those
from outside core. The probe's device module classifies a request in the
Fastly adapter's tests, through the function the entry point calls for every
request, and no request sent to a router is classified by it.