Skip to content

Add identity provider core with one KV write per page load - #1436

Closed
dhruv8sh wants to merge 33 commits into
spec/1246-server-side-identity-providersfrom
feat/1246-identity-single-write
Closed

dhruv8sh wants to merge 33 commits into
spec/1246-server-side-identity-providersfrom
feat/1246-identity-single-write

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Implements Parts 0 and 1 of the plan for the server-side identity providers spec (Add design spec for server-side identity providers #1428). Part 2 (the first vendor provider and its rollout controls) is not in this PR.
  • Part 0 keeps TS identity cookies away from parties that should not see them. The publisher origin keeps ts-ec but no longer receives ts-eids or the new identity cookies, Prebid Server receives no TS identity cookie in any mode, and the sealed cookies can't be template-cache keys.
  • Part 1 holds every page load to one KV write. Only a real browser's document navigation writes, once, after the response is sent. /auction, /_ts/page-bids, /identify and proxy calls no longer write. They carry their changes forward in a sealed __Host-ts-eids-s cookie and per-batch __Host-ts-eids-r-* removal cookies. This keeps a busy page well under Fastly's per-key write limit, and pull sync now runs as the first provider on the new IdentityProvider trait.

Changes

File Change
crates/trusted-server-core/src/constants.rs, cookies.rs Identity cookie name and prefix lists; strip helpers that remove cookies by name or prefix
crates/trusted-server-core/src/auction/routing.rs, integrations/prebid.rs Strip identity cookies from Prebid Server requests in every transport mode; drop a non-UTF-8 Cookie header instead of forwarding it raw
crates/trusted-server-core/src/publisher.rs Strip ID cookies from origin requests; pass settings to auction EID targeting
crates/trusted-server-core/src/cookies/template_cache_policy.rs Refuse the sealed identity cookies as template-cache keys
crates/trusted-server-core/src/ec/identity/ (new) Provider trait, descriptors, catalog, consent policy and fingerprint, record revisions, servability check, sealed AES-256-GCM cookies, staged mutation, coordinator, dispatch, generic pull provider, request and finalize wiring, after-send flush, invariant tests
crates/trusted-server-core/src/ec/kv_types.rs, kv.rs, kv_backend.rs Record revision, provider state and opt-out fields; single write_identity path; write-counting backend for tests; remove the old sync and bulk-upsert writes
crates/trusted-server-core/src/ec/finalize.rs, ec/mod.rs Writer decision, browser EIDs seeded into new-visitor creation, removal carry on rotation, pull marks after send, cookie size measurement logs
crates/trusted-server-core/src/ec/eids.rs, identify.rs, admin.rs Shared servability check for the bidstream, /identify and the admin view; /identify no longer writes; admin redacts provider state
crates/trusted-server-core/src/ec/pull_sync.rs, pull_sync_marker.rs Reduced to URL, eligibility and completeness helpers; dispatch moves to the generic pull provider
crates/trusted-server-core/src/auction/endpoints.rs Auction EIDs resolved from the server set; /auction stages a KV gap instead of writing
crates/trusted-server-core/src/http_util.rs, integrations/registry.rs is_prefetch_request moved here, plus is_document_navigation; writer decision in proxy dispatch
crates/trusted-server-core/src/settings.rs, config.rs, migration_guards.rs, trusted-server.example.toml [ec.identity] settings and validation; warning for the deprecated key
crates/trusted-server-adapter-fastly/src/app.rs, main.rs Writer decision and generation seeding; after-send identity flush with an outcome log
crates/trusted-server-adapter-{axum,cloudflare,spin}/src/app.rs Pass is_real_browser: false
Cargo.toml, crates/trusted-server-core/Cargo.toml, Cargo.lock Add aes-gcm = "0.10" (the spec's choice for the sealed cookie)
docs/guide/configuration.md, edge-cookies.md, api-reference.md, CHANGELOG.md Document the single write, the new cookies and the stripped cookies

Changes from the plan

  • The __Host-ts-eids-s source index covers every registered [[ec.partners]] source, not only provider-claimed ones. Otherwise a browser entry for a partner that no provider claims could not be staged.
  • The invariant tests run in core, driving the same steps the adapter runs. The Fastly adapter's KV store comes from the runtime, so a counting store can't be injected through the router.
  • A pull sync 404 now becomes "no ID, retry in 24 hours" (the spec's backoff) instead of a retry on every request.
  • Metrics are log lines in the existing measurement style, not per-request counters.
  • The existing test-utils feature and TcfConsent::has_vendor_consent are reused instead of new equivalents.
  • Hooks that Part 2 uses carry #[expect(dead_code)], so the build fails once Part 2 starts using them and the attribute can't be forgotten.

Closes

Part of #1246. It doesn't close the issue; #1416 does.

Spec

Spec: docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md (#1428)

Test plan

  • cargo test-fastly && cargo test-axum
  • cargo clippy-fastly && cargo clippy-axum
  • cargo fmt --all -- --check
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run
  • JS format: cd crates/trusted-server-js/lib && npm run format
  • Docs format: cd docs && npm run format
  • WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
  • Manual testing via fastly compute serve
  • Other:
    • the remaining clippy aliases, and cargo test-fastly-reuse, test-cloudflare and test-spin;
    • the parity test, the native build-digest test and its clippy, the wasm clock lint, the CLI tests, and the Markdown format check;
    • the docs-proposal node --test and test.sh suites. shellcheck was not run because it isn't installed locally, and this PR doesn't touch scripts/.

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Every Cookie field is now read and checked, and a field that is not valid UTF-8 is dropped instead of forwarded. Two transport tests that pinned raw forwarding of malformed cookie bytes now expect the field to be dropped.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The origin keeps ts-ec and the pull-complete marker. ts-eids, __Host-ts-eids-s and the __Host-ts-eids-r- cookies no longer reach it. An origin that read ts-eids must read x-ts-eids instead.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Documents which identity cookies reach the publisher origin and Prebid Server.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Adds aes-gcm, which the sealed identity cookies use.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
New fields are optional and omitted when absent, so legacy rows read and write unchanged and the schema version stays 1. KvPartnerId and KvEntry derive Default, and literals move to constructors. The admin lookup redacts provider state.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Reuses TcfConsent::has_vendor_consent and the effective TCF lookup the consent module already uses. Adds shared consent fixtures for identity tests.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Expired IDs, IDs from disabled or opted-out writers, and IDs whose writer or owner policy fails are no longer sent to bidders, the admin view or /identify. /identify uses the request's consent, never the row's stored snapshot. Stored atype and UID-level ext are emitted. The provider catalog is empty until generic pull is wired in.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Removals apply first and win equal-second ties, a provider opt-out removes every record that provider wrote, stateful results apply atomically, refresh anchors survive while their owner holds state, and an invalid row falls back to removals only.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Each removal-producing response gets its own sealed cookie. A batch is never coarsened: an oversized one is planned as an overflow write, or split across cookies when that write fails.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Validated at deploy and at startup. ec.pull_sync_concurrency is deprecated in favour of [ec.identity].max_concurrency and is still honoured when changed from its default.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Adds the merged eligibility view of KV, the identity cookies and removal batches, the eligibility steps, and the mapping from each call outcome to KV changes and backoff. Also adds the consent fingerprint bits a provider opt-out records.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Each call is checked for https, a declared host and no Cookie header, capped per visitor, sent with the provider's timeout, and decoded under max_response_bytes before the provider sees it. An undeclared input read fails the call and nothing is sent.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Each pull-enabled partner becomes a provider with an unowned source, its allowed domains as hosts, and the parity consent policy. Its legacy hourly rate limit maps to the per-minute cap the limiter already enforces. Adds the basic provider harness and a KV test store that counts writes by mode.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Only an organic real-browser top-level document navigation that is not a prefetch is the writer. EcContext now captures the user agent, DNT and the raw identity cookies when the request is read, because after-send code no longer has the request. Integration-proxy EC generation now requires a real browser explicitly, and is_prefetch_request moves to http_util.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
/identify uses a stored cluster size, or computes one without writing it. The writer navigation stores it in its single flush.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Returning-visitor finalize no longer writes KV. The writer navigation adds browser ingest, flush-checked staged entries and removal batches to its single mutation. /auction saves the KV gap of its body EIDs in __Host-ts-eids-s, and /_ts/page-bids no longer ingests. A new visitor's create now carries its browser EIDs, so it stays one write. Orphan rotation carries removals into the new row and never positive staged IDs. Consent loss and tombstone reads expire __Host-ts-eids-s.

The provider catalog now indexes every registered partner source, so a KV-gap entry can be staged for any of them. The old ingest-sync path in kv.rs is removed with its tests. The queued mutation is written by the after-send flush added later in this series.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
When a removal batch does not fit one cookie, or four removal cookies are already present, it is applied to KV with the withdrawal path's CAS retries, which retry only on a generation mismatch. If that write fails it is split across new cookies, never coarsened.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The server set is KV plus the __Host-ts-eids-s entries, staged ones included, with removed, expired and unservable records left out. A removal cookie hides its record before the removal reaches KV. Server values keep their stored atype and UID-level ext and gain the writing provider's EID-level provenance.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Phase 4 replaces the pull sync after-send hook. It reuses the phase 1 snapshot when no provider is marked, otherwise re-reads the row, dispatches the marked providers in an hourly rotation, and applies the whole mutation in one IfGenerationMatch write that is never retried. Pull partners now run only on a returning visitor's writer navigation, are marked pending in phase 3 after a read shows their source missing, and back off through ps.

A read that shows a pull partner's sources present records a cookie-only next-due time, so the unowned read trigger does not fire on every navigation. The pull sync dispatcher, its bulk upsert and their tests are removed; the adapter builds a KV graph after send only when there is something to flush.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Drives the create before routing, finalize before send and the after-send flush, as the adapter runs them, against a write-counting store: one write per page load, none on stagers or non-2xx/304 navigations, no retry on a CAS conflict, zero reads for a recent cookie with nothing due, and an /auction body-only source reaching KV on the next navigation even after a refused flush.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Test-only helpers are gated on cfg(test). The capture, hold-mode and breaker hooks the providers change uses carry a targeted dead_code expectation, which fails once that change starts using them.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
One measurement line per identity request reports the inbound Cookie header size, the number and total size of removal cookies, and whether the writer read KV and queued changes. The adapter logs each flush outcome. No ID or cookie value is logged.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh dhruv8sh self-assigned this Oct 22, 2026
@dhruv8sh

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #1428

@dhruv8sh dhruv8sh closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant