Template cache keys and config pushes vary per load when allowed_context_keys has 2+ keys #1197

Description

[auction] allowed_context_keys is a HashSet<String>. Serde writes a HashSet as a JSON array in the set's iteration order. Rust's default RandomState gives every newly built set its own hash keys, so the same configuration can serialize in a different order each time it is parsed. With n keys there are up to n! orders. Two hashes are computed over that array:

1. template_fingerprint in publisher.rs, which is part of every shared-template cache key. The Fastly adapter parses Settings in a fresh Wasm instance for every request. With [creative_opportunities] assembly_mode = "esi" and two or more context keys, the fingerprint, and therefore the template cache key, changes from request to request. Each iteration order gets its own template, so the cache fills with duplicates and hits drop. With six keys I measured no hits at all.
2. The EdgeZero envelope sha256 that ts config push and ts config diff compare. Each CLI run parses the TOML again, so an unchanged TOML can produce a different sha. Whenever the order differs from the stored one, diff reports a change and push rewrites the store.

Who is affected:

• Fastly deployments that opt into ESI assembly (assembly_mode = "esi", default inline) and list two or more context keys.
• Operators running ts config push or ts config diff with two or more context keys, on any adapter. Both commands share the EdgeZero CLI code that compares the sha.

Since when:

• The field has been a HashSet since send permutive segments to adserver #263 (7f342ea).
• The template fingerprint was added by Add shared root caching with hybrid ESI and streaming assembly #1013 (squash commit d97bda6, 2026-08-22).
• Typed ts config push and ts config diff were added by Add ts cli #799 (6800560).

I verified the behavior only at a4e01eb.

Steps to reproduce

1. Failing tests

Add this test inside mod template_fingerprint_tests in crates/trusted-server-core/src/publisher.rs:

#[test]
fn a_context_key_allowlist_fingerprints_identically_across_parses() {
    // Fastly parses `Settings` once per request, so every request builds its
    // own `allowed_context_keys` set with fresh hasher keys.
    let toml = format!(
        "{}\n[auction]\nallowed_context_keys = [\"alpha\", \"bravo\", \"charlie\", \
         \"delta\", \"echo\", \"foxtrot\"]\n",
        crate::test_support::tests::crate_test_settings_str()
    );
    let fingerprints = (0..32)
        .map(|_| {
            let settings = Settings::from_toml(&toml).expect("should parse settings");
            template_fingerprint(&settings)
        })
        .collect::<std::collections::BTreeSet<_>>();

    assert_eq!(
        fingerprints.len(),
        1,
        "should fingerprint one configuration identically, got {} distinct values",
        fingerprints.len()
    );
}

Add this test inside mod template_cache_end_to_end_tests in the same file, next to one_integration_configuration_keys_one_template:

#[tokio::test]
async fn one_context_key_allowlist_keys_one_template() {
    let stub = Arc::new(StubHttpClient::new());
    let cache = Arc::new(MemoryTemplateCache::default());
    let services = services(Arc::clone(&stub), Arc::clone(&cache));
    for _ in 0..8 {
        queue_shareable_html(&stub);
    }

    for _ in 0..8 {
        // One `Settings` per request, as on Fastly.
        let mut settings = settings_with_mode("esi");
        settings.auction.allowed_context_keys =
            ["alpha", "bravo", "charlie", "delta", "echo", "foxtrot"]
                .into_iter()
                .map(str::to_string)
                .collect();
        let _ = run(&Arc::new(settings), &services, navigation_request()).await;
    }

    let distinct_keys = looked_up_cache_keys(&cache)
        .into_iter()
        .collect::<std::collections::BTreeSet<_>>();
    assert_eq!(
        distinct_keys.len(),
        1,
        "should name one key for one configuration, got {} distinct keys",
        distinct_keys.len()
    );
    assert_eq!(
        stub.recorded_request_uris().len(),
        1,
        "should serve every later request from the first template"
    );
}

Add this test at the end of mod tests in crates/trusted-server-core/src/config.rs:

#[test]
fn pushed_envelope_sha_is_stable_for_one_context_key_allowlist() {
    // `ts config push` and `ts config diff` compare this sha to decide whether
    // anything changed.
    let toml = format!(
        "{}\n[auction]\nallowed_context_keys = [\"alpha\", \"bravo\", \"charlie\", \
         \"delta\", \"echo\", \"foxtrot\"]\n",
        crate_test_settings_str()
    );
    let shas = (0..16)
        .map(|_| {
            let app_config: TrustedServerAppConfig =
                toml::from_str(&toml).expect("should deserialize app config wrapper");
            let data =
                serde_json::to_value(&app_config).expect("should serialize app config");
            edgezero_core::blob_envelope::BlobEnvelope::new(data, String::new()).sha256
        })
        .collect::<HashSet<_>>();

    assert_eq!(
        shas.len(),
        1,
        "should build one envelope sha for an unchanged TOML, got {} distinct values",
        shas.len()
    );
}

Run them one at a time. On wasm32-wasip1 the test binary aborts at the first failing test.

cargo test -p trusted-server-core --target wasm32-wasip1 --lib -- \
  publisher::tests::template_fingerprint_tests::a_context_key_allowlist_fingerprints_identically_across_parses --exact
cargo test -p trusted-server-core --target wasm32-wasip1 --lib -- \
  publisher::tests::template_cache_end_to_end_tests::one_context_key_allowlist_keys_one_template --exact
cargo test -p trusted-server-core --target wasm32-wasip1 --lib -- \
  config::tests::pushed_envelope_sha_is_stable_for_one_context_key_allowlist --exact

Observed at a4e01eb:

assertion `left == right` failed: should fingerprint one configuration identically, got 32 distinct values
  left: 32
 right: 1

assertion `left == right` failed: should name one key for one configuration, got 8 distinct keys
  left: 8
 right: 1

assertion `left == right` failed: should build one envelope sha for an unchanged TOML, got 16 distinct values
  left: 16
 right: 1

2. ts config diff reports changes for an unchanged TOML

From the repository root, after cargo build -p trusted-server-cli --target aarch64-apple-darwin (use your host triple). The manifests are copied to a temporary directory so the tracked fastly.toml is not edited.

TS="$PWD/target/aarch64-apple-darwin/debug/ts"
WORK=$(mktemp -d)
cp edgezero.toml fastly.toml "$WORK"/ && ln -s "$PWD/crates" "$WORK/crates"
sed 's/^allowed_context_keys = \[\]$/allowed_context_keys = ["alpha", "bravo", "charlie", "delta", "echo", "foxtrot"]/' \
  trusted-server.example.toml > "$WORK/app.toml"
set -a && source .env.dev && set +a
cd "$WORK"
"$TS" config push --adapter fastly --local --manifest edgezero.toml --app-config app.toml --yes --no-diff >/dev/null
for i in 1 2 3; do
  "$TS" config diff --adapter fastly --local --manifest edgezero.toml --app-config app.toml --exit-code --format json
  echo "exit=$?"
done

Observed (first run shown in full, the other two abbreviated):

{
  "local_sha256": "468d4a5532bfaa744990dd8a463260d70ffc334b8348abc669434b3519cf7705",
  "remote_sha256": "55415ea930859c66f8eab33bfb5d8e63cc4cabd1863dcad9d2fc240633ad6da7",
  "added": {},
  "removed": {},
  "changed": {
    "auction.allowed_context_keys": {
      "from": ["delta", "charlie", "echo", "foxtrot", "alpha", "bravo"],
      "to": ["delta", "foxtrot", "echo", "alpha", "bravo", "charlie"]
    }
  }
}
exit=1
"local_sha256": "1ffaa35010e6d88d3a325229ab6b90826d1510732c87c39820e26d47f6444132", ... exit=1
"local_sha256": "8dbfd35a4ec048ab635ccd390087ab34713558ab6eda2aede38d34fe8b52b52c", ... exit=1

With allowed_context_keys = [] the same steps report # no changes (sha256 matches: ...) and exit 0 (10 of 10 runs in my check).

3. Measured effect (supporting data)

Template cache, end to end under Viceroy, which runs each request in a fresh instance like Fastly. I used an adaptation of scripts/template-cache-local-test.sh in esi mode: the same stub origin, generated config and isolated config-store seeding, with allowed_context_keys as the only config change. It sends identical navigations to /article and reads x-ts-template-cache.

Distinct values, from a scratch crate that depends on this repository's trusted-server-core. For the fingerprint, each wasmtime instance loads one fixed envelope through settings_from_config_blob (the Fastly loader) and computes a verbatim copy of template_fingerprint. For the envelope, each process serializes the TOML through TrustedServerAppConfig, as the CLI does.

With the real CLI and an unchanged TOML, ts config diff --exit-code reported a change in 10 of 10 runs with 6 keys, 9 of 10 with 3 keys and 8 of 10 with 2 keys. Repeated ts config push --yes rewrote the store in 5 of 5 runs with 6 keys.

Expected behavior

One configuration produces one template fingerprint, one template cache key per URL variant, and one envelope sha, whatever the process or instance. A second eligible request for the same URL is a template cache hit. ts config diff --exit-code on an unchanged TOML exits 0, and a repeated ts config push reports no changes.

Actual behavior

With n context keys, each value takes up to n! forms (one per iteration order). Every order needs its own cold template fill, ts config diff reports a change to auction.allowed_context_keys on most runs, and ts config push --yes rewrites the config store.

Root cause

The field is a hash set (crates/trusted-server-core/src/auction_config_types.rs:78-79):

#[serde(default = "default_allowed_context_keys")]
pub allowed_context_keys: HashSet<String>,

template_fingerprint hashes the serialized settings (crates/trusted-server-core/src/publisher.rs:2080-2084). Its comment covers maps only. serde_json::Value sorts object keys when preserve_order is off, but it keeps array order as written:

// `serde_json::Value` uses a sorted object map without `preserve_order`, making
// independently deserialized HashMaps canonical before they are serialized again.
let canonical = serde_json::to_value(settings)
    .and_then(|value| serde_json::to_vec(&value))
    .expect("serializing typed settings should be infallible");

The fingerprint goes into the key at publisher.rs:4616 (template_fingerprint: template_fingerprint(settings),) and TemplateCacheKey::to_cache_key hashes it (platform/template_cache.rs:107).

On Fastly every request builds new settings. main handles one request (crates/trusted-server-adapter-fastly/src/main.rs:78-79, let req = FastlyRequest::from_client();), and AppState is documented as "effectively per-request" (app.rs:176-177).

The envelope sha uses EdgeZero's canonical form (edgezero-core v0.0.8, canonical_form.rs:55-70), which sorts object keys and writes arrays in order. The CLI compares that sha to decide whether anything changed (edgezero-cli v0.0.8, config.rs:685 for diff and config.rs:1035 for push: if remote_envelope.sha256 == local_sha {).

The existing stability tests miss it because their fixtures have no context keys: the_fingerprint_is_stable_across_calls_for_one_configuration (publisher.rs:8883) and one_integration_configuration_keys_one_template (publisher.rs:11696).

Other hash-typed fields reachable from Settings

I checked every HashSet and HashMap field in the types Settings serializes (settings.rs, creative_opportunities.rs, auction_config_types.rs, auction/plan.rs, consent_config.rs). Only allowed_context_keys serializes as an array.

Notes on the table:

• Objects are stable in the fingerprint because the Fastly adapter graph does not enable serde_json/preserve_order. I confirmed this with cargo tree -p trusted-server-adapter-fastly --target wasm32-wasip1 -e features -i serde_json.
• The CLI graph does enable preserve_order (through handlebars v6.4.2 from edgezero-cli), so object key order in the stored envelope bytes can vary. The sha sorts object keys, so push and diff are unaffected. The control run above, with the example config and its several [integrations.*] tables, reported no changes in 10 of 10 runs.
• Integration configs reach Settings only as raw JSON, so typed integration structs do not affect either hash. The only integration config with hash maps, LegacyPrebidServerConfig (integrations/prebid.rs:336-338), is #[cfg(test)]. DataDome's ProtectionScope hash sets (integrations/datadome/protection_scope.rs:146, 155) belong to a compiled runtime struct that is never serialized.
• vec_from_seq_or_map (settings.rs:3610-3630), which accepts env-overlay index maps for Vec fields, sorts by index and is deterministic.

Impact

Default deployments are not affected by the cache problem:

• assembly_mode defaults to inline (creative_opportunities.rs:209-220). The example config calls esi an "opt-in Fastly Core Cache experiment" and ships allowed_context_keys = [] (trusted-server.example.toml:284).
• Only the Fastly adapter has a template cache. The other adapters use UnavailableTemplateCache (platform/template_cache.rs:736-745).

When ESI is enabled with two or more keys, the damage grows with the number of keys:

• Each URL variant needs up to n! cold fills instead of one: 2 for 2 keys, 6 for 3, 24 for 4, 120 for 5 and 720 for 6. With 2 or 3 keys the cost is a few extra misses per URL per template lifetime. From about 4 keys up it dominates.
• Templates live at most 60 seconds by default (creative_opportunities.rs:19). A URL whose eligible requests within one lifetime are well below n! almost never hits.
• Every miss fetches the origin, runs the full transform, writes a Core Cache entry, and buffers the response instead of streaming it (publisher.rs:2406-2413). Inline mode streams, so each miss is slower than the same request in inline mode. With 6 keys every request in my run paid that cost.
• Only requests already eligible for a shared template are affected: GET, no disqualifying Authorization or cookies (publisher.rs:4557-4563), and an origin response that grants shared freshness (publisher.rs:6253-6335).

For the CLI:

• ts config diff --exit-code reports a change on most runs of an unchanged TOML, so a CI gate built on it is unreliable for such a config.
• ts config push --yes writes the store again whenever the order differs (5 of 5 runs with 6 keys).
• For envelopes over 8,000 characters, each rewrite creates a new generation of content-addressed chunk entries (edgezero-adapter-fastly v0.0.8, chunked_config.rs:229, 251). The old chunks stay until ts config gc runs.

Issue #1192, implemented by PR #1193, caches template_fingerprint once per warm instance. PR #1179 adds opt-in reusable sandboxes. Both changes make the value stable within one instance, but each instance still builds its own set. They reduce the number of distinct keys but do not remove the bug.

Proposed fix

Make the set ordered. Change allowed_context_keys to BTreeSet<String>, as NotificationConfig::suppress_seats already is (auction/plan.rs:232). Serde then writes it sorted. The change touches:

• auction_config_types.rs:5, 79, 93 and 156-158 (the import, the field, Default and the default function).
• The only runtime consumer, settings.auction.allowed_context_keys.contains(key) (auction/formats.rs:236), which works unchanged.
• Three test constructors: settings.rs:6244, auction/formats.rs:1060 and auction/orchestrator.rs:4704. After the change, use std::collections::HashSet; in the settings test module (settings.rs:3685) becomes unused.

I applied exactly this in a scratch copy of a4e01eb. The three tests above passed, along with the existing context-key tests and all of template_fingerprint_tests: 12 passed, 0 failed.

Also correct the comment at publisher.rs:2080-2081 so it no longer implies that all collections are canonical.

To catch the next set-typed field, add a determinism test. It should parse a config that gives every map- and set-typed field two or more entries, several times, and assert identical serde_json::to_vec(&serde_json::to_value(&settings)) bytes and an identical envelope sha.

Compatibility:

• The wire format stays a JSON array of strings, and older binaries accept a sorted array.
• After the upgrade, the first ts config push may write once more (to store the sorted order). Later pushes report no changes.
• The first deploy changes each template fingerprint once, which costs one cold fill per URL variant, the same as any bundle change today.
• No config migration is needed.

Alternative considered: sort arrays inside template_fingerprint or in the canonical form. I rejected this. Other arrays are ordered (handlers, slots, cookie lists), and it would not fix the envelope sha that push and diff compare.

Done when

• auction.allowed_context_keys serializes in a deterministic order (BTreeSet<String> or a sorted Vec).
• a_context_key_allowlist_fingerprints_identically_across_parses, one_context_key_allowlist_keys_one_template and pushed_envelope_sha_is_stable_for_one_context_key_allowlist are added and pass.
• A determinism test covers every set- and map-typed field reachable from Settings, each with two or more entries.
• Repeated ts config diff --exit-code on an unchanged TOML with six context keys exits 0.
• The comment at publisher.rs:2080-2081 is corrected.

Affected area

Fastly runtime

Version

main at a4e01eb

Related

• send permutive segments to adserver #263 (merged): added allowed_context_keys as a HashSet.
• Add shared root caching with hybrid ESI and streaming assembly #1013 (merged, d97bda6): added template_fingerprint and the shared template cache.
• Add ts cli #799 (merged): added typed ts config push and ts config diff, which compare the envelope sha.
• Add per-cookie template cache policy #1150 (merged): per-cookie template cache policy. Its cookie lists are Vecs and are deterministic.
• Reduce Fastly adapter per-request fixed cost: defer KV/geo work, cache settings/registry/template fingerprint #1192 (open): caches the template fingerprint per warm instance. Stable within an instance, not across instances. Does not fix this.
• Reduce time-to-first-byte across the Fastly adapter #1193 (open PR, closes Reduce Fastly adapter per-request fixed cost: defer KV/geo work, cache settings/registry/template fingerprint #1192): implements that cache (AppCache). Same limitation.
• Adopt EdgeZero reusable-sandbox lifecycle for Fastly #1179 (open PR): opt-in reusable Fastly sandboxes. Same limitation.
• Define integrations crate split and ordered configuration #1194 (open PR): its design spec proposes a canonical composition digest to replace this fingerprint. This fix is independent and can land first.
• Per-request fixed cost: release profile, settings pipeline trim, build-time tsjs hashes #855 (open): per-request fixed cost, including runtime hashing. Related cost, not this bug.
• Companion issue Template cache key ignores gpt_bootstrap.js and other Rust-inlined head scripts #1198: the same fingerprint misses the binary's inline head scripts.