Audit and extend all xtask E2E qualifications for missing runtime validations #344

Goal

Audit all Grid xtask end-to-end qualifications and add missing validations for behavior that must be proved with deployed Kubernetes operators, Services, peer connections, overlays, and gateway pods. Keep the release qualification matrix, each topology README, and the tests themselves aligned.

This is a standing coverage task, not a qualification of one commit, branch, local build, or particular CI run. Add each new proof to the xtask and topology that owns the behavior. An existing standalone qualification counts as coverage even if it is not in a triggered workflow.

Audit and implementation

1. Inventory every xtask E2E scenario, including provider traffic, static weighted selection, token quota, multi-gateway, combined-site, GLB and no-ingress, llm-d metrics, operator/site lifecycle, and other xtask environment qualifications. Record the behavior each actually asserts at the HTTP and deployed-resource boundaries, including quick/full differences and optional skips.
2. Compare that inventory with supported Grid behavior and feature documentation. Identify gaps where a Rust unit or integration test cannot prove that the installed system behaves correctly.
3. Extend the relevant xtasks with focused, deterministic scenarios. Assert resource changes, operator convergence, accepted and serving revisions, gateway responses and attribution, recovery, and cleanup where applicable. Preserve first failures and exact source/image provenance in evidence.
4. Run the affected qualifications against the same source and images being claimed. Report passing, failing, blocked, and skipped paths separately. Do not weaken a proof or count a skipped optional path as executed.
5. Update the qualification matrix and topology READMEs to state what each suite proves and when it is required.

Starting gaps to verify

These are candidates for the audit, not a claim that every behavior below is absent from every xtask:

• Cross-site EPP signals and measured site selection. Prove the operator scrapes local EPP metrics, receives peer signals through verified mTLS, and the serving gateway polls its local operator. Change load and readiness while traffic runs; verify site choice, stale/unmeasured behavior, refusal of mismatched identities, and recovery without a gateway restart. The poll architecture describes the contract, and the hub-site E2E README says that topology does not cover poll transport.
• Weighted final-provider withdrawal. Extend the static-weighted qualification through the zero-provider boundary. Verify authoritative empty-overlay publication, consumer acceptance and serving revision, no-route responses for fresh and bound requests, provider non-contact, and restoration without restart. Check the existing provider-traffic qualification first and reuse its lifecycle patterns.
• Network loss versus signal loss. Determine which partition, staleness, and reconvergence cases already run in the operator xtasks. Add only missing deployed traffic and gateway assertions, and distinguish loss of SWIM connectivity from loss of the metrics poll path.
• Other supported runtime paths. Check whether geography-tier routing and external-provider deployment have an asserted end-to-end path when claimed as supported. Include credential projection/rotation and unauthorized-path denial for external providers where applicable.

Existing coverage and prerequisites

• Do not duplicate a scenario simply because its xtask is absent from a triggered workflow. Track workflow selection separately from missing feature validation.
• Auto-discovered GridSite specs currently omit region and zone in discovery rendering. A geography-tier test must use a supported source for that metadata; if no such path exists, record the product prerequisite rather than assert nonexistent behavior.
• Keep optional external-provider paths optional until their supported release scope is defined. Record a skip as a skip.
• Correct stale image lists in release documentation through source validation; deploying another cluster is not needed for that documentation fix.