Skip to content

feat(gateway): retry on another site when the dial fails #419

Open

Description

@hexfusion

grid_site_route picks one site per request and never picks again. When the dial to that site fails, the client gets the upstream error, even if another candidate site could serve the model. Praxis retries endpoints within a cluster, but no retry crosses to another site's cluster. A 5xx response only evicts the prefix-affinity entry for that cluster (route.rs, settle).

The window this leaves: a site that becomes unreachable stays selectable until its signals age past load_window_ms or the praxis health check marks its cluster down. Requests picked for it in that window fail.

Proposal: on a connect failure to the picked site, before any request bytes are sent, pick once more from the remaining candidates, excluding the failed site. Pinned requests (stored Responses API state) are out of scope, since only the holding site can serve them.

Open questions:

  • whether the praxis filter API lets a route filter re-select the cluster after a connect failure, or whether this needs a praxis change;
  • the retry budget, and whether non-idempotent requests that fail before any byte is sent are safe to retry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions