Currently, model="auto" builds the fallback chain automatically based on the strategy's ranking (top 5 candidates). But for my use case, I want to define explicit failover chains for specific models.
Use Case
I have a production workload that should always use gpt-5.5 as primary, with claude-opus-4-7 as the first fallback, and gemini-3.1-pro-preview as the second fallback. I don't want the auto-ranking logic to reorder these — I've tested them and know this is the right order for my prompts.
Proposed API
PUT /v1/routing
{
"strategy": "balanced",
"custom_fallback_chains": {
"gpt-5.5": ["claude-opus-4-7", "gemini-3.1-pro-preview"],
"claude-opus-4-7": ["gpt-5.5", "gemini-3.1-pro-preview"]
}
}When a request comes in with model="gpt-5.5" (pinned, not auto), the router would use the custom chain instead of the auto-generated one.
For model="auto", the behavior could be:
- Auto-rank as usual to pick the primary
- If the primary has a custom chain defined, use that chain as the fallbacks
- Otherwise, use the auto-generated fallbacks
Implementation
The chat.py handler already builds fallbacks_arg from the auto candidates:
if was_auto and len(candidates) > 1:
fallbacks_arg = [{candidates[0]: candidates[1:]}]This could be extended to check a custom_fallback_chains dict stored on the RoutingConfig model and override the fallbacks for pinned or auto-resolved primaries.
Why Not Just Use preferred_models?
preferred_models restricts the eligible set but doesn't control the order. I want explicit control over the failover sequence, not just a filter.
Alternatives Considered
- Using LiteLLM's
fallbacks config directly — but that requires editing the code, not the API
- Running multiple OrcaRouter instances with different configs — overkill for this use case
Would love to hear if this is in scope for Lite or if it's better suited for the Teams edition.
Currently,
model="auto"builds the fallback chain automatically based on the strategy's ranking (top 5 candidates). But for my use case, I want to define explicit failover chains for specific models.Use Case
I have a production workload that should always use
gpt-5.5as primary, withclaude-opus-4-7as the first fallback, andgemini-3.1-pro-previewas the second fallback. I don't want the auto-ranking logic to reorder these — I've tested them and know this is the right order for my prompts.Proposed API
When a request comes in with
model="gpt-5.5"(pinned, not auto), the router would use the custom chain instead of the auto-generated one.For
model="auto", the behavior could be:Implementation
The
chat.pyhandler already buildsfallbacks_argfrom the auto candidates:This could be extended to check a
custom_fallback_chainsdict stored on theRoutingConfigmodel and override the fallbacks for pinned or auto-resolved primaries.Why Not Just Use preferred_models?
preferred_modelsrestricts the eligible set but doesn't control the order. I want explicit control over the failover sequence, not just a filter.Alternatives Considered
fallbacksconfig directly — but that requires editing the code, not the APIWould love to hear if this is in scope for Lite or if it's better suited for the Teams edition.