Feature

Model Fallbacks

Set backup models per request. If the primary model returns a rate limit or error, Rhodes automatically retries with the next model in the chain. You are only charged for the successful response.

How it works

Primary fails

Request sent to first model. Returns 429 rate limit or 503 unavailable.

Fallback triggered

Same messages array sent to next model in chain. No delay between attempts.

First success wins

Successful response returned. You are billed for that model only.

app.ts

const completion = await client.chat.completions.create({
  model: 'rhodes/llama-3.1-70b',
  fallbacks: [
    'rhodes/llama-3.3-70b',
    'rhodes/claude-sonnet-4'
  ],
  messages: [{ role: 'user', content: 'Explain TCP.' }],
});

// Request tries llama-3.1-70b first.
// On 429 rate limit or 503 error, retries with llama-3.3-70b.
// If that fails, falls back to claude-sonnet-4.
// Returns the first successful response.

0.8s Median fallback time

Primary failure detected and backup model responding, measured across 10k requests.

When to use fallbacks

Cost optimization

Try a cheaper open-weight model first, fall back to frontier only if the request is too complex or gets rate-limited.

Provider redundancy

Route through two providers. If one is down or throttling, the other takes the request without changing application code.

Rate limit smoothing

Deploy with modest rate limits per model. When one model hits its limit, traffic shifts to another without dropping requests.

Regional routing

Primary routes to a region-local model. If that region throttles, fallback routes to a different geographic deployment.

Frequently asked questions

Does fallback count as two requests?

No. You are only billed for the model that successfully completes the request. Failed attempts before the fallback do not consume credits.

What triggers a fallback?

Rate limit errors (429), service unavailable (503), or timeout after 60 seconds. 4xx client errors like invalid parameters do not trigger fallback because they would fail on any model.

Can I set different prompts per fallback model?

No. The same messages array is sent to each model in the chain. If you need model-specific prompting, make separate requests with conditional logic in your application.

How long does retry take?

Immediate. There is no backoff delay between fallback attempts. The total request time is the sum of each failed attempt plus the successful one.

Is this available on all models?

Yes. Any model in the catalog can be a primary or fallback. The models in your chain do not need to be from the same provider.

Related resources