ai-proxy-multi
Proxies requests to several LLM providers with balancing and fallback.
ai-proxy-multiPriority 1041What it does
ai-proxy-multi sends each chat request a route takes to one of several language models, listed under Instances. Each instance carries its own provider, key, model and weight. The Balancer picks one instance per request, by round robin unless you choose another Algorithm.
Without a Fallback strategy, a failure of the chosen instance reaches the client. With one, http_429 or http_5xx for example, a request whose instance answers with such a status moves on to another instance.
When to use it
- Two providers behind one assistant, most requests to one and a share to the other, for example 80 to 20 by weight.
- A second provider that takes over when the first answers that it is overloaded or failing.
- Each partner kept on one instance, with the consistent hash algorithm hashing on the consumer.
- Prompts sent by what they ask, translations to one model and poems to another, with the semantic algorithm and a few examples per instance.
- A token budget for each instance, counted by ai-rate-limiting under the instance's name.
Not for one model at one provider — see ai-proxy (Proxies requests to an LLM provider with a unified API), balancing requests over your own backend instances — see Upstream (Upstreams are the backends that routes and services send traffic to) or splitting traffic between ordinary upstreams — see traffic-split (Splits traffic between upstreams by weight and match rules).
Set it up
Split chat requests between two providers by weight
Each Authorization value is sent to its provider as written, so write it as Bearer, a space and that provider's key.
Under Basics fill Instances as in the JSON below.
{
"instances": [
{
"name": "primary",
"provider": "openai",
"weight": 80,
"auth": {
"header": {
"Authorization": "<secret>"
}
},
"options": {
"model": "gpt-4o"
}
},
{
"name": "secondary",
"provider": "deepseek",
"weight": 20,
"auth": {
"header": {
"Authorization": "<secret>"
}
},
"options": {
"model": "deepseek-chat"
}
}
]
}
Things to know
- name — ai-rate-limiting on the same route counts the tokens of the instance that answered under its name. An instance named
primarygives the answerX-Ai-Ratelimit-Limit-Primary. - name — Two instances with one name are refused when you save.
- endpoint — An endpoint that is not an address with a scheme and a host is refused when you save.
- Algorithm — Semantic takes no part in health checks, fallback or retries. It falls back only when no instance clears its threshold or the embedding fails.
- Hash on — Read only with the Chash algorithm, where a Key is required unless Hash on is Consumer. Under any other algorithm a value here is stored and ignored.
- Key — Read only with the Chash algorithm, where a Key is required unless Hash on is Consumer. Under any other algorithm a value here is stored and ignored.
- Fallback strategy — Instance health and rate limiting works only as the single choice, and Rate limiting only inside a list. The engine refuses either word in the other shape.
- Fallback retry limit — Empty, a failed request keeps trying until one instance succeeds or all have been tried.
- Retry failures within — A failure slower than this window is returned to the client instead of moving on.
- Semantic options — Read only while the balancer algorithm is Semantic, and under any other algorithm even an invalid embedding endpoint is stored and ignored.
- examples — Under the semantic algorithm, an instance without examples is refused when you save.
- Timeout — Every request waits for the embedding call.
- Fallback — A name that no instance carries is refused.
- Fallback — Empty, the first instance answers.
- Threshold — The debugging header rounds scores to four places, so
1.0000can still fall short of 1. - Response size limit — A non-streamed answer over the cap reaches the client as
502withmax_response_bytes exceeded.
Settings
The fields as the console shows them, section by section.
Basics
instancesarray of objectsRequiredThe AI services to balance across, one object each with a name, provider, weight, auth and options, as in the example. Other keys are stored as written.
[{"name":"primary","provider":"openai","weight":80,"auth":{"header":{"Authorization":"Bearer YOUR_API_KEY"}},"options":{"model":"gpt-4o"}},{"name":"fallback","provider":"deepseek","weight":20,"auth":{"header":{"Authorization":"Bearer YOUR_API_KEY"}},"options":{"model":"deepseek-chat"}}]instances[].authobjectRequiredinstances[].auth.awsobjectOther keys are stored as written.
instances[].auth.aws.access_key_idstringRequiredinstances[].auth.aws.secret_access_keystringRequiredStored on the engine and shown only to roles that manage this object.
instances[].auth.aws.session_tokenstringStored on the engine and shown only to roles that manage this object.
instances[].auth.gcpobjectOther keys are stored as written.
instances[].auth.gcp.expire_early_secsintegerdefault 60instances[].auth.gcp.max_ttlintegerinstances[].auth.gcp.service_account_jsonstringStored on the engine and shown only to roles that manage this object.
instances[].auth.headerobjectStored on the engine and shown only to roles that manage this object. An object whose keys are letters, digits, dots, underscores and hyphens, each with a string as its value.
instances[].auth.queryobjectStored on the engine and shown only to roles that manage this object. An object whose keys are letters, digits, dots, underscores and hyphens, each with a string as its value.
instances[].namestringRequiredinstances[].providerstringRequiredinstances[].weightintegerRequiredinstances[].checksobjectOther keys are stored as written.
instances[].checks.activeobjectRequiredOther keys are stored as written.
instances[].checks.active.concurrencyintegerdefault 10instances[].checks.active.healthyobjectOther keys are stored as written.
instances[].checks.active.healthy.http_statusesarray of integersdefault [200,302]instances[].checks.active.healthy.intervalintegerdefault 1instances[].checks.active.healthy.successesintegerdefault 2instances[].checks.active.hoststringpattern
^\*$|^\*?[0-9a-zA-Z-._\[\]:]+$instances[].checks.active.http_methodstringdefault GETinstances[].checks.active.http_pathstringdefault /instances[].checks.active.http_req_bodystringdefault instances[].checks.active.https_verify_certificatebooleandefault trueinstances[].checks.active.portintegerinstances[].checks.active.req_headersarray of stringsinstances[].checks.active.timeoutnumberdefault 1instances[].checks.active.typestringdefault httpinstances[].checks.active.unhealthyobjectOther keys are stored as written.
instances[].checks.active.unhealthy.http_failuresintegerdefault 5instances[].checks.active.unhealthy.http_statusesarray of integersdefault [429,404,500,501,502,503,504,505]instances[].checks.active.unhealthy.intervalintegerdefault 1instances[].checks.active.unhealthy.tcp_failuresintegerdefault 2instances[].checks.active.unhealthy.timeoutsintegerdefault 3instances[].examplesarray of stringsinstances[].optionsobjectOther keys are stored as written.
instances[].options.modelstringinstances[].overrideobjectOther keys are stored as written.
instances[].override.endpointstringpattern
^https?://[^:/]+instances[].override.llm_optionsobjectinstances[].override.llm_options.max_tokensintegerinstances[].override.request_bodyobjectinstances[].override.request_body.anthropic-messagesobjectinstances[].override.request_body.bedrock-converseobjectinstances[].override.request_body.openai-chatobjectinstances[].override.request_body.openai-embeddingsobjectinstances[].override.request_body.openai-responsesobjectinstances[].override.request_body.passthroughobjectinstances[].override.request_body_force_overridebooleandefault falseinstances[].priorityintegerdefault 0instances[].provider_confobjectOther keys are stored as written.
instances[].provider_conf.project_idstringinstances[].provider_conf.regionstringinstances[].thresholdnumberBehavior
balancerobjectdefault {"algorithm":"roundrobin"}balancer.algorithmstringHow an instance is picked for each request: round robin, consistent hash, or semantic by prompt similarity. Semantic takes no part in health checks, fallback or retries. It falls back only when no instance clears its threshold or the embedding fails.
balancer.hash_onstringdefault varsbalancer.keystringThe request value the Chash balancer hashes, such as remote_addr or a header name, read according to Hash on.
fallback_strategystring or array of stringsfallback_http_statusesarray of integersUpstream status codes that also send the request to another instance, beside the ones the strategy names.
loggingobjectlogging.payloadsbooleandefault falseAdds the request and response bodies to the record that a logger plugin on this route writes. Nothing is written unless a logger plugin runs on the route.
logging.summariesbooleandefault falseAdds the model, the duration and the token counts to the record that a logger plugin on this route writes. Nothing is written unless a logger plugin runs on the route.
max_retriesintegerHow many more instances a failed request may try, used only with a fallback strategy. Empty keeps trying until one succeeds or all have been tried.
retry_on_failure_within_msintegerA failed request moves to another instance only when it failed within this window. A slower failure is returned to the client.
semantic_optsobjectsemantic_opts.embeddingsobjectRequiredsemantic_opts.embeddings.authobjectRequiredsemantic_opts.embeddings.auth.headerobjectStored on the engine and shown only to roles that manage this object. An object whose keys are any name, each with a string as its value.
semantic_opts.embeddings.auth.queryobjectStored on the engine and shown only to roles that manage this object. An object whose keys are any name, each with a string as its value.
semantic_opts.embeddings.modelstringRequiredThe embedding model that turns the prompt into a vector for the similarity match.
semantic_opts.embeddings.providerstringRequiredThe service that computes the embeddings for the semantic match.
semantic_opts.embeddings.endpointstringThe embedding service address, which OpenAI may leave empty and Azure OpenAI needs as the full deployment URL with its API version.
pattern
^https?://[^:/]+semantic_opts.embeddings.ssl_verifybooleandefault truesemantic_opts.embeddings.timeoutintegerdefault 3000How long the embedding call may take before the request falls back. Every request waits for this call.
semantic_opts.debuggingbooleandefault falseAdds the per-instance scores and the chosen instance to the response headers.
semantic_opts.fallbackstringThe name of the instance that answers when no instance clears its threshold or the embedding fails. Empty uses the first instance.
semantic_opts.thresholdnumberdefault 0The similarity score an instance must reach to be chosen. Below it the request goes to the fallback instance, or to the first instance when none is named. The debugging header rounds scores to four places, so 1.0000 can still fall short of 1.
timeoutintegerdefault 30000How long the gateway waits for the model to answer one call.
ssl_verifybooleandefault truemax_response_bytesintegerThe largest model response in bytes, streamed or not, where empty sets no cap. A non-streamed answer over the cap reaches the client as 502 'max_response_bytes exceeded'.
max_stream_duration_msintegerThe longest a streamed answer may run before it is cut off, and unset means no limit.
streaming_flush_interval_msintegerdefault 10How often buffered streamed output is sent on to the client, where zero sends each chunk as it arrives.
Advanced
max_req_body_sizeintegerdefault 67108864The largest request body in bytes the gateway reads, and a larger one is refused with 413.
keepalivebooleandefault truekeepalive_poolintegerdefault 30keepalive_timeoutintegerdefault 60000How long an idle connection to the provider is kept open for reuse.