ai-proxy

Proxies requests to an LLM provider with a unified API.

Category AIAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin ai-proxyPriority 1040

What it does

ai-proxy sends the requests a route takes to a language model at a provider, in place of an upstream. The Provider picks the service and the request format. The Auth holds the key the provider expects, and the Options name the model, for example gpt-4o.

A client posts a chat request to the route without a provider key of its own and gets the model's answer back in the same call. A provider with no address of its own, openai-compatible for example, is called at the address under Endpoint.

When to use it

• An assistant inside the orders API that calls one model, with the provider key held at the gateway instead of in every service that calls it.
• A model served inside your own network behind an OpenAI-style API, reached with openai-compatible and the address under Endpoint.
• A model on Amazon Bedrock, or a Gemini model on Vertex AI, signed with AWS keys or a Google service account instead of a header.
• One place to cap what a call may produce, with Output token limit replacing whatever the client asked for.
• A token budget per route or per consumer, counted by ai-rate-limiting from the usage the model reports.

Not for spreading requests over several models or providers, with fallback — see ai-proxy-multi (Proxies requests to several LLM providers with balancing and fallback), sending requests to your own backend instances — see Upstream (Upstreams are the backends that routes and services send traffic to) or changing the path or headers of a request to an ordinary backend — see proxy-rewrite (Rewrites the request URI, method, host and headers before proxying).

Guides

Pages that walk through a job with ai-proxy, step by step.

• Put a language model behind the gatewaySend your apps' chat requests to one language model through a route, with the model's key on the gateway, a token budget, a prompt guard and a cache.

Set it up

Send chat requests to a model on Amazon Bedrock

Under Basics set Provider to bedrock, fill Provider settings as in the JSON below, fill Auth as in the JSON below and fill Options as in the JSON below.

ai-proxy · as the console saves it

{
  "provider": "bedrock",
  "auth": {
    "aws": {
      "access_key_id": "$env://AWS_ACCESS_KEY_ID",
      "secret_access_key": "<secret>"
    }
  },
  "provider_conf": {
    "region": "us-east-1"
  },
  "options": {
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0"
  }
}

More scenarios

Send chat requests to a Gemini model on Vertex AI

Under Basics set Provider to vertex-ai, fill Provider settings as in the JSON below, fill Auth as in the JSON below and fill Options as in the JSON below.

ai-proxy · as the console saves it

{
  "provider": "vertex-ai",
  "auth": {
    "gcp": {
      "service_account_json": "$env://GOOGLE_SERVICE_ACCOUNT_JSON"
    }
  },
  "provider_conf": {
    "project_id": "orders-prod",
    "region": "europe-west4"
  },
  "options": {
    "model": "gemini-2.0-flash"
  }
}

Things to know

• A client posts a body with messages to the route and is answered 200 with the model's answer, Hello from the mock model in the ai-cache example this row cites. The route carries no upstream, and the client sends no provider key.
• Region — With Provider bedrock, a configuration without a Region is refused when you save.
• Project ID — Vertex AI needs the project and the region together, or an endpoint override instead, and one of the two alone is refused.
• Auth — A configuration without Auth is refused when you save.
• Auth — For Bedrock, write the AWS keys under aws; a header or query credential is refused with that provider.
• Endpoint — A scheme and host alone keep the provider's own path. An address with a path is used as written, so https://llm.internal/v1 is called as /v1.
• Endpoint — Two providers carry no address of their own, openai-compatible and azure-openai, so a document that names one of them is refused until this override carries an address; the engine alone would store it and answer every request with 500.
• Request body — Objects merge key by key. A list or a single value replaces the client's only with Request body force override on.
• Output token limit — The limit is sent under the name the provider expects, for example max_completion_tokens for OpenAI chat.
• Summaries — Nothing is written unless a logger plugin runs on the route.
• Timeout — The longest Timeout is 600000 milliseconds; 600001 is refused when you save.
• Response size limit — A non-streamed answer over the cap reaches the client as 502 with max_response_bytes exceeded.
• ai-rate-limiting on the same route counts the tokens the model reports and names this plugin after its provider, ai-proxy-openai-compatible. The answer then carries X-Ai-Ratelimit-Limit-Ai-Proxy-Openai-Compatible.
• AdviceWrite a provider key as a $secret:// or $env:// reference rather than as plain text in the route.

Settings

The fields as the console shows them, section by section. A field under “When … is …” is there only while that value is set.

Basics

ProviderproviderstringRequired

The AI service the request is sent to, which sets the request format and the default address. Endpoint under Override replaces the address.

one of aimlapi, anthropic, azure-openai, bedrock, deepseek, gemini, openai, openai-compatible, openrouter, vertex-ai

When Provider is bedrock

Provider settingsprovider_confobjectRequired

Regionprovider_conf.regionstringRequired

The cloud region the model is served from, such as us-east-1 for Bedrock or us-central1 for Vertex AI.

at least 1 character

When Provider is vertex-ai· 3 fields

Provider settingsprovider_confobject

Project IDprovider_conf.project_idstring

The Google Cloud project the Vertex AI model belongs to.

at least 1 character

Regionprovider_conf.regionstring

The cloud region the model is served from, such as us-east-1 for Bedrock or us-central1 for Vertex AI.

at least 1 character

AuthauthobjectRequired

example {"header":{"Authorization":"Bearer YOUR_API_KEY"}}

awsauth.awsobject

Other keys are stored as written.

access_key_idauth.aws.access_key_idstringRequired

at least 1 character

secret_access_keyauth.aws.secret_access_keystringRequired

Stored on the engine and shown only to roles that manage this object.

at least 1 character

session_tokenauth.aws.session_tokenstring

Stored on the engine and shown only to roles that manage this object.

at least 1 character

gcpauth.gcpobject

Other keys are stored as written.

expire_early_secsauth.gcp.expire_early_secsintegerdefault 60

at least 0

max_ttlauth.gcp.max_ttlinteger

at least 1

service_account_jsonauth.gcp.service_account_jsonstring

Stored on the engine and shown only to roles that manage this object.

headerauth.headerobject

Stored on the engine and shown only to roles that manage this object. An object whose keys are letters, digits, dots, underscores and hyphens, each with a string as its value.

queryauth.queryobject

Stored on the engine and shown only to roles that manage this object. An object whose keys are letters, digits, dots, underscores and hyphens, each with a string as its value.

Optionsoptionsobject

Settings passed to the model with every request, such as the model name, the temperature or the maximum tokens. Other keys are stored as written.

example {"model":"gpt-4o","temperature":0.2}

modeloptions.modelstring

OverrideoverrideobjectRequired when Provider is azure-openai or openai-compatible

Endpointoverride.endpointstringRequired when Provider is azure-openai or openai-compatible

An address to send requests to instead of the provider's own, either a scheme and host or a full URL with its path. A scheme and host alone keep the provider's own path. An address with a path is used exactly as written, so https://llm.internal/v1 is called as /v1.

at least 1 character · example https://llm.internal/v1

LLM optionsoverride.llm_optionsobject

Output token limitoverride.llm_options.max_tokensinteger

The most output tokens a call may produce, replacing whatever the client asked for. It is sent under the name the provider expects, such as max_completion_tokens for OpenAI chat or max_output_tokens for the Responses API.

at least 1

Request bodyoverride.request_bodyobject

JSON merged into the outgoing request body, under the key of the protocol the request is sent in. Objects merge key by key. Lists and single values replace the client's only with force override on.

Anthropic-messagesoverride.request_body.anthropic-messagesobject

Merged into the request body when the target protocol is Anthropic messages.

example {"metadata":{"user_id":"gateway"}}

Bedrock-converseoverride.request_body.bedrock-converseobject

Merged into the request body when the target protocol is Bedrock converse.

example {"inferenceConfig":{"topP":0.9}}

OpenAI chatoverride.request_body.openai-chatobject

Merged into the request body when the target protocol is OpenAI chat.

example {"top_p":0.9}

OpenAI embeddingsoverride.request_body.openai-embeddingsobject

Merged into the request body when the target protocol is OpenAI embeddings.

example {"encoding_format":"float"}

OpenAI responsesoverride.request_body.openai-responsesobject

Merged into the request body when the target protocol is OpenAI responses.

example {"store":false}

Passthroughoverride.request_body.passthroughobject

Merged into the request body when the request is passed through in its own format.

example {"stream":false}

Request body force overrideoverride.request_body_force_overridebooleandefault false

On, the override values replace what the client sent, and off they only fill in what the client left out.

Behavior

Loggingloggingobject

Payloadslogging.payloadsbooleandefault false

Adds the request and response bodies to the record that a logger plugin on this route writes. Nothing is written unless a logger plugin runs on the route.

Summarieslogging.summariesbooleandefault false

Adds the model, the duration and the token counts to the record that a logger plugin on this route writes. Nothing is written unless a logger plugin runs on the route.

Timeouttimeoutintegerdefault 30000

How long the gateway waits for the model to answer one call.

1 to 600000 · in milliseconds

SSL verifyssl_verifybooleandefault true

Response size limitmax_response_bytesinteger

The largest model response in bytes, streamed or not, where empty sets no cap. A non-streamed answer over the cap reaches the client as 502 'max_response_bytes exceeded'.

at least 1

Stream duration limitmax_stream_duration_msinteger

The longest a streamed answer may run before it is cut off, and unset means no limit.

at least 1 · in milliseconds

Streaming flush intervalstreaming_flush_interval_msintegerdefault 10

How often buffered streamed output is sent on to the client, where zero sends each chunk as it arrives.

at least 0 · in milliseconds

Advanced

Request body limitmax_req_body_sizeintegerdefault 67108864

The largest request body in bytes the gateway reads, and a larger one is refused with 413.

at least 1

Keepalivekeepalivebooleandefault true

Keepalive poolkeepalive_poolintegerdefault 30

at least 1

Keepalive timeoutkeepalive_timeoutintegerdefault 60000

How long an idle connection to the provider is kept open for reuse.

at least 1000 · in milliseconds

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.