ai-rate-limiting

Limits requests to an LLM provider by tokens or count.

Category AIAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin ai-rate-limitingPriority 1030

What it does

ai-rate-limiting holds the requests of a route to a budget of language model tokens, not of requests. It counts what ai-proxy or ai-proxy-multi on the same route reads from the model's answer. The Limit is a number of tokens for each Time window.

Each answer within the budget carries the limit and what remains of it, under headers named after the instance that answered. With a limit of 10000, one answer that used 30 tokens leaves 9970 in X-Ai-Ratelimit-Remaining-Ai-Proxy-Openai-Compatible.

When to use it

A cap of 10000 tokens a minute on an assistant route, so one busy client cannot spend the provider budget of all the others.
A budget of tokens an hour for each consumer an authentication plugin names, kept apart with a rule keyed by $consumer_name.
Output tokens that cost more than input tokens, counted three times as much with a Cost expression.
One budget that several gateway nodes count together, kept in Redis, a Redis cluster or Redis Sentinel.
A different budget for each model behind ai-proxy-multi, set per instance under Instances.

Not for counting requests rather than model tokens — see limit-count (Limits the number of requests in a time window, per key) or sending the request to the model in the first place — see ai-proxy (Proxies requests to an LLM provider with a unified API).

Guides

Pages that walk through a job with ai-rate-limiting, step by step.

Put a language model behind the gatewaySend your apps' chat requests to one language model through a route, with the model's key on the gateway, a token budget, a prompt guard and a cache.

Set it up

Share the token budget across gateway nodes through Redis

Counts the tokens ai-proxy or ai-proxy-multi on the same route report from the model's answer.

Under Basics choose One budget of tokens, set Limit to 10000 and set Time window to 60 seconds. Under Behavior set Policy to redis, set Redis host to cache.example.com, leave Redis port at 6379 and set Redis password to <secret>.

ai-rate-limiting · as the console saves it
{
  "limit": 10000,
  "time_window": 60,
  "policy": "redis",
  "redis_host": "cache.example.com",
  "redis_port": 6379,
  "redis_password": "<secret>"
}

More scenarios

Share the token budget across gateway nodes through a Redis cluster

Counts the tokens ai-proxy or ai-proxy-multi on the same route report from the model's answer.

Under Basics choose One budget of tokens, set Limit to 10000 and set Time window to 60 seconds. Under Behavior set Policy to redis-cluster, set Redis cluster name to ai-limits, fill Redis cluster nodes as in the JSON below and set Redis password to <secret>.

ai-rate-limiting · as the console saves it
{
  "limit": 10000,
  "time_window": 60,
  "policy": "redis-cluster",
  "redis_cluster_name": "ai-limits",
  "redis_cluster_nodes": [
    "10.0.3.10:6379",
    "10.0.3.11:6379",
    "10.0.3.12:6379"
  ],
  "redis_password": "<secret>"
}

Share the token budget across gateway nodes through Redis Sentinel

Counts the tokens ai-proxy or ai-proxy-multi on the same route report from the model's answer.

Under Basics choose One budget of tokens, set Limit to 10000 and set Time window to 60 seconds. Under Behavior set Policy to redis-sentinel, set Redis master name to ai-limits, fill Redis sentinels as in the JSON below and set Redis password to <secret>.

ai-rate-limiting · as the console saves it
{
  "limit": 10000,
  "time_window": 60,
  "policy": "redis-sentinel",
  "redis_master_name": "ai-limits",
  "redis_sentinels": [
    {
      "host": "10.0.3.20",
      "port": 26379
    },
    {
      "host": "10.0.3.21",
      "port": 26379
    }
  ],
  "redis_password": "<secret>"
}

Allow 10000 tokens a minute on a route

Counts the tokens ai-proxy or ai-proxy-multi on the same route report from the model's answer.

Under Basics choose One budget of tokens, set Limit to 10000 and set Time window to 60 seconds. Under Behavior set Rejected code to 429.

ai-rate-limiting · as the console saves it
{
  "limit": 10000,
  "time_window": 60,
  "rejected_code": 429
}

What you will see

A prompt is passed to the model and answered 200; the answer carries X-Ai-Ratelimit-Limit-Ai-Proxy-Openai-Compatible: 10000, the budget of the window.
The same answer carries X-Ai-Ratelimit-Remaining-Ai-Proxy-Openai-Compatible: 9970: the tokens the model reported for this answer are already taken from the budget.
Behind ai-proxy-multi the header is named after the instance that answered, X-Ai-Ratelimit-Limit-Primary: 10000.

Count an output token three times as much as an input token

The names are the ones the model's answer carries: prompt_tokens and completion_tokens for a chat completion, input_tokens and output_tokens for a Messages or Responses answer; a name the answer lacks counts as zero.

Under Basics choose One budget of tokens, set Limit to 20000, set Time window to 60 seconds, set Limit strategy to expression and set Cost expression to prompt_tokens + 3 * completion_tokens.

ai-rate-limiting · as the console saves it
{
  "limit": 20000,
  "time_window": 60,
  "limit_strategy": "expression",
  "cost_expr": "prompt_tokens + 3 * completion_tokens"
}

Give each consumer its own token budget an hour

Needs an authentication plugin on the same route, which sets the consumer name, and ai-proxy or ai-proxy-multi, which report the tokens the model used.

Under Basics choose A budget for each key and fill Rules as in the JSON below.

ai-rate-limiting · as the console saves it
{
  "rules": [
    {
      "count": 50000,
      "time_window": 3600,
      "key": "$consumer_name"
    }
  ]
}

What you will see

A prompt sent with the key of mobile-app is answered 200; the answer carries X-Ai-1-Ratelimit-Limit: 50000, that consumer's budget for the hour.
The same answer carries X-Ai-1-Ratelimit-Remaining: 49970: the tokens of this answer are taken from that consumer's budget alone.
A further prompt from the same consumer, here behind ai-proxy-multi, is answered with X-Ai-1-Ratelimit-Remaining: 49940; the count carries on across the two plugins.
A prompt without a key is answered 401 by the authentication plugin, so no budget is touched.

Things to know

Limit — Behind ai-proxy the headers name the plugin and its provider. A first answer carries X-Ai-Ratelimit-Limit-Ai-Proxy-Openai-Compatible with 10000, and after an answer of 30 tokens the remaining count is 9970.
Limit — Text is read on each request as a $variable, ${name}, $env:// or $secret:// reference. Any other text is saved and then fails every request with 500.
Time window — A Limit without a Time window is refused when you save.
Time window — Text is read on each request as a $variable, ${name}, $env:// or $secret:// reference. Any other text is saved and then fails every request with 500.
Instances — Behind ai-proxy-multi the headers carry the instance's own name. An instance named primary gives X-Ai-Ratelimit-Limit-Primary.
Instances — Each row's name must match the instance the AI proxy picks; the plain ai-proxy is named ai-proxy-<provider>. A row no request picks is never counted, and a limit set outside this list counts every instance without a row.
Key — A rule keyed by $consumer_name keeps one budget per consumer, whichever proxy plugin answers. The consumer's answers carry X-Ai-1-Ratelimit-Remaining with 49970, then 49940, and a request without a key is answered 401 by key-auth.
Key — A rule whose key resolves to nothing is skipped, and with no rule left the request is answered 500.
Header prefix — Without a Header prefix, the headers of the first rule are X-Ai-1-Ratelimit-Limit and X-Ai-1-Ratelimit-Remaining.
Rules — A Limit beside Rules is refused when you save, so a configuration takes one shape or the other.
Cost expression — With prompt_tokens + 3 * completion_tokens, an answer of 10 prompt and 20 completion tokens takes 10 plus three times 20 from the budget, and 20000 becomes 19930.
Cost expression — A name the model's answer does not carry counts as zero.
Cost expression — An expression that does not compile is refused when you save.
Redis host — With Policy redis, a configuration without a Redis host is refused when you save.
Rejected code — A code outside 200 to 599, 600 for example, is refused when you save.
Allow degradation — When the limit cannot be counted, because the store is unreachable or a value resolves to nothing, off answers 500 and on lets every request through unlimited.
AdviceSet a Rejected code, 429 for example, so a client over its budget can tell it apart from a fault of the model.

Settings

The fields as the console shows them, section by section. A field under “When … is …” is there only while that value is set.

Basics

Choose one:

One budget of tokensA count and a window for the route, for each AI instance the proxy picks, or both together.
A budget for each keyA list of rules, each with a key such as $remote_addr and a count and window of its own. A rule whose key resolves to nothing is skipped, and with no rule left the request is answered 500.

One budget of tokens

Limitlimitinteger or stringRequired unless Instances is set

The number of tokens allowed in each window, as a number or a request variable such as $http_x_token_limit. Needs Time window beside it.

integer over 0 · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

Time windowtime_windowinteger or stringRequired unless Instances is set

How long each window of Limit lasts: a number, or a $variable read from every request, such as $http_x_window. Needs Limit beside it.

integer over 0 · in seconds · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

Instancesinstancesarray of objectsRequired unless Limit with Time window is set

at least 1 item

Each item:

Limitinstances[].limitinteger or stringRequired

The tokens this instance may use in each window: a number, or a $variable read from every request.

integer over 0 · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

Nameinstances[].namestringRequired

Time windowinstances[].time_windowinteger or stringRequired

How long each window of this instance lasts: a number, or a $variable read from every request.

integer over 0 · in seconds · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

A budget for each key

Rulesrulesarray of objectsRequired

Budgets per key, each keyed by a request variable written with its dollar sign, such as $remote_addr. The header prefix names the X-AI-<prefix>-RateLimit-* headers of that rule.

Each item:

Countrules[].countinteger or stringRequired

The tokens one key may use in each window of this rule: a number, or a $variable read from every request.

integer over 0 · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

Keyrules[].keystringRequired

Time windowrules[].time_windowinteger or stringRequired

How long each window of this rule lasts: a number, or a $variable read from every request.

integer over 0 · in seconds · a whole number above zero, a $variable, or a $secret:// or $env:// reference

pattern

^([0-9]*[1-9][0-9]*|[$](env|secret)://.+|[$][{][^}]*[}]|[$][A-Za-z_][A-Za-z0-9_.]*)$

Header prefixrules[].header_prefixstring

The prefix the rate-limit headers of this rule carry.

Limit strategylimit_strategystringdefault total_tokens

Which count the limit is measured on: total tokens, prompt tokens, completion tokens, or an expression of your own.

one of completion_tokens, expression, prompt_tokens, total_tokens

When Limit strategy is expression

Cost expressioncost_exprstringRequired

An arithmetic expression over the usage the model reports, such as input_tokens + output_tokens. A name the answer does not carry counts as zero.

at least 1 character · example input_tokens + cache_creation_input_tokens + output_tokens

Behavior

Policypolicystringdefault local

one of local, redis, redis-cluster, redis-sentinel

When Policy is redis

Redis hostredis_hoststringRequired

at least 2 characters

Redis portredis_portintegerdefault 6379

at least 1

Redis databaseredis_databaseintegerdefault 0

at least 0

Redis usernameredis_usernamestring

at least 1 character

Redis passwordredis_passwordstring

Stored on the engine and shown only to roles that manage this object.

Redis SSLredis_sslbooleandefault false

Redis SSL verifyredis_ssl_verifybooleandefault false

Server name for TLSredis_server_namestring

The server name sent in the TLS handshake and, with verification on, checked against the certificate; unset, the host is sent, and an address never is.

at least 1 character

Redis timeoutredis_timeoutintegerdefault 1000

at least 1 · in milliseconds

Redis keepalive poolredis_keepalive_poolintegerdefault 100

at least 1

Redis keepalive timeoutredis_keepalive_timeoutintegerdefault 10000

at least 1000 · in milliseconds

When Policy is redis-cluster· 8 fields

Redis cluster nameredis_cluster_namestringRequired

Redis cluster nodesredis_cluster_nodesarray of stringsRequired

at least 1 item; each 2 to 100 characters

Redis cluster SSLredis_cluster_sslbooleandefault false

Redis cluster SSL verifyredis_cluster_ssl_verifybooleandefault false

Redis passwordredis_passwordstring

Stored on the engine and shown only to roles that manage this object.

Redis timeoutredis_timeoutintegerdefault 1000

at least 1 · in milliseconds

Redis keepalive poolredis_keepalive_poolintegerdefault 100

at least 1

Redis keepalive timeoutredis_keepalive_timeoutintegerdefault 10000

at least 1000 · in milliseconds

When Policy is redis-sentinel· 13 fields

Redis master nameredis_master_namestringRequired

at least 1 character

Redis sentinelsredis_sentinelsarray of objectsRequired

at least 1 item

Each item:

Hostredis_sentinels[].hoststringRequired

at least 2 characters

Portredis_sentinels[].portintegerRequired

1 to 65535

Redis roleredis_rolestringdefault master

one of master, slave

Redis databaseredis_databaseintegerdefault 0

at least 0

Redis usernameredis_usernamestring

at least 1 character

Redis passwordredis_passwordstring

Stored on the engine and shown only to roles that manage this object.

Sentinel usernamesentinel_usernamestring

at least 1 character

Sentinel passwordsentinel_passwordstring

Stored on the engine and shown only to roles that manage this object.

Redis connect timeoutredis_connect_timeoutintegerdefault 1000

at least 1 · in milliseconds

Redis read timeoutredis_read_timeoutintegerdefault 1000

at least 1 · in milliseconds

Redis keepalive timeoutredis_keepalive_timeoutintegerdefault 60000

at least 1 · in milliseconds

Rejected coderejected_codeintegerdefault 503

200 to 599

Rejected messagerejected_msgstring

at least 1 character

Show limit quota headershow_limit_quota_headerbooleandefault true

Allow degradationallow_degradationbooleandefault false

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.

G8 dev · Apache APISIX® 3.19.0

Apache, Apache APISIX and APISIX are trademarks of The Apache Software Foundation.

What it does​

When to use it​

Guides​

Set it up​

Share the token budget across gateway nodes through Redis​

More scenarios​

What you will see​

What you will see​

Things to know​

Settings​

Basics​

One budget of tokens​

A budget for each key​

When Limit strategy is expression​

Behavior​

When Policy is redis​

What it does

When to use it

Guides

Set it up

Share the token budget across gateway nodes through Redis

More scenarios

What you will see

What you will see

Things to know

Settings

Basics

One budget of tokens

A budget for each key

When Limit strategy is expression

Behavior

When Policy is redis