Put a language model behind the gateway

ai-proxy sends the requests a route takes to a language model, in place of a backend. Your clients post chat requests to the route and get the model's reply in the same call. The model's key stays on the gateway, so no client holds it.

This guide puts a model served behind an OpenAI-style API on a new route, assistant-api, which takes chat requests under /assistant/. Three optional steps follow: a budget of tokens with ai-rate-limiting, a guard that refuses prompts with card numbers with ai-prompt-guard, and a cache for repeated prompts with ai-cache. Amazon Bedrock and Vertex AI are covered at the end. At the end, a chat request posted to assistant-api without any key is answered by the model.

Before you start

This guide creates a new route, assistant-api, with no other plugin. The requests below post to /assistant/chat. A plugin from another guide that this page does not name can change the answers below; remove it first or work on a copy of the route.
From the team that runs the model: the full address of its chat endpoint, path included, the key it expects, and the name of the model. This guide uses the model gpt-4o.
Gateway nodes that can reach that address.
For the cache in step 5, a Redis server and its password from the team that runs it. This guide uses cache.example.com on port 6379.
The host name of your gateway. In the console it is the first part of the Address line on a route's Overview tab, api.example.com in this page's examples; replace it in every request below.
The Editor or Environment administrator role in this environment.

Steps

1. Create a route for the model

In the menu, select Routes, then New route. Under General, set Name to assistant-api. Under Matching, type /assistant/* in Paths and press Enter. Check POST under Methods, the method of a chat request.

Under Target, choose None. The route is created without a backend and gets its plugin, ai-proxy, in the next step. Select Create route.

2. Add ai-proxy and point it at your model

Select the route's Plugins tab, then Add plugin at the top right of the tab. Open the AI category, or type ai-proxy in the search box. Select ai-proxy, then Configure. Under Basics, fill in these fields and leave the rest as they are.

Field	Value	Why
Provider	`openai-compatible`	The provider for a model behind an OpenAI-style API at an address of its own.
Auth	the first text below, with the model's key	The key the model expects. It is held on the gateway, and your clients never send it.
Options	the second text below	The model every request is sent to.
Endpoint, under Override	the full address of the model's chat endpoint	An address with a path is used exactly as written.

Auth and Options are JSON boxes. In Auth, type this, with the model's key in place of the text in angle brackets:

{"header": {"Authorization": "Bearer <the model's key>"}}

In Options, type this:

{"model": "gpt-4o"}

With openai-compatible, a configuration without an endpoint is refused when you save, because that provider has no address of its own. A configuration without Auth is refused too. Where you can, write the key as $env:// followed by the name of a variable of the gateway host that holds it. The gateway reads that variable once when the configuration loads.

Select Add plugin at the bottom of the panel. To see the values again, select ai-proxy in the list of plugins.

3. Optional: hold the route to a token budget

ai-rate-limiting counts the tokens ai-proxy reads from each of the model's replies, not the requests. Select Add plugin at the top right of the tab. Open the AI category, or type ai-rate-limiting in the search box. Select ai-rate-limiting, then Configure.

Under Basics, keep One budget of tokens chosen and fill in these fields.

Field	Value	Why
Limit	10000	The tokens the whole route may use in one window.
Time window	60	The window, in seconds. A limit without a window is refused when you save.
Limit strategy	leave it	The default counts the total tokens of each reply, prompt and completion together.
Rejected code, under Behavior	429	The status a request over the budget is refused with.

Select Add plugin at the bottom of the panel. The plugin is saved like this:

ai-rate-limiting · as the console saves it
{
  "limit": 10000,
  "time_window": 60,
  "rejected_code": 429
}

4. Optional: refuse prompts that carry a card number

ai-prompt-guard reads the prompt of a chat request before it goes on to the model. Select Add plugin at the top right of the tab. Open the AI category, or type ai-prompt-guard in the search box. Select ai-prompt-guard, then Configure.

Under Deny patterns, type a value and press Enter for each of these patterns.

Pattern	What it refuses
`\b(?:\d[ -]?){15}\d\b`	A card number of 16 digits, with or without spaces or dashes between them.
`[Ii]gnore (all )?previous instructions`	A prompt that asks the model to ignore all previous instructions, with Ignore or ignore.

Beside the field, the form says: A pattern that is empty is stored and matches every request, so one empty row here refuses all of them.

Turn on Match all roles. Off, the plugin reads only the messages the user sent; on, it reads the last message of the conversation whatever its role. Select Add plugin at the bottom of the panel. The plugin is saved like this:

ai-prompt-guard · as the console saves it
{
  "deny_patterns": [
    "\\b(?:\\d[ -]?){15}\\d\\b",
    "[Ii]gnore (all )?previous instructions"
  ],
  "match_all_roles": true
}

5. Optional: answer repeated prompts from a cache

ai-cache keeps the model's replies in Redis and answers a repeated prompt from there. It acts only on a route that also carries ai-proxy or ai-proxy-multi. Select Add plugin at the top right of the tab. Open the AI category, or type ai-cache in the search box. Select ai-cache, then Configure.

Under Basics, check exact under Layers, and only that layer. Under Exact layer, set TTL to 300 seconds, five minutes. Keep Policy at redis, the only store the plugin offers.

Fill in Redis host with cache.example.com, Redis port with 6379 and Redis password with the password. A configuration without a Redis host is refused when you save. Select Add plugin at the bottom of the panel. The plugin is saved like this:

ai-cache · as the console saves it
{
  "layers": [
    "exact"
  ],
  "exact": {
    "ttl": 300
  },
  "policy": "redis",
  "redis_host": "cache.example.com",
  "redis_port": 6379,
  "redis_password": "<secret>"
}

Check it

Post a chat request to /assistant/chat. The body carries messages, and the client sends no key.

curl
curl -i -L -X POST 'https://<gateway host>/assistant/chat' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Where is my order?"}]}'

The model replies

The request is answered 200 with the model's reply. The route has no backend of its own, and the client sent no key.

The token budget

Each reply within the budget carries the budget and what remains of it. The headers are named after ai-proxy and its provider, openai-compatible here.

The budget: A prompt is passed to the model and answered 200; the answer carries X-Ai-Ratelimit-Limit-Ai-Proxy-Openai-Compatible: 10000, the budget of the window.
What remains: The same answer carries X-Ai-Ratelimit-Remaining-Ai-Proxy-Openai-Compatible: 9970: the tokens the model reported for this answer are already taken from the budget.

Refused prompts

Send prompts that carry what the patterns refuse, and one that does not. The prompt with a card number written with spaces is sent like this:

curl
curl -i -L -X POST 'https://<gateway host>/assistant/chat' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"My card is 4111 1111 1111 1111, charge it again"}]}'

Told to ignore its instructions: A prompt that tells the model to ignore its instructions is answered 400 with Request contains prohibited content; the model never sees it.
A card number with spaces: A prompt carrying a card number written with spaces is answered 400 with the same message.
A card number with dashes: A card number written with dashes is refused the same way, 400.
An order number: A prompt that only names an order number is passed to the model and answered 200.

The answer to the order number is the model's reply.

Replies from the cache

Send the first request above a few times.

The first time: The first time a prompt is seen it is passed to the model and answered 200 with X-Ai-Cache-Status: MISS; the answer is kept in Redis.
Sent again: The same prompt sent again is answered 200 with the reply the model gave the first time.
Its cache status: That repeated answer carries X-Ai-Cache-Status: HIT; the model was not called.

Give each consumer its own budget instead

A budget for the whole route lets one busy client spend what the others need. To give each client a budget of its own, the route first needs an authentication plugin that names the consumer. Put key-auth on assistant-api as Require an API key shows, and give each client a consumer with a key, mobile-app here.

Then select ai-rate-limiting in the list of plugins, then Edit. Choose A budget for each key under Basics and add one row under Rules.

Field	Value	Why
Count	50000	The tokens each consumer may use in one window.
Time window	3600	The window, in seconds, an hour.
Key	`$consumer_name`	One budget for each consumer.

Count is the budget of each key in a rule. Limit in step 3 is the budget of the whole route. Leave Header prefix empty and select Save plugin at the bottom of the panel.

ai-rate-limiting · as the console saves it
{
  "rules": [
    {
      "count": 50000,
      "time_window": 3600,
      "key": "$consumer_name"
    }
  ]
}

The request below sends the key in the apikey header. key-auth reads apikey by default, and Require an API key renames it to x-api-key, so if you followed that guide, send the key in x-api-key instead. Each reply carries the headers named after the rule, the first rule here, with that consumer's budget and what remains of it.

curl
curl -i -L -X POST 'https://<gateway host>/assistant/chat' \
  -H 'apikey: <the key of mobile-app>' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Where is my order?"}]}'

The consumer's budget: A prompt sent with the key of mobile-app is answered 200; the answer carries X-Ai-1-Ratelimit-Limit: 50000, that consumer's budget for the hour.
What remains: The same answer carries X-Ai-1-Ratelimit-Remaining: 49970: the tokens of this answer are taken from that consumer's budget alone.
Without a key: A prompt without a key is answered 401 by the authentication plugin, so no budget is touched.

Other providers

ai-proxy also calls models at Amazon Bedrock and Vertex AI, which sign each call with cloud credentials instead of a header. For these two, leave Endpoint empty and fill in Provider settings, which appears once the provider is chosen. Bedrock and Vertex AI are not shown with answers on this page.

For a model on Amazon Bedrock, set Provider to bedrock and Region to the region the model is served from, us-east-1 here. A configuration without a region is refused when you save. In the Auth box, write access_key_id and secret_access_key under aws, as the example below shows. Beside the box, the form says: For Bedrock, write the AWS keys under aws; a header or query credential is refused with that provider.

ai-proxy · as the console saves it
{
  "provider": "bedrock",
  "auth": {
    "aws": {
      "access_key_id": "$env://AWS_ACCESS_KEY_ID",
      "secret_access_key": "<secret>"
    }
  },
  "provider_conf": {
    "region": "us-east-1"
  },
  "options": {
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0"
  }
}

In the example, $env://AWS_ACCESS_KEY_ID reads the access key ID from a variable of the gateway host, and <secret> stands for the secret access key you type in its place.

For a Gemini model on Vertex AI, set Provider to vertex-ai. Set Project ID to your project and Region to its region, orders-prod and europe-west4 here. In the Auth box, write service_account_json under gcp with the JSON key of a Google service account, as the example below shows.

ai-proxy · as the console saves it
{
  "provider": "vertex-ai",
  "auth": {
    "gcp": {
      "service_account_json": "$env://GOOGLE_SERVICE_ACCOUNT_JSON"
    }
  },
  "provider_conf": {
    "project_id": "orders-prod",
    "region": "europe-west4"
  },
  "options": {
    "model": "gemini-2.0-flash"
  }
}

What to know

An answer over the budget is refused with 429; this page does not show one.
Send a few requests with all three optional plugins in place before your clients call the route.
Cache only replies that every caller of the route may see.
ai-proxy sends every request to one model. For several models or providers with a fallback, see ai-proxy-multi.

Require an API key: the consumers and keys a budget per consumer needs.
Serve answers from a cache: a cache for the answers of an ordinary route.
AI plugins in the reference, every plugin of the family.
ai-proxy, ai-rate-limiting, ai-prompt-guard and ai-cache in the reference, with the Redis shapes of the token budget and the semantic layer of the cache.

G8 dev

Before you start​

Steps​

1. Create a route for the model​

2. Add ai-proxy and point it at your model​

3. Optional: hold the route to a token budget​

4. Optional: refuse prompts that carry a card number​

5. Optional: answer repeated prompts from a cache​

Check it​

The model replies​

The token budget​

Refused prompts​

Replies from the cache​

Give each consumer its own budget instead​

Other providers​

What to know​

Related​