ai-cache

Caches LLM responses by exact or semantic match.

Category AIAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin ai-cachePriority 1035

What it does

ai-cache keeps the answers a language model gives on a route in Redis and serves a repeated prompt from there, without calling the model again. It acts only on a route that also carries ai-proxy or ai-proxy-multi. The Exact layer matches a prompt word for word, and the Semantic layer adds prompts that mean the same.

The first request reaches the model and gets its answer. The same prompt again is answered from the cache, and the answer carries X-Ai-Cache-Status with HIT.

When to use it

• A help assistant where many customers ask where their order is, answered once by the model and then from Redis for five minutes.
• Prompts that ask the same thing in other words, matched by the semantic layer through embeddings and served the stored answer.
• Answers that differ by caller, kept apart per consumer with Include consumer.
• A client that must always reach the model, for example a test, sending a header listed under Bypass rules.

Not for keeping the answers of an ordinary backend — see proxy-cache (Caches upstream responses on disk or in memory) or holding a route to a budget of model tokens — see ai-rate-limiting (Limits requests to an LLM provider by tokens or count).

Guides

Pages that walk through a job with ai-cache, step by step.

• Put a language model behind the gatewaySend your apps' chat requests to one language model through a route, with the model's key on the gateway, a token budget, a prompt guard and a cache.

Set it up

Answer a repeated prompt from Redis for five minutes

Acts only on a route that also carries ai-proxy or ai-proxy-multi.

Under Basics fill Layers as in the JSON below, fill Exact layer as in the JSON below, leave Policy at redis, set Redis host to cache.example.com, leave Redis port at 6379 and set Redis password to <secret>.

ai-cache · as the console saves it

{
  "layers": [
    "exact"
  ],
  "exact": {
    "ttl": 300
  },
  "policy": "redis",
  "redis_host": "cache.example.com",
  "redis_port": 6379,
  "redis_password": "<secret>"
}

What you will see

• The first time a prompt is seen it is passed to the model and answered 200 with X-Ai-Cache-Status: MISS; the answer is kept in Redis.
• The same prompt sent again is answered 200 with the reply the model gave the first time.
• That repeated answer carries X-Ai-Cache-Status: HIT; the model was not called.
• The kept answer is shared with ai-proxy-multi on the same route: a repeated prompt there is answered 200 with X-Ai-Cache-Status: HIT too.

More scenarios

Answer prompts that mean the same thing from the cache with Azure OpenAI embeddings

Acts only on a route that also carries ai-proxy or ai-proxy-multi; the embeddings address is used as written, so it carries the api-version the deployment answers.

Under Basics fill Layers as in the JSON below and set Redis host to cache.example.com. Under Behavior fill Semantic layer as in the JSON below.

ai-cache · as the console saves it

{
  "layers": [
    "exact",
    "semantic"
  ],
  "redis_host": "cache.example.com",
  "semantic": {
    "embedding": {
      "azure_openai": {
        "api_key": "<secret>",
        "endpoint": "https://orders-ai.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2024-02-01"
      }
    },
    "similarity_threshold": 0.95,
    "vector_search": {
      "redis": {
        "index": "prompt-cache"
      }
    }
  }
}

Answer prompts that mean the same thing from the cache with OpenAI embeddings

Acts only on a route that also carries ai-proxy or ai-proxy-multi.

Under Basics fill Layers as in the JSON below and set Redis host to cache.example.com. Under Behavior fill Semantic layer as in the JSON below.

ai-cache · as the console saves it

{
  "layers": [
    "exact",
    "semantic"
  ],
  "redis_host": "cache.example.com",
  "semantic": {
    "embedding": {
      "openai": {
        "api_key": "<secret>",
        "model": "text-embedding-3-small"
      }
    },
    "similarity_threshold": 0.95,
    "vector_search": {
      "redis": {
        "index": "prompt-cache"
      }
    }
  }
}

Things to know

• TTL — With an exact layer in Redis, the first request is answered by the model, Hello from the mock model in the example. The same request again is answered 200 with X-Ai-Cache-Status set to HIT.
• The stored answer does not depend on which proxy plugin called the model. A third request, sent through ai-proxy-multi on the same route, is also a HIT.
• Layers — Only a plain-text OpenAI chat prompt reaches the semantic layer, and anything else is served by the exact layer alone. An image, an audio block or a tool call in the prompt also keeps it out of the semantic layer.
• Semantic layer — Used only when Layers includes semantic; naming semantic there without this block is refused. A block that Layers does not name is ignored and only logged.
• Embedding — An embedding names one provider, and one that names both is refused when you save.
• Endpoint — An Azure OpenAI embedding without an Endpoint is refused when you save.
• Similarity threshold — A threshold above 1, for example 1.5, is refused when you save.
• Include variables — Each name is a request variable written without its dollar sign, and a variable the request does not carry joins the key as an empty value.
• Cache status headers — On, every answer carries X-AI-Cache-Status, and a hit also carries X-AI-Cache-Age or, for a semantic hit, X-AI-Cache-Similarity.
• Redis host — A configuration without a Redis host is refused when you save, whatever the layers.
• AdviceCache only answers that every caller of the route may see. For an answer that depends on who asks, turn on Include consumer.

Settings

The fields as the console shows them, section by section. A field under “When … is …” is there only while that value is set.

Basics

Layerslayersarray of stringsdefault ["exact"]

The cache layers to use, where exact is always needed and semantic adds a similarity match. A list without exact is refused.

at least 1 item · one of exact, semantic · always carries exact

Exact layerexactobjectdefault {}

TTLexact.ttlintegerdefault 3600

How long an answer stays in the exact layer.

at least 1 · in seconds

Policypolicystringdefault redis

Redis is the only store this plugin offers.

one of redis

When Policy is redis

Redis hostredis_hoststringRequired

at least 2 characters

Redis portredis_portintegerdefault 6379

at least 1

Redis databaseredis_databaseintegerdefault 0

at least 0

Redis usernameredis_usernamestring

The Redis user name for ACL sign-in, read only when a password is set.

at least 1 character

Redis passwordredis_passwordstring

Stored on the engine and shown only to roles that manage this object.

Redis SSLredis_sslbooleandefault false

Redis SSL verifyredis_ssl_verifybooleandefault false

Server name for TLSredis_server_namestring

The server name sent in the TLS handshake and, with verification on, checked against the certificate; unset, the host is sent, and an address never is.

at least 1 character

Redis timeoutredis_timeoutintegerdefault 1000

at least 1 · in milliseconds

Redis keepalive poolredis_keepalive_poolintegerdefault 100

at least 1

Redis keepalive timeoutredis_keepalive_timeoutintegerdefault 10000

at least 1000 · in milliseconds

Behavior

Semantic layersemanticobjectRequired when Layers includes semantic

Embeddingsemantic.embeddingobjectRequired

Choose one:

• Embed with OpenAI
• Embed with Azure OpenAI

Embed with OpenAI

OpenAIsemantic.embedding.openaiobjectRequired

API keysemantic.embedding.openai.api_keystringRequired

Stored on the engine and shown only to roles that manage this object.

Modelsemantic.embedding.openai.modelstringRequired

Dimensionssemantic.embedding.openai.dimensionsinteger

The vector length asked of the provider, where the index is built for the length it actually returns.

at least 1

Endpointsemantic.embedding.openai.endpointstring

The full embeddings URL, used as written, where empty uses the public OpenAI one.

SSL verifysemantic.embedding.openai.ssl_verifybooleandefault true

Timeoutsemantic.embedding.openai.timeoutintegerdefault 5000

How long the embedding call may take, where a slower provider is served as a cache miss.

at least 1 · in milliseconds

Embed with Azure OpenAI

Azure OpenAIsemantic.embedding.azure_openaiobjectRequired

API keysemantic.embedding.azure_openai.api_keystringRequired

Stored on the engine and shown only to roles that manage this object.

Endpointsemantic.embedding.azure_openai.endpointstringRequired

The whole embeddings URL of the deployment; no path of its own is added to it.

Dimensionssemantic.embedding.azure_openai.dimensionsinteger

Asked of the provider; the index is built for the vector length it answers with.

at least 1

SSL verifysemantic.embedding.azure_openai.ssl_verifybooleandefault true

Timeoutsemantic.embedding.azure_openai.timeoutintegerdefault 5000

Connect, send and read alike; a provider slower than this is served as a miss.

at least 1 · in milliseconds

Vector searchsemantic.vector_searchobjectRequired

Redissemantic.vector_search.redisobjectRequireddefault {}

Needs the RediSearch module on Redis, because without it every semantic lookup misses and only the log says so.

Indexsemantic.vector_search.redis.indexstringdefault ai-cache

The prefix of the index name and its keys, to which the embedding model and vector length are added.

Message windowsemantic.matchobjectdefault {}

Ignore assistant promptssemantic.match.ignore_assistant_promptsbooleandefault true

Ignore system promptssemantic.match.ignore_system_promptsbooleandefault true

Ignore tool promptssemantic.match.ignore_tool_promptsbooleandefault true

Message countbacksemantic.match.message_countbackintegerdefault 1

How many of the latest messages, after the ignore switches are applied, are embedded as the query.

at least 1

Similarity thresholdsemantic.similarity_thresholdnumberdefault 0.95

A stored answer is served when its cosine similarity reaches this.

0 to 1

Top Ksemantic.top_kintegerdefault 1

How many nearest answers the search asks for, of which the nearest decides.

at least 1

TTLsemantic.ttlintegerdefault 86400

How long an answer stays in the semantic layer.

at least 1 · in seconds

Cache keycache_keyobjectdefault {}

Include consumercache_key.include_consumerbooleandefault false

On, a signed-in consumer is served only what was cached for them.

Include variablescache_key.include_varsarray of stringsdefault []

Share across routescache_key.share_across_routesbooleandefault false

Off, every route keeps a cache of its own.

Cache status headerscache_headersbooleandefault true

Bypass rulesbypass_onarray of objects

A request carrying one of these headers with that value is served by the upstream and never cached.

at least 1 item

Each item:

Equalsbypass_on[].equalsstringRequired

Headerbypass_on[].headerstringRequired

at least 1 character

Fail modefail_modestringdefault skip

What a request that did not pass through an AI proxy meets here: skip and warn let it through, error answers 500.

one of error, skip, warn

Advanced

Maximum cached bodymax_cache_body_sizeintegerdefault 1048576

The largest response body in bytes that is cached, where a larger answer is served but not stored.

at least 0

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.