ai-rag

Adds retrieved documents to the prompt from a vector search.

Category AIAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin ai-ragPriority 1060

What it does

ai-rag adds documents from your own search index to a prompt before the model sees it. The question is turned into an embedding by the service under Embeddings provider, the index under Vector search provider is searched with it, and what the search answers is appended to the prompt as a user message. ai-proxy on the same route then sends the prompt to the model.

The client asks for the search in its body. Beside messages it sends an ai_rag object, with the question under embeddings.input and the vector field of the index under vector_search.fields. A request without that object is answered 400, so the route serves only clients written for it.

When to use it

• A support assistant on the orders API whose answers about shipping and returns should come from your own product documents, not from what the model knows.
• Product documents that change every week, kept in a search index that the assistant reads at the moment a customer asks.
• An index in Azure AI Search whose vectors were made by an Azure OpenAI deployment, which every chat request should consult first.
• A chat client your team writes, which can send the question and the vector field beside the messages of every request.

Not for sending the prompt to a model and its answer back to the client — see ai-proxy (Proxies requests to an LLM provider with a unified API), replacing the request body with what a model makes of it — see ai-request-rewrite (Rewrites the request body with an LLM before proxying) or refusing prompts that match a pattern — see ai-prompt-guard (Allows or denies prompts that match configured patterns).

Set it up

Add search results to a prompt that asks for them in an ai_rag object

Each request carries an ai_rag object beside its messages, with the question under `embeddings.input` and the vector field under `vector_search.fields`, and one without it is refused with 400. Both addresses are used as written, so they carry the api-version the service answers.

Fill Embeddings provider as in the JSON below and fill Vector search provider as in the JSON below.

ai-rag · as the console saves it

{
  "embeddings_provider": {
    "azure_openai": {
      "api_key": "<secret>",
      "endpoint": "https://orders-ai.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2024-02-01"
    }
  },
  "vector_search_provider": {
    "azure_ai_search": {
      "api_key": "<secret>",
      "endpoint": "https://orders-search.search.windows.net/indexes/product-docs/docs/search?api-version=2024-07-01"
    }
  }
}

Things to know

• Vector search provider — In the example the search answers Orders ship within two working days, and the model receives that sentence together with the client's question.
• Embeddings provider — Only a request body carrying ai_rag, with an embeddings input and vector search fields, is served, and any other is refused with 400. For example: "ai_rag": {"embeddings": {"input": "the question"}, "vector_search": {"fields": "contentVector"}}.
• Embeddings provider — Azure OpenAI is the one embeddings provider, and a configuration that names openai instead is refused when you save.
• Endpoint — Both addresses, this one and the search's, are used as written, so each carries the api-version query parameter its service answers to.
• AdviceKeep the route behind an authentication plugin, for example key-auth, so that only your own clients reach the index and the model.

Settings

The fields as the console shows them, section by section.

Embeddings providerembeddings_providerobjectRequired

up to 1 key

Azure OpenAIembeddings_provider.azure_openaiobjectRequired

API keyembeddings_provider.azure_openai.api_keystringRequired

Stored on the engine and shown only to roles that manage this object.

Endpointembeddings_provider.azure_openai.endpointstringRequired

The full embeddings URL of the deployment, used exactly as written.

Vector search providervector_search_providerobjectRequired

What the search answers with is appended to the prompt as a user message.

up to 1 key

Azure AI Searchvector_search_provider.azure_ai_searchobjectRequired

API keyvector_search_provider.azure_ai_search.api_keystringRequired

Stored on the engine and shown only to roles that manage this object.

Endpointvector_search_provider.azure_ai_search.endpointstringRequired

The full search URL of the index, used exactly as written.

SSL verifyssl_verifybooleandefault true

Checks the certificates of both the embeddings and the search service, and turning it off is logged as a security risk.

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.