prometheus

Exposes request metrics for Prometheus.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin prometheusPriority 500

What it does

prometheus counts the requests of the route or other object it is added to, and their response times, and the gateway exposes these counts for Prometheus to collect. Analytics reads the request counts of an environment from that Prometheus. On a global rule one copy counts every route of the environment.

Each count is labelled with the route and the service it went through, by id unless Prefer name is on. Analytics reads those labels by id, so a route counted under its name reads empty there. The labels a count carries are set for the whole environment in the plugin metadata, under Disabled labels.

When to use it

Every route of an environment should show its requests and response times in Analytics: one copy on a global rule counts them all.
A route that no global rule counts, the orders API for example, should still appear in Analytics with its own copy.
Prometheus holds more series than anyone reads because each route's counts are split by every path and host it matched, and those two labels should go.
The operations team keeps its own Grafana dashboard on the same Prometheus.

Not for a record of each request, sent to an HTTP endpoint — see http-logger (Sends request logs to an HTTP endpoint in batches), a record of each request, kept in Elasticsearch — see elasticsearch-logger (Sends request logs to an Elasticsearch index) or following one request through the services it reaches — see zipkin (Sends traces to a Zipkin collector).

Set it up

Count requests under the route and service ids, as Analytics reads them

Analytics reads the counts by route and service id, so it needs Prefer name off in the copy that runs for a route; on, the labels carry the names and the route's counts read empty in Analytics.

Turn off Prefer name.

prometheus · as the console saves it
{
  "prefer_name": false
}

More scenarios

Drop the labels that split request counts into too many seriesenvironment-wide settings

Analytics reads the route, service, consumer, node and response_source labels, and a label dropped here leaves the counts it keys empty there; these two it does not read.

Fill Disabled labels as in the JSON below.

prometheus environment-wide settings · as the console saves it
{
  "disabled_labels": {
    "http_status": [
      "matched_uri",
      "matched_host"
    ]
  }
}

Things to know

Prefer name — Off, a request to the route orders-api of the service orders is counted under the route's id and the service's id, not under these names. The count also carries the matched path, the host, the consumer and the node, for example matched_uri, matched_host, consumer and node.
Prefer name — On, the counts of each route and its service are labelled by name, and Analytics, which reads them by id, shows those routes and services as empty.
Disabled labels — Dropping matched_uri and matched_host from http_status leaves both labels on each count with an empty value, while route keeps the route's id.
Disabled labels — Analytics reads request counts by route, service, consumer, node and response_source from http_status, and response times from http_latency. A label dropped from either empties the counts or times Analytics reads by it.
Disabled labels — Where a metric has a label that tells its own series apart, code, type or layer, that label cannot be dropped and its list leaves it out.
Disabled labels — A label that the metric does not carry is refused when you save, for example route on llm_latency.
On a global rule, the plugin steps aside on every route that already runs its own copy, so the settings of the route's copy decide how that route is counted.
A route that no copy of the plugin runs on is not counted, and Analytics says so on the route.
AdviceRun one copy from a global rule with Prefer name off, and add a copy to a route only when that route needs settings of its own.

Settings

The fields as the console shows them, section by section.

Prefer nameprefer_namebooleandefault false

On, the route and service labels carry the object's name, and its id where it has none.

Environment-wide settings

Set once per environment, under Plugins › prometheus. Every prometheus in the environment uses these.

Disabled labelsdisabled_labelsobject

A label named here stays on the metric with an empty value, which opens a series of its own beside the old ones. Analytics reads request counts by route, service, consumer, node and response_source from http_status, and response times by type, route and service from http_latency; a label dropped from either empties the counts or times by that label, and without response_source the status classes no longer say who answered: the backend, no node, or the gateway.

AI cache bypasses totaldisabled_labels.ai_cache_bypasses_totalarray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, route_id, service, service_id

AI cache embedding latencydisabled_labels.ai_cache_embedding_latencyarray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, route_id, service, service_id

AI cache hits totaldisabled_labels.ai_cache_hits_totalarray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, route_id, service, service_id

AI cache misses totaldisabled_labels.ai_cache_misses_totalarray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, route_id, service, service_id

Bandwidthdisabled_labels.bandwidtharray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, service

HTTP latencydisabled_labels.http_latencyarray of strings

one of consumer, llm_model, node, request_llm_model, request_type, route, service

HTTP statusdisabled_labels.http_statusarray of strings

one of consumer, llm_model, matched_host, matched_uri, node, request_llm_model, request_type, response_source, route, service

LLM active connectionsdisabled_labels.llm_active_connectionsarray of strings

one of consumer, llm_model, matched_host, matched_uri, node, request_llm_model, request_type, route, route_id, service, service_id

LLM completion tokensdisabled_labels.llm_completion_tokensarray of strings