Skip to main content

Send request records to Elasticsearch

elasticsearch-logger writes a record of every request that reaches a route to an Elasticsearch index, in batches. This guide sends the records of orders-api to the index gateway-access on an Elasticsearch your own team runs. At the end, every request orders-api takes leaves a record in that index, a few seconds after its answer.

G8 already keeps request records of its own for each environment. They go to the environment's log store, and Traffic Explorer reads them. For that, G8 adds a copy of elasticsearch-logger to the global rule g8-request-records when request records are turned on, as Where request records go describes. The copy you add here is separate from that one. Use it when your team searches the records of a route with its own tools, in an Elasticsearch of its own.

Before you start​

  • This guide starts from orders-api as Route requests to a backend leaves it, with key-auth from Require an API key. The request of the first check calls /orders/42/invoice, and the request of the second check calls /orders/43/invoice. A plugin from another guide that this page does not name can change what is recorded; remove it first or work on a copy of the route.
  • From the team that runs Elasticsearch: the address and port of the cluster, the index the records go to, and a user and password with the right to write that index. This guide uses https://elasticsearch.example.com:9200, the index gateway-access and the user gateway.
  • An Elasticsearch the gateway nodes can reach at that address.
  • The host name of your gateway. In the console it is the first part of the Address line on a route's Overview tab, api.example.com in this page's examples; replace it in every request below.
  • The Editor or Environment administrator role in this environment.

Steps​

1. Choose elasticsearch-logger​

Open orders-api under Routes, select its Plugins tab, then Add plugin at the top right of the tab. Open the Observability category, or type elasticsearch-logger in the search box. Select elasticsearch-logger, then Configure.

2. Name the address and the index​

Under Basics, keep One address chosen. Set Endpoint address to https://elasticsearch.example.com:9200, with no slash at the end. The bulk path is added to this address, which is why one that ends in a slash is refused.

Set the Index field under Field to gateway-access. The address and the index are both required.

3. Add the user and password​

Under Auth, fill in Username with gateway and Password with the account's password. The two go together, and one without the other is refused when you save. Leave the block empty when the cluster takes records without a user.

Ask for an account that may write the indexes this copy names and nothing more.

4. Leave the rest at its defaults and add the plugin​

The Behavior section holds the format of each record and whether it carries the request and response bodies. The Advanced section holds the batch settings. Leave both as they are and select Add plugin at the bottom of the panel. The plugin is saved like this:

The Add elasticsearch-logger panel with one server address, the index name and the account filled in.
elasticsearch-logger · as the console saves it
{
  "endpoint_addr": "https://elasticsearch.example.com:9200",
  "field": {
    "index": "gateway-access"
  },
  "auth": {
    "username": "gateway",
    "password": "<secret>"
  }
}

Check it​

Send a request to the route. key-auth is on orders-api, so add the key of mobile-app in the x-api-key header to the request below.

curl
curl -i -L 'https://<gateway host>/orders/42/invoice'

The client's answer does not change. The gateway sends the record of the request to the address you named, as a POST to /_bulk. With the defaults, a batch that holds a single request goes out once nothing new has arrived for Inactive timeout, 5 seconds.

Wait a few seconds, then search the index gateway-access of your Elasticsearch for the path of the request. Use the address and an account of your own team:

curl -u '<your user>' -G 'https://<your Elasticsearch address>/gateway-access/_search' \
  --data-urlencode 'q="/orders/42/invoice"'

The hits hold a record of the request. When the index is still empty, wait for the batch and search again before you look for a fault.

Send the records to any node of a cluster​

When the cluster has several nodes that all take records, choose Several addresses under Basics instead. Type each node's address in Endpoint addresses and press Enter after each one. Every address must accept the same records: each batch goes to one of them at random.

The example below also writes each day's records to an index of their own, with gateway-access-{%Y.%m.%d} as the index. The name is built again for every request: braces hold a time format and a $name is a request variable. Each record goes to the index of its day, gateway-access- followed by the year, the month and the day.

elasticsearch-logger · as the console saves it
{
  "endpoint_addrs": [
    "https://10.0.6.31:9200",
    "https://10.0.6.32:9200",
    "https://10.0.6.33:9200"
  ],
  "field": {
    "index": "gateway-access-{%Y.%m.%d}"
  },
  "auth": {
    "username": "gateway",
    "password": "<secret>"
  }
}

To check it, send the request below with the same x-api-key header, then search the index of today, gateway-access- with today's date, the same way as above.

curl
curl -i -L 'https://<gateway host>/orders/43/invoice'

A configuration names one address or a list of them, never both.

Cap the backlog for every route​

Records wait on the gateway until their batch goes out. While more records wait than the Pending entry limit, 8192 by default, new records are dropped rather than queued.

Raise the limit when Elasticsearch falls behind for a while and you would rather keep more records waiting than lose them. The limit is an environment-wide setting, so it also applies to the copy G8 adds for its own request records.

To set it, open the plugin's page under Plugins. Under Environment-wide settings, set Pending entry limit to the number you choose and select Save settings. The example below raises it:

elasticsearch-logger environment-wide settings · as the console saves it
{
  "max_pending_entries": 50000
}

A dropped record is missing from the index, and nothing in the client's answer shows that it was dropped.

What to know​

  • A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
  • A copy on a route or a service may use a Log format of your own. Leave the Log format of the environment-wide settings empty, because a format set there replaces the record G8 reads and Traffic Explorer can no longer read this environment's records.
  • To record several routes the same way, put the plugin on a service they share, as Share plugins between routes shows.
G8 dev