Skip to main content

Send request records to Kafka

kafka-logger writes a record of every request that reaches a route to a Kafka topic, in batches. This guide sends the records of orders-api to the topic gateway-access. G8 already keeps request records of each environment in its log store, where Traffic Explorer reads them, as Where request records go describes. Use this plugin when the records have to reach a platform the rest of the company reads from Kafka. At the end, every request orders-api takes leaves a record in the topic gateway-access, a few seconds after its answer.

Before you start​

  • This guide starts from orders-api as Route requests to a backend leaves it, with key-auth from Require an API key. The request below calls /orders/42. A plugin from another guide that this page does not name can change what is recorded; remove it first or work on a copy of the route.
  • From the team that runs Kafka: the brokers' host names and ports, the topic the records go to, and whether the brokers ask for a user and password or for an encrypted connection. This guide uses kafka-1.internal and kafka-2.internal on port 9092 and the topic gateway-access.
  • Brokers the gateway nodes can reach under those names. Each gateway node collects its records and sends them to the brokers itself.
  • The host name of your gateway. In the console it is the first part of the Address line on a route's Overview tab, api.example.com in this page's examples; replace it in every request below.
  • The Editor or Environment administrator role in this environment.

Steps​

1. Choose kafka-logger​

Open orders-api under Routes, select its Plugins tab, then Add plugin at the top right of the tab. Open the Observability category, or type kafka-logger in the search box. Select kafka-logger, then Configure.

2. Name the brokers and the topic​

Under Brokers, select Add entry once for each broker and fill in its Host and Port. Set Kafka topic to gateway-access. The topic and at least one broker are required.

HostPort
kafka-1.internal9092
kafka-2.internal9092

3. Add the user and password if the brokers ask for them​

SASL config sits in each broker's row under Brokers. When the brokers take connections without a user, leave it empty in every row. When they ask for one, fill it in every row. Set Mechanism to the one the Kafka team names; PLAIN is the default. Then fill in User and Password. A user without a password is refused when you save.

When the brokers ask for an encrypted connection, turn on Verify under TLS in the Behavior section. That one switch turns TLS on and checks the brokers' certificate. Beside it, the form says: On, the gateway connects to the brokers over TLS and checks their certificate; a certificate it does not trust stops every record.

4. Decide what each record carries​

To choose the fields of each record, fill in Log format under Behavior. It is a JSON box. Each name you write becomes a field of the record, and its value names the request variable that fills it. It replaces the default record with the fields you name. For the client address, the route, the status and the time, type this:

{"client": "$remote_addr", "route": "$route_id", "status": "$status", "at": "$time_iso8601"}

Leave the box empty to keep the default record. The same format can be set once for every kafka-logger in the environment, under the plugin's Environment-wide settings on its page under Plugins. The reference page shows one under More scenarios.

5. Leave the rest at its defaults and add the plugin​

The Advanced section holds the batch and producer settings. Leave them as they are unless the Kafka team asks for other values. Select Add plugin at the bottom of the panel. The plugin is saved like this:

The Add kafka-logger panel with the two brokers and the topic filled in.
kafka-logger · as the console saves it
{
  "brokers": [
    {
      "host": "kafka-1.internal",
      "port": 9092
    },
    {
      "host": "kafka-2.internal",
      "port": 9092
    }
  ],
  "kafka_topic": "gateway-access",
  "required_acks": 1
}

The example also carries required_acks at 1. In your form, Required acks shows 1 (default), and the console does not save a default. Your object therefore carries no required_acks, and the gateway uses 1 all the same.

Check it​

Send a request to the route with the key of mobile-app in the x-api-key header, then read gateway-access with the Kafka team's tools.

curl -i 'https://<gateway host>/orders/42' -H 'x-api-key: <the key of mobile-app>'

With the defaults, a batch that holds a single request goes out once nothing new has arrived for Inactive timeout, 5 seconds. An empty topic right after your request does not yet mean the record is lost. The client's answer does not change, and what reaches the topic is read on the Kafka side.

What to know​

  • A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
  • Every kafka-logger with the same Cluster name sends through one set of brokers, the first the gateway connected to. When another kafka-logger in the environment sends to other brokers, give this one a cluster name of its own.
  • Name the brokers either under Brokers or under the older Broker list, never both and never neither.
  • For every request of the environment, put kafka-logger on a global rule, as Apply a plugin to every request shows.
G8 dev