rocketmq-logger

Sends request logs to a RocketMQ topic.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin rocketmq-loggerPriority 402

What it does

rocketmq-logger sends the records of the requests that reach the route or other object it is added to, to a RocketMQ topic. It needs the name servers under Nameserver list and the Topic; the records are collected on the gateway node and sent in batches, so one message can carry several records.

A Log format of your own, here or in the plugin metadata, replaces the default record with the fields you name, and Log format extra adds fields to the default record only. Under the origin Meta format neither is read, and the request itself is sent as raw text.

When to use it

• Request records have to reach the platform the rest of the company already reads from RocketMQ, without a collector in between.
• The records of the orders API should carry a Tag of their own, orders for example, so a reader of the topic can pick them out.
• Every route should write the same few fields, the client address, the method, the path, the status and the time, set once in the plugin metadata.
• The topic accepts only messages signed with an access key and a secret key.

Not for a Kafka topic — see kafka-logger (Sends request logs to a Kafka topic), a plain HTTP collector or a webhook — see http-logger (Sends request logs to an HTTP endpoint in batches) or a syslog server — see syslog (Sends request logs to a syslog server).

Set it up

Write a record of each request to a RocketMQ topic

Under Basics fill Nameserver list as in the JSON below and set Topic to gateway-access. Under Behavior set Tag to orders.

rocketmq-logger · as the console saves it

{
  "nameserver_list": [
    "rocketmq-1.internal:9876",
    "rocketmq-2.internal:9876"
  ],
  "topic": "gateway-access",
  "tag": "orders"
}

More scenarios

Shape the record every route writes to RocketMQenvironment-wide settings

Fill Log format as in the JSON below.

rocketmq-logger environment-wide settings · as the console saves it

{
  "log_format": {
    "client": "$remote_addr",
    "method": "$request_method",
    "path": "$uri",
    "status": "$status",
    "at": "$time_iso8601"
  }
}

Things to know

• Nameserver list — Each entry is a host and port written as text, rocketmq-1.internal:9876 for example. A bare port number, 9876 alone, is refused when you save.
• Secret key — Read only next to an access key: on its own it is sealed and stored, and the messages travel unsigned.
• Use TLS — The broker's certificate is never checked, and no server name is sent in the handshake.
• Meta format — Only default and origin are accepted; raw is refused when you save.
• Log format — Under the origin meta format the request is sent as raw text and no log format is read.
• Log format — A format is a set of named fields; a single variable written as text, $time_iso8601 for example, is refused when you save.
• Include request body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Include response body — Under the origin meta format the response body is still buffered and then written nowhere.
• Include response body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• In the plugin metadata, while more entries than the Pending entry limit wait to be sent, new ones are dropped rather than queued.
• AdviceTurn on the request and response bodies only on routes whose topic is meant to hold what clients and the backend send.
• Log format extra — Under the origin meta format the request is sent as raw text and no log format is read.
• Response body limit — Under the origin meta format the response body is still buffered and then written nowhere.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.

Settings

The fields as the console shows them, section by section.

Basics

Nameserver listnameserver_listarray of stringsRequired

at least 1 item

TopictopicstringRequired

Access keyaccess_keystringdefault

Signs the messages together with Secret key. A secret key without it is not used.

Secret keysecret_keystringdefault

Stored on the engine and shown only to roles that manage this object.

Behavior

Keykeystring

Tagtagstring

Meta formatmeta_formatstringdefault default

With origin the request line, its headers and the request body are sent as raw text, and the log format and response body rows below are not read.

one of default, origin

Use TLSuse_tlsbooleandefault false

Timeouttimeoutintegerdefault 3

at least 1 · in seconds

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Include request bodyinclude_req_bodybooleandefault false

Include request body wheninclude_req_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Include response bodyinclude_resp_bodybooleandefault false

Include response body wheninclude_resp_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Request body limitmax_req_body_bytesintegerdefault 524288

at least 1

Response body limitmax_resp_body_bytesintegerdefault 524288

at least 1

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Namenamestringdefault rocketmq logger

Environment-wide settings

Set once per environment, under Plugins › rocketmq-logger. Every rocketmq-logger in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.