Log store on OpenSearch — accounts, index template and what differs

The five installation guides run an Elasticsearch beside the gateway. This one is for an environment whose request records go to an OpenSearch that already exists: the two accounts, the index template, and what differs from Elasticsearch. Installing OpenSearch itself is not covered.

Where this guide was run

The commands were run on 2026-10-02 against OpenSearch 2.18.0 with its security plugin on, on a development stack, with the outputs shown, and the steps of section 6 were walked in the console there. The certificate warning of that section was measured against the same version with TLS on. On 2026-10-03 the lab's dev and prod stores were set up on OpenSearch 3.9.0 with the accounts and the template of this guide's 2026-10-02 version; the bulk permission of the gateway's role was measured there. The ingest pipeline and the role line that reads it were run on 2026-10-04 on the development stack, and that a batch sent twice is stored once was measured on OpenSearch 3.9.0 in the lab.

1. What this covers

part	where it comes from
The gateway and the rule of its metrics	one of the five installation guides, up to its section on wiring APISIX
The two accounts of the log store	section 3
The ingest pipeline and the index template of request records	section 4
Turning on the request records	section 6, in the g8 console

g8 runs the retention: once a day it deletes, by its name, every daily index older than the days the environment keeps. The store carries no policy for these indices.

2. Prerequisites

requirement	value	note
OpenSearch	2.18.0, security plugin on	The version the commands were run against
An administrator of the store	`admin` in the commands	It opens roles and users and applies an index template
The CA that signed the store's certificate	`/etc/g8/opensearch-ca.pem`	The commands verify the certificate with it; g8 takes the same file under CA certificates
The store's address	`https://opensearch.example:9200` in the commands	Replace it with the address of your store
The slug of the environment in g8	`dev` in the commands	The daily indices, the roles, the users and the file of passwords carry it
The shell	bash	The commands and the script are written for it

The administrator's password is typed once and kept in a file that only root reads, one file for each environment; it reaches neither this document nor the repository. Run again, the block leaves a file that carries the line as it is:

sudo install -d -m 0700 /etc/g8
sudo test -e /etc/g8/os-dev.env || sudo install -m 0600 /dev/null /etc/g8/os-dev.env
read -r -s -p 'Password of the OpenSearch administrator: ' OSP
sudo grep -q '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env || \
  printf 'G8_OS_ADMIN_PASSWORD=%s\n' "$OSP" | sudo tee -a /etc/g8/os-dev.env >/dev/null
unset OSP

3. Accounts

Two accounts are opened in OpenSearch, and neither of them is the administrator:

account	used by	may
`g8-dev-gateway`	the gateway, to deliver the request records	write records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster
`g8-dev-server`	the g8 server, to show the records and to clean them up	read the state of the cluster, the index template and the ingest pipeline, read the daily indices of this environment and delete a day by its name

The daily indices carry the slug of the environment in g8 (g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is registered under in the g8 console. The roles name the days by a regular expression and not by g8-dev-access-*: a wildcard would also reach the days of an environment whose slug is dev-access or begins with dev-access-.

sudo tee /etc/g8/store-accounts.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
ENV=dev
OS=https://opensearch.example:9200
CA=/etc/g8/opensearch-ca.pem
for who in GATEWAY SERVER; do
  grep -q "^G8_OS_${who}_PASSWORD=" /etc/g8/os-$ENV.env || \
    printf 'G8_OS_%s_PASSWORD=%s\n' "$who" "$(openssl rand -base64 24 | tr -d /+= | head -c 24)-Aa1" >> /etc/g8/os-$ENV.env
done
OSP=$(grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-$ENV.env | cut -d= -f2-)
GWP=$(grep '^G8_OS_GATEWAY_PASSWORD=' /etc/g8/os-$ENV.env | cut -d= -f2-)
SVP=$(grep '^G8_OS_SERVER_PASSWORD=' /etc/g8/os-$ENV.env | cut -d= -f2-)
put() {
  body=$(cat)
  for try in 1 2 3 4 5; do
    code=$(curl -s --cacert "$CA" -o /dev/null -w '%{http_code}' -u "admin:$OSP" -X PUT "$OS$1" \
      -H 'Content-Type: application/json' --data-binary "$body")
    [ "$code" = 409 ] || break
    sleep 1
  done
  echo "$1 -> $code"
  case "$code" in 200|201) ;; *) exit 1 ;; esac
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_plugins/_security/api/roles/g8-$ENV-gateway <<EOF
{"cluster_permissions":["indices:data/write/bulk"],
 "index_permissions":[{"index_patterns":["$DAYS"],"allowed_actions":["index","create_index"]}]}
EOF
put /_plugins/_security/api/roles/g8-$ENV-server <<EOF
{"cluster_permissions":["cluster_monitor","indices:admin/index_template/get","cluster:admin/ingest/pipeline/get"],
 "index_permissions":[{"index_patterns":["$DAYS"],"allowed_actions":["read",
 "indices:admin/mappings/get","indices:monitor/settings/get","indices:monitor/stats","indices:admin/delete"]}]}
EOF
put /_plugins/_security/api/internalusers/g8-$ENV-gateway <<EOF
{"password":"$GWP","opendistro_security_roles":["g8-$ENV-gateway"]}
EOF
put /_plugins/_security/api/internalusers/g8-$ENV-server <<EOF
{"password":"$SVP","opendistro_security_roles":["g8-$ENV-server"]}
EOF
SCRIPT
sudo chmod 700 /etc/g8/store-accounts.sh
sudo /etc/g8/store-accounts.sh

/_plugins/_security/api/roles/g8-dev-gateway -> 201
/_plugins/_security/api/roles/g8-dev-server -> 201
/_plugins/_security/api/internalusers/g8-dev-gateway -> 201
/_plugins/_security/api/internalusers/g8-dev-server -> 201

The script can be run again: it keeps the passwords it finds in /etc/g8/os-dev.env and answers 200 for what exists. OpenSearch answers 409 to a write that meets another change of its security configuration; the script tries that write five times, and it stops at any answer other than 200 or 201.

The gateway's account must not be able to read

The password of g8-dev-gateway is part of the gateway's configuration: whoever can read the global rules of the gateway can obtain it. That is why this account can only write records. When traffic logging is turned on with a plain password, g8 tries the account and refuses one that can read the cluster or a record; a password given as a secret reference is resolved on the gateway and is not tried.

dev and dev-access-x must not share a store

OpenSearch refuses a listing by wildcard as a whole when it matches an index the account may not read. If this store also holds the days of an environment whose slug is dev-access or begins with dev-access-, g8's listing of the days of dev is refused with 403: Test connection and Settings show the log store as failed, and the daily cleanup deletes nothing, so the days of dev pile up. Elasticsearch narrows the same listing to what the account may read. g8 itself refuses to create such a pair of environments; the warning is for a store that another installation also uses.

4. Ingest pipeline and index template

The ingest pipeline stores each record under its request id. The gateway's logger sends the records in batches and sends a batch again when the store answers it late, and by then the store has usually written the first one: under the request id, the second copy replaces the first instead of adding a record. The index template below sends every record through this pipeline, so the pipeline is applied first. A store that holds the template and not the pipeline refuses every record while the gateway's bulk request still answers 200; g8 checks that the store holds this version of the pipeline and does not turn traffic logging on without it. The store needs at least one node with the ingest role, which a node carries by default.

OSP=$(sudo grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
curl -s --cacert /etc/g8/opensearch-ca.pem -u "admin:$OSP" -X PUT https://opensearch.example:9200/_ingest/pipeline/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
  "version": 1,
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{request_id}}}",
        "if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
      }
    }
  ]
}
EOF

{"acknowledged":true}

The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.

OSP=$(sudo grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
curl -s --cacert /etc/g8/opensearch-ca.pem -u "admin:$OSP" -X PUT https://opensearch.example:9200/_index_template/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "index_patterns": [
    "g8-*-access-*"
  ],
  "priority": 200,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "codec": "best_compression",
        "auto_expand_replicas": "0-1",
        "default_pipeline": "g8-access",
        "analysis": {
          "analyzer": {
            "g8_nodes": {
              "type": "pattern",
              "pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
              "lowercase": false
            }
          }
        }
      }
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "latency": {
          "type": "float"
        },
        "upstream_latency": {
          "type": "float"
        },
        "apisix_latency": {
          "type": "float"
        },
        "client_ip": {
          "type": "ip",
          "ignore_malformed": true
        },
        "route_id": {
          "type": "keyword"
        },
        "service_id": {
          "type": "keyword"
        },
        "upstream": {
          "type": "keyword",
          "ignore_above": 1024,
          "fields": {
            "node": {
              "type": "text",
              "analyzer": "g8_nodes",
              "norms": false,
              "index_options": "docs"
            }
          }
        },
        "consumer": {
          "properties": {
            "username": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "properties": {
            "method": {
              "type": "keyword"
            },
            "uri": {
              "type": "wildcard"
            },
            "url": {
              "type": "keyword",
              "index": false,
              "doc_values": false
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "querystring": {
              "type": "object",
              "enabled": false
            }
          }
        },
        "request_body": {
          "type": "text",
          "index": false
        },
        "response": {
          "properties": {
            "status": {
              "type": "short"
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "body": {
              "type": "text",
              "index": false
            }
          }
        },
        "server": {
          "properties": {
            "hostname": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "request_id": {
          "type": "keyword"
        },
        "route_name": {
          "type": "keyword"
        },
        "service_name": {
          "type": "keyword"
        },
        "upstream_status": {
          "type": "keyword"
        },
        "x_request_id": {
          "type": "keyword"
        },
        "logged_at": {
          "type": "date",
          "format": "epoch_second"
        }
      },
      "_meta": {
        "g8_records": 2
      },
      "_source": {
        "excludes": [
          "response.headers.set-cookie"
        ]
      }
    }
  }
}
EOF

{"acknowledged":true}

A store that already holds the template of an earlier g8 is moved in this order: run the whole block of section 3 again, which rewrites the saved script with the new server role and runs it, keeping the passwords (the saved script alone writes the old role); apply the pipeline and then the template above; and send the day of today, born under the earlier template, through the pipeline at once, once for each environment's slug the store serves. A store that holds no record of today yet answers 404 index_not_found_exception to that call, and nothing more is needed. Until then Settings reads Index template Outdated and Ingest pipeline Cannot be read; the records keep arriving and the settings can still be saved, and g8 refuses only to turn logging on anew.

OSP=$(sudo grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
curl -s --cacert /etc/g8/opensearch-ca.pem -u "admin:$OSP" -X PUT "https://opensearch.example:9200/g8-dev-access-$(date -u +%Y.%m.%d)/_settings" \
  -H 'Content-Type: application/json' -d '{"index.default_pipeline":"g8-access"}'

{"acknowledged":true}

5. What the accounts may do

What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:

OS=https://opensearch.example:9200
GWP=$(sudo grep '^G8_OS_GATEWAY_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
SVP=$(sudo grep '^G8_OS_SERVER_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
ask() { curl -s --cacert /etc/g8/opensearch-ca.pem -o /dev/null -w "$1 -> %{http_code}\n" -u "$2" "${@:3}"; }
ask 'g8-dev-gateway reads the cluster     ' "g8-dev-gateway:$GWP" "$OS/"
ask 'g8-dev-gateway writes a day          ' "g8-dev-gateway:$GWP" -X POST "$OS/g8-dev-access-2000.01.01/_doc" -H 'Content-Type: application/json' -d '{}'
ask 'g8-dev-gateway reads the day         ' "g8-dev-gateway:$GWP" "$OS/g8-dev-access-2000.01.01/_search"
ask 'g8-dev-gateway writes beside its days' "g8-dev-gateway:$GWP" -X POST "$OS/g8-dev-access-x-access-2000.01.01/_doc" -H 'Content-Type: application/json' -d '{}'
ask 'g8-dev-server reads the template     ' "g8-dev-server:$SVP" "$OS/_index_template/g8-access"
ask 'g8-dev-server reads the pipeline     ' "g8-dev-server:$SVP" "$OS/_ingest/pipeline/g8-access"
ask 'g8-dev-server reads the day          ' "g8-dev-server:$SVP" "$OS/g8-dev-access-2000.01.01/_search"
ask 'g8-dev-server reads beside its days  ' "g8-dev-server:$SVP" "$OS/g8-dev-access-x-access-2000.01.01/_search"
ask 'g8-dev-server writes a record        ' "g8-dev-server:$SVP" -X POST "$OS/g8-dev-access-2000.01.01/_doc" -H 'Content-Type: application/json' -d '{}'
ask 'g8-dev-server deletes the day        ' "g8-dev-server:$SVP" -X DELETE "$OS/g8-dev-access-2000.01.01"

g8-dev-gateway reads the cluster      -> 403
g8-dev-gateway writes a day           -> 201
g8-dev-gateway reads the day          -> 403
g8-dev-gateway writes beside its days -> 403
g8-dev-server reads the template      -> 200
g8-dev-server reads the pipeline      -> 200
g8-dev-server reads the day           -> 200
g8-dev-server reads beside its days   -> 403
g8-dev-server writes a record         -> 403
g8-dev-server deletes the day         -> 200

6. Turning on the request records

The request records are turned on from the g8 console, once the gateway is registered there as the environment dev.

Environments › dev › Edit. Under Observability, Elasticsearch URL: the address of the store; Username g8-dev-server, password the value of G8_OS_SERVER_PASSWORD in /etc/g8/os-dev.env. Under Engine, add the store's CA to CA certificates and keep what is already there: the one bundle serves every HTTPS connection of the environment. Test connection must answer a sentence that begins Connected · OpenSearch 2.18.0; before the first record it ends no indices g8-dev-access-* yet and is drawn as a warning. Then save.
Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Version OpenSearch 2.18.0, Index template Applied and Ingest pipeline Applied.
Turn on traffic logging. Log store address, as the gateway reaches it: the address of the store. Username g8-dev-gateway, password the value of G8_OS_GATEWAY_PASSWORD. Turn on.

The gateway verifies the store's certificate

The rule g8 writes leaves the logger's ssl_verify at its default, which is on. With a certificate the gateway does not trust, no record arrives: the logger rule reads Enabled, Last record keeps saying No record today or yesterday, and the gateway's error log says:

failed to get Elasticsearch version: 21: unable to verify the first certificate

A line that ends otherwise is not this fault. With the account of section 3 the log also carries the line below while the records arrive: the logger asks the store for its version, and this account may not read it.

failed to get Elasticsearch version: server returned status: 403

Measured way out: switch ssl_verify off on the logger under Global rules › g8-request-records. The rule stays Enabled and g8 keeps the setting when the Settings are saved. Making the gateway trust the CA instead was not measured.

7. What differs from Elasticsearch

	Elasticsearch	OpenSearch
Reading the index template	the cluster privilege `monitor` covers it	`indices:admin/index_template/get` is needed beside `cluster_monitor`
Reading the ingest pipeline	the cluster privilege `read_pipeline`	`cluster:admin/ingest/pipeline/get` is needed beside `cluster_monitor`
An account that is refused at sign-in	a JSON `security_exception`	the plain text `Unauthorized`; g8 reads both
A listing by wildcard that matches an index the account may not read	narrowed to what the account may read	refused as a whole (403)
Deleting by wildcard	refused by default	accepted by default; g8 deletes a day by its name and never by a pattern
The plugin that delivers the records	`elasticsearch-logger`	the same plugin; it writes to OpenSearch unchanged

8. Removing

In the g8 console, delete the global rule g8-request-records under Global rules: the gateway stops writing. g8 does not remove this rule when the environment is deleted.
Delete the environment in the g8 console, or clear its log store there: g8 connects to the store once a day to delete the days past the retention.
The days that remain are deleted by their names; nobody deletes them afterwards:

OS=https://opensearch.example:9200
OSP=$(sudo grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
curl -s --cacert /etc/g8/opensearch-ca.pem -u "admin:$OSP" "$OS/_cat/indices/g8-dev-access-*?h=index" \
  | grep -E '^g8-dev-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}$' | sort | while read -r day; do
  curl -s --cacert /etc/g8/opensearch-ca.pem -o /dev/null -w "$day -> %{http_code}\n" -u "admin:$OSP" -X DELETE "$OS/$day"
done

g8-dev-access-2000.01.01 -> 200
g8-dev-access-2000.01.02 -> 200

Then the two accounts go; the index template stays, other environments of the store use it.

OS=https://opensearch.example:9200
OSP=$(sudo grep '^G8_OS_ADMIN_PASSWORD=' /etc/g8/os-dev.env | cut -d= -f2-)
for one in internalusers/g8-dev-gateway internalusers/g8-dev-server roles/g8-dev-gateway roles/g8-dev-server; do
  curl -s --cacert /etc/g8/opensearch-ca.pem -o /dev/null -w "$one -> %{http_code}\n" -u "admin:$OSP" \
    -X DELETE "$OS/_plugins/_security/api/$one"
done
sudo rm -f /etc/g8/os-dev.env /etc/g8/store-accounts.sh

internalusers/g8-dev-gateway -> 200
internalusers/g8-dev-server -> 200
roles/g8-dev-gateway -> 200
roles/g8-dev-server -> 200

9. Scope limits

limit	where it stands
Installing and sizing OpenSearch	not covered
The gateway trusting the store's CA	not measured; see the warning in section 6
Versions other than 2.18.0	not measured
Reading and deleting large volumes	not measured on OpenSearch

G8 dev · Apache APISIX® 3.19.0

Apache, Apache APISIX and APISIX are trademarks of The Apache Software Foundation.

1. What this covers​

2. Prerequisites​

3. Accounts​

4. Ingest pipeline and index template​

5. What the accounts may do​

6. Turning on the request records​

7. What differs from Elasticsearch​

8. Removing​

9. Scope limits​