Test environment — 1 APISIX + 1 etcd on a virtual machine

The host names, addresses and the registry on this page are examples. Our lab run used its own values in their place; replace them with yours, and take the registry address and its credentials from your G8 contact.

No Docker. APISIX and etcd are installed directly on the operating system and managed by systemd. This is the installation path for customer environments that cannot — or will not — run containers.

The customer machine needs neither a compiler nor internet access: we build APISIX from source ourselves and publish the result as a runtime tarball in our own Harbor. Installation is nothing more than extracting that tarball. How the tarball is produced is covered in section 20.

Single node

This document installs 1 APISIX + 1 etcd. There is no fault tolerance: if etcd stops, APISIX cannot receive new configuration. For a three-node installation see the prod document.

1. What we are installing

component	version	source
APISIX	3.19.0	harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04
etcd	v3.5.17	harbor.g8.dev/infra/etcd:v3.5.17

Both come from our own Harbor registry. The customer environment may be air-gapped; we leave no dependency on upstream (Docker Hub, GitHub Releases).

Extracting the tarball writes the following to the filesystem:

path	contents
/usr/local/apisix	APISIX Lua sources, Lua dependencies under deps/, conf/
/usr/local/openresty	APISIX's patched runtime (apisix-runtime 1.3.18)
/usr/local/bin/apisix	CLI wrapper
/usr/bin/openresty	copy so that command -v openresty succeeds

2. Prerequisites

requirement	value	note
Operating system	Ubuntu 24.04 LTS	The tarball was built against this release; libc compatibility on other releases is unverified
CPU / RAM / disk	4 vCPU / 8 GB / 100 GB	etcd is sensitive to fsync; SSD/NVMe recommended
Clock synchronisation	chrony or systemd-timesyncd active	etcd enters leader election on clock drift
Harbor access	harbor.g8.dev:443	Source of the runtime and etcd
DNS	harbor.g8.dev must resolve
Harbor CA file	g8-lab-ca.crt in the working directory	Shipped with the installation package; the CA step below copies this file
Internet access	archive.ubuntu.com	Required. APISIX and etcd come from Harbor, but the OS packages in 2.2 are installed from upstream. A genuinely air-gapped site needs a local apt mirror for that step. This document does not install Docker

2.1 Ports

port	component	who should reach it
9080	APISIX data plane	Clients
9180	APISIX Admin API	The g8 server only. Must not be exposed to the internet
9091	APISIX Prometheus	Prometheus only
9200	Elasticsearch	The g8 server and the edge only. Authenticated, but there is no TLS — the password crosses the network in the clear
9099	Prometheus	The g8 server and the edge only. No authentication. Not 9090: that port belongs to the APISIX control API
2379	etcd client	Binds to 127.0.0.1 only; not exposed
2380	etcd peer	Unused on a single node

2.2 Runtime packages

The tarball carries only APISIX's own files; it does not carry the distribution's shared libraries. APISIX will not run until these are installed.

sudo apt-get update
sudo apt-get install -y skopeo python3 curl \
  libpcre3 libyaml-0-2 libxml2 libxslt1.1 libicu74 liblzma5 \
  libpcre2-8-0 zlib1g libstdc++6 libcrypt1

Install the whole list

Which of these packages your image already carries varies by image; in an independent installation attempt both skopeo and libpcre3 were absent. libpcre3 is the most insidious: its absence does not surface during the first installation steps — it surfaces when apisix init reaches schema validation. Details: the libpcre3 trap.

3. Trust the Harbor CA

Harbor presents a certificate signed by our own CA. Nothing can be pulled until that CA is trusted — you get x509: certificate signed by unknown authority.

sudo cp g8-lab-ca.crt /usr/local/share/ca-certificates/g8-lab-ca.crt
sudo update-ca-certificates

Verification — you must see sslverify=0:

curl -s -o /dev/null -w "sslverify=%{ssl_verify_result} status=%{http_code}\n" \
  https://harbor.g8.dev/api/v2.0/systeminfo

sslverify=0 status=200

4. Pull and extract the runtime

The tarball sits in Harbor as a single-layer OCI artifact. skopeo fetches it from the registry; the layer is extracted directly under /.

sudo rm -rf /tmp/apisix-rt
skopeo copy docker://harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04 dir:/tmp/apisix-rt
LAYER=$(python3 -c "import json;print(json.load(open('/tmp/apisix-rt/manifest.json'))['layers'][0]['digest'].split(':')[1])")
sudo tar -xzf "/tmp/apisix-rt/$LAYER" -C /

Verification — the command being executed is printed before the version line; this is normal:

cd /usr/local/apisix && apisix version

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
3.19.0

cd /usr/local/apisix is not incidental

If the apisix wrapper sees ./apisix/cli/apisix.lua in the working directory, it adds deps/share/lua/5.1/?/init.lua to the Lua search path. Called from any other directory that entry is not added and dependencies such as lyaml cannot be found. Details: the WorkingDirectory trap.

5. Installing etcd

5.1 Binaries

The etcd image is in Harbor; three binaries are extracted from it. The binaries are spread across several layers, so every layer is scanned.

sudo rm -rf /tmp/etcd-img /tmp/etcd-x
mkdir -p /tmp/etcd-x
skopeo copy docker://harbor.g8.dev/infra/etcd:v3.5.17 dir:/tmp/etcd-img
for blob in /tmp/etcd-img/*; do
  tar -tzf "$blob" >/dev/null 2>&1 || continue
  tar -xzf "$blob" -C /tmp/etcd-x --wildcards \
    "usr/local/bin/etcd" "usr/local/bin/etcdctl" "usr/local/bin/etcdutl" 2>/dev/null || true
done
sudo install -m 0755 -o root -g root /tmp/etcd-x/usr/local/bin/etcd*  /usr/local/bin/

Verification:

etcd --version | head -1
etcdctl version | head -1
etcdutl version | head -1

etcd Version: 3.5.17
etcdctl version: 3.5.17
etcdutl version: 3.5.17

5.2 User and directories

etcd is not run as root. A system user is created and the data directory is given to it alone.

sudo useradd --system --home-dir /var/lib/etcd --shell /usr/sbin/nologin etcd
sudo mkdir -p /var/lib/etcd /etc/g8 /var/backups/g8-gw
sudo chown etcd:etcd /var/lib/etcd
sudo chmod 700 /var/lib/etcd

5.3 systemd unit

--auto-compaction-*, --quota-backend-bytes and --snapshot-count exist for long-running operation; the reasoning is in section 18.

sudo tee /etc/systemd/system/etcd.service >/dev/null <<'EOF'
[Unit]
Description=etcd - g8 gateway configuration store
Documentation=https://etcd.io/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=etcd
Group=etcd
ExecStart=/usr/local/bin/etcd \
  --name=etcd-test \
  --data-dir=/var/lib/etcd \
  --listen-client-urls=http://127.0.0.1:2379 \
  --advertise-client-urls=http://127.0.0.1:2379 \
  --listen-peer-urls=http://127.0.0.1:2380 \
  --initial-advertise-peer-urls=http://127.0.0.1:2380 \
  --initial-cluster=etcd-test=http://127.0.0.1:2380 \
  --initial-cluster-state=new \
  --initial-cluster-token=g8-test \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --quota-backend-bytes=2147483648 \
  --snapshot-count=10000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/etcd

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now etcd

ProtectSystem=strict and ReadWritePaths go together

ProtectSystem=strict makes the whole filesystem read-only. Remove the ReadWritePaths line and etcd cannot write to its data directory and dies at startup.

6. Admin key

APISIX's documented default admin key (edd1c9f0…) is public knowledge. Left unchanged, the Admin API is wide open with full privileges.

The key lives in a separate file and systemd passes it to the unit as an environment variable; it is never written by hand into config.yaml.

printf 'G8_APISIX_ADMIN_KEY=%s\n' "$(openssl rand -hex 24)" | sudo tee /etc/g8/apisix.env >/dev/null
sudo chmod 600 /etc/g8/apisix.env
sudo chown root:root /etc/g8/apisix.env

7. APISIX configuration

The ${{G8_APISIX_ADMIN_KEY}} syntax belongs to APISIX: it reads the value from the environment at startup. There is no plaintext secret in this file.

sudo tee /usr/local/apisix/conf/config.yaml >/dev/null <<'EOF'
apisix:
  node_listen:
    - 9080

deployment:
  role: traditional
  role_traditional:
    config_provider: etcd
  admin:
    admin_listen:
      ip: 0.0.0.0
      port: 9180
    admin_key:
      - name: admin
        key: "${{G8_APISIX_ADMIN_KEY}}"
        role: admin
    allow_admin:
      - 192.0.2.0/24
      - 127.0.0.0/8
  etcd:
    host:
      - "http://127.0.0.1:2379"
    prefix: /apisix
    timeout: 30

nginx_config:
  error_log_level: warn
  http:
    access_log: /usr/local/apisix/logs/access.log
    access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","status":"$status","method":"$request_method","uri":"$uri","rt":"$request_time","ua":"$http_user_agent"}'
    access_log_format_escape: json

plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0
      port: 9091
    metrics:
      http_status:
        extra_labels:
          - route_name: $route_name
          - service_name: $service_name
      http_latency:
        extra_labels:
          - route_name: $route_name
          - service_name: $service_name
EOF

The 192.0.2.0/24 entry in allow_admin is the lab network. Replace it with the network the g8 server sits on for a customer installation. 127.0.0.0/8 must stay; the verification commands run from the machine itself.

8. APISIX systemd unit

sudo tee /etc/systemd/system/apisix.service >/dev/null <<'EOF'
[Unit]
Description=Apache APISIX - g8 gateway data plane
Documentation=https://apisix.apache.org/docs/
After=network-online.target etcd.service
Wants=network-online.target etcd.service

[Service]
Type=forking
WorkingDirectory=/usr/local/apisix
PIDFile=/usr/local/apisix/logs/nginx.pid
EnvironmentFile=/etc/g8/apisix.env
ExecStartPre=/bin/rm -f /usr/local/apisix/logs/worker_events.sock
ExecStartPre=/usr/local/bin/apisix init
ExecStart=/usr/local/bin/apisix start
ExecStop=/usr/local/bin/apisix stop
ExecReload=/usr/local/bin/apisix reload
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload

Without LimitNOFILE, systemd grants 1024 and APISIX starts returning 5xx with too many open files under moderate load.

9. Log rotation

The equivalent of Docker's max-size/max-file here is logrotate. After rotation nginx is sent USR1; nginx reopens the log files and no line is lost.

sudo tee /etc/logrotate.d/apisix >/dev/null <<'EOF'
/usr/local/apisix/logs/*.log {
    daily
    maxsize 50M
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        [ -f /usr/local/apisix/logs/nginx.pid ] && kill -USR1 "$(cat /usr/local/apisix/logs/nginx.pid)"
    endscript
}
EOF

Syntax check. The output is searched with grep, not sliced with head: logrotate 3.21.0 prints a debug banner and state-file chatter first, pushing the lines you want to positions 11-12.

sudo logrotate -d /etc/logrotate.d/apisix 2>&1 | grep -A1 'rotating pattern'

rotating pattern: /usr/local/apisix/logs/*.log  after 1 days (14 rotations)
empty log files are not rotated, log files >= 52428800 are rotated earlier, old logs are removed

The scheduled rotation fails inside the sandbox

On Ubuntu 24.04 logrotate.service ships with ProtectSystem=full and no ReadWritePaths — everything under /usr is read-only inside the sandbox. Because the APISIX logs live under /usr/local/apisix/logs, the rotation the timer runs fails every night with:

error: error creating output file /usr/local/apisix/logs/access.log.1.gz: Read-only file system

Running sudo logrotate -f … by hand runs outside the sandbox and passes cleanly, so this defect is invisible to manual verification. A drop-in is required:

sudo mkdir -p /etc/systemd/system/logrotate.service.d
sudo tee /etc/systemd/system/logrotate.service.d/g8-apisix.conf >/dev/null <<'EOF'
[Service]
ReadWritePaths=/usr/local/apisix/logs
EOF
sudo systemctl daemon-reload

And the verification has to go through the path the timer uses:

sudo systemctl start logrotate.service
systemctl show -p Result --value logrotate.service

success

Write maxsize, not size

With size 50M, logrotate abandons time-based rotation entirely; the file never rotates until it exceeds 50 MB. Details: the logrotate trap.

10. Starting

sudo systemctl enable --now apisix

11. Verification

11.1 Services

systemctl is-active etcd apisix
ss -lntH | awk '{print $4}' | grep -E ':(9080|9180|9091|2379)$' | sort -u

active
active
0.0.0.0:9080
0.0.0.0:9091
0.0.0.0:9180
127.0.0.1:2379
[::]:9080

Whether the process limit was actually applied is read from the running process, not from the unit file:

awk '/Max open files/{print $4}' /proc/$(systemctl show -p MainPID --value apisix)/limits

65536

11.2 Admin API and key substitution

Three requests: the real key, the unsubstituted raw template, and no key. If the second does not return 401, ${{...}} substitution did not happen and the key was taken as literal text.

KEY=$(sudo grep '^G8_APISIX_ADMIN_KEY=' /etc/g8/apisix.env | cut -d= -f2-)
umask 077
printf 'header = "X-API-KEY: %s"\n' "$KEY" > /tmp/g8adm.conf
printf 'header = "X-API-KEY: ${{G8_APISIX_ADMIN_KEY}}"\n' > /tmp/g8raw.conf
curl -s -o /dev/null -w "real=%{http_code}\n" -K /tmp/g8adm.conf http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -w "raw=%{http_code}\n"  -K /tmp/g8raw.conf http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -w "none=%{http_code}\n"                    http://127.0.0.1:9180/apisix/admin/routes

real=200
raw=401
none=401

The key is never written on the command line; curl -K reads it from a file. A secret placed on the command line is visible in ps output and in shell history.

11.3 Smoke test

curl -s -o /dev/null -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
  -X PUT http://127.0.0.1:9180/apisix/admin/routes/g8-smoke \
  -H "Content-Type: application/json" \
  -d '{"uri":"/smoke","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"G8-GW-TEST-OK"}}}'
sleep 2
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"

PUT=201
G8-GW-TEST-OK
status=200

11.4 Is the configuration persisted in etcd

sudo etcdctl --endpoints=127.0.0.1:2379 get /apisix/routes --prefix --keys-only | grep -v '^$'

/apisix/routes/
/apisix/routes/g8-smoke

11.5 Restart resilience

sudo systemctl restart etcd apisix
for i in $(seq 1 15); do
  [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9080/smoke)" = "200" ] && break
  sleep 4
done
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"
systemctl is-active etcd apisix

G8-GW-TEST-OK
status=200
active
active

11.6 Is log rotation lossless

What is measured is that requests arriving after rotation are written to the new file. The number of lines left in the old file is counted too; unless both are verified, rotation may be silently swallowing lines. The log is truncated first and the count targets only /smoke lines; counting total lines is unstable because of nginx's delayed flush.

sudo truncate -s 0 /usr/local/apisix/logs/access.log
for i in 1 2 3; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
sudo logrotate -f /etc/logrotate.d/apisix
sleep 2
for i in 1 2 3 4; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
echo "new=$(sudo grep -c '"uri":"/smoke"' /usr/local/apisix/logs/access.log)"
echo "old=$(sudo grep -c '"uri":"/smoke"' /usr/local/apisix/logs/access.log.1)"

new=4
old=3

11.7 Does the data plane survive an etcd outage

etcd is stopped. APISIX must not stop; it must keep serving the existing routes from cache and merely be unable to receive new configuration. Two things are measured together: the service state and the response to a real request.

sudo systemctl stop etcd
sleep 6
echo "apisix=$(systemctl is-active apisix) etcd=$(systemctl is-active etcd)"
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"
sudo systemctl start etcd
sleep 8
echo "apisix=$(systemctl is-active apisix) etcd=$(systemctl is-active etcd)"

apisix=active etcd=inactive
G8-GW-TEST-OK
status=200
apisix=active etcd=active

APISIX coming back inactive is a unit file error, not an etcd problem — 17.8.

11.8 Cleaning up verification leftovers

11.2 writes the admin key into /tmp/g8adm.conf in cleartext. If the file is not deleted the key stays on disk, and the care section 6 takes to protect the key with chmod 600 is wasted here.

rm -f /tmp/g8adm.conf /tmp/g8raw.conf
ls /tmp/g8*.conf 2>/dev/null | wc -l

0

12. Observability

This environment gets its own Elasticsearch and Prometheus. There is no Docker on this machine; both are extracted from the Harbor images and run as systemd units.

component	version	source
Elasticsearch	9.5.2	harbor.g8.dev/infra/elasticsearch:9.5.2
Prometheus	3.15.0	harbor.g8.dev/infra/prometheus:v3.15.0

Prometheus starts with --enable-feature=promql-extended-range-selectors: Analytics counts each bucket exactly with the anchored range selector, which Prometheus 3.15 takes only behind that flag. From Prometheus 3.16 the selector is on by default and the line can go; without it, Analytics shows its counts as estimates.

No separate Java install is needed: the Elasticsearch image carries its own JDK.

Prometheus cannot go on 9090

On this machine host port 9090 belongs to APISIX's own control API (openresty). In containerised installs that port lives inside the container and the host port is free; here it is not. Prometheus therefore listens on 9099. Try 9090 and the service fails with bind: address already in use, and systemd keeps restarting it forever.

12.1 Extracting from the images

sudo apt-get install -y skopeo

sudo rm -rf /tmp/es-img /tmp/es-x && mkdir -p /tmp/es-x
skopeo copy docker://harbor.g8.dev/infra/elasticsearch:9.5.2 dir:/tmp/es-img
for blob in /tmp/es-img/*; do
  tar -tzf "$blob" >/dev/null 2>&1 || continue
  tar -xzf "$blob" -C /tmp/es-x usr/share/elasticsearch 2>/dev/null || true
done

sudo rm -rf /tmp/pr-img /tmp/pr-x && mkdir -p /tmp/pr-x
skopeo copy docker://harbor.g8.dev/infra/prometheus:v3.15.0 dir:/tmp/pr-img
for blob in /tmp/pr-img/*; do
  tar -tzf "$blob" >/dev/null 2>&1 || continue
  tar -xzf "$blob" -C /tmp/pr-x bin/prometheus bin/promtool 2>/dev/null || true
done

id elasticsearch >/dev/null 2>&1 || sudo useradd -r -s /usr/sbin/nologin -d /usr/share/elasticsearch elasticsearch
id prometheus    >/dev/null 2>&1 || sudo useradd -r -s /usr/sbin/nologin -d /var/lib/prometheus prometheus

sudo rm -rf /usr/share/elasticsearch
sudo cp -a /tmp/es-x/usr/share/elasticsearch /usr/share/elasticsearch
sudo install -m0755 /tmp/pr-x/bin/prometheus /tmp/pr-x/bin/promtool /usr/local/bin/
sudo install -d -o elasticsearch -g elasticsearch -m0750 /var/lib/elasticsearch /var/log/elasticsearch
sudo install -d -o prometheus -g prometheus -m0750 /var/lib/prometheus
sudo install -d -m0755 /etc/prometheus /etc/g8
sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
sudo rm -rf /tmp/es-img /tmp/es-x /tmp/pr-img /tmp/pr-x

12.2 Elasticsearch configuration

/usr/share/elasticsearch/config/elasticsearch.yml

cluster.name: g8-test
node.name: es-test
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: true
xpack.security.enrollment.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false
ingest.geoip.downloader.enabled: false

The heap is capped to fit the machine — left alone, Elasticsearch asks for half of it:

sudo tee /usr/share/elasticsearch/config/jvm.options.d/heap.options >/dev/null <<'EOF'
-Xms1g
-Xmx1g
EOF
sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config

/etc/systemd/system/elasticsearch.service

[Unit]
Description=Elasticsearch
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=elasticsearch
Group=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/usr/share/elasticsearch/config
ExecStart=/usr/share/elasticsearch/bin/elasticsearch
LimitNOFILE=65536
LimitMEMLOCK=infinity
LimitNPROC=4096
TimeoutStartSec=300
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

12.3 Prometheus configuration

external_labels marks the environment; when metrics from several environments meet, this is the only thing that tells them apart.

/etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    g8_environment: test

scrape_configs:
  - job_name: apisix
    metrics_path: /apisix/prometheus/metrics
    static_configs:
      - targets: ["127.0.0.1:9091"]
        labels:
          g8_environment: test

  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9099"]

/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=40d \
  --storage.tsdb.retention.size=5GB \
  --enable-feature=promql-extended-range-selectors \
  --web.listen-address=0.0.0.0:9099
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/prometheus

[Install]
WantedBy=multi-user.target

12.4 Start and password

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch prometheus
systemctl is-active elasticsearch prometheus

On first start Elasticsearch generates a password for the elastic user. The command below resets it and writes it to a file that stays on the machine. The password reaches neither this document nor the repository. It is the store administrator's password and no rule of the gateway carries it: the gateway writes with an account of its own (12.6):

P=$(sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -b -s | tail -1)
printf 'G8_ES_PASSWORD=%s\n' "$P" | sudo tee /etc/g8/es.env >/dev/null
sudo chmod 600 /etc/g8/es.env
unset P

12.5 Wiring APISIX

The metrics are enabled through a global rule, which leaves prefer_name off: route and service on the metrics carry the ids Analytics counts by, and metrics.*.extra_labels in config.yaml put each route's and service's name beside them as route_name and service_name. The request records are not wired here: g8 writes their rule itself when traffic logging is turned on (12.7).

sudo tee /etc/g8/wire-observability.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
KEY=$(sudo grep '^G8_APISIX_ADMIN_KEY=' /etc/g8/apisix.env | cut -d= -f2-)
code=$(curl -s -o /dev/null -w '%{http_code}' -H "X-API-KEY: $KEY" \
  -X PUT http://127.0.0.1:9180/apisix/admin/global_rules/1 \
  -H 'Content-Type: application/json' -d '{"plugins":{"prometheus":{"prefer_name":false}}}')
echo "global_rule -> $code"
SCRIPT
sudo chmod 700 /etc/g8/wire-observability.sh
sudo /etc/g8/wire-observability.sh

global_rule -> 201

The prometheus plugin must be enabled; plugin_attr is not enough

plugin_attr.prometheus in config.yaml only sets the address the metrics are published on. Unless the plugin is enabled on a global rule, no request metrics are collected at all. The symptom is treacherous: the Prometheus target reads up, /apisix/prometheus/metrics returns 200, and yet a query for apisix_http_status comes back with zero series.

12.6 Accounts, the ingest pipeline and the index template in the log store

Unverified

The commands of this section were run on a development stack on 2026-10-02 (APISIX 3.19.0, Elasticsearch 8.15.3) with the outputs shown. On Elasticsearch 9.5.2 the privileges were measured with roles that name the days by a wildcard, and without the line that reads the template; the roles below name the days by a regular expression, which was measured on 8.15.3 alone. The commands have not been run on this machine yet.

The ingest pipeline, the line of the server's role that reads it and the move of 16.2 were run on 2026-10-04 against a development stack's Elasticsearch 8.15.3 with the outputs shown; that a batch sent twice is stored once was also measured in the lab on Elasticsearch 9.5.2 and OpenSearch 3.9.0.

Two accounts are opened in Elasticsearch, and neither of them is elastic:

account	used by	may
g8-test-gateway	the gateway, to deliver the request records	write records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster
g8-test-server	the g8 server, to show the records and to clean them up	read the state of the cluster and g8's ingest pipeline, read the daily indices of this environment and delete a day by its name

The daily indices carry the slug of the environment in g8 (g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is registered under in the g8 console. The roles name the days by a regular expression and not by g8-test-access-*: a wildcard would also reach the days of an environment whose slug begins with test-access.

sudo tee /etc/g8/store-accounts.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
ENV=test
ES=http://127.0.0.1:9200
for who in GATEWAY SERVER; do
  grep -q "^G8_ES_${who}_PASSWORD=" /etc/g8/es.env || \
    printf 'G8_ES_%s_PASSWORD=%s\n' "$who" "$(openssl rand -base64 24 | tr -d /+= | head -c 24)" >> /etc/g8/es.env
done
ESP=$(grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
GWP=$(grep '^G8_ES_GATEWAY_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
SVP=$(grep '^G8_ES_SERVER_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
put() {
  code=$(curl -s -o /dev/null -w '%{http_code}' -u "elastic:$ESP" -X PUT "$ES$1" \
    -H 'Content-Type: application/json' --data-binary @-)
  echo "$1 -> $code"
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_security/role/g8-$ENV-gateway <<EOF
{"indices":[{"names":["$DAYS"],"privileges":["index","auto_configure"]}]}
EOF
put /_security/role/g8-$ENV-server <<EOF
{"cluster":["monitor","read_pipeline"],"indices":[{"names":["$DAYS"],
 "privileges":["read","view_index_metadata","monitor","delete_index"]}]}
EOF
put /_security/user/g8-$ENV-gateway <<EOF
{"password":"$GWP","roles":["g8-$ENV-gateway"]}
EOF
put /_security/user/g8-$ENV-server <<EOF
{"password":"$SVP","roles":["g8-$ENV-server"]}
EOF
SCRIPT
sudo chmod 700 /etc/g8/store-accounts.sh
sudo /etc/g8/store-accounts.sh

/_security/role/g8-test-gateway -> 200
/_security/role/g8-test-server -> 200
/_security/user/g8-test-gateway -> 200
/_security/user/g8-test-server -> 200

The script can be run again: it keeps the passwords it finds in /etc/g8/es.env.

The gateway's account must not be able to read

The password of g8-test-gateway is part of the gateway's configuration: whoever can read the global rules of the gateway can obtain it. That is why this account can only add records. When traffic logging is turned on with a plain password, g8 tries the account and refuses one that can read the cluster or a record; a password given as a secret reference is resolved on the gateway and is not tried. create_doc is not enough for the account either: the plugin sends an index action, and with create_doc every record is refused while the request itself answers 200.

The ingest pipeline stores each record under its request id. The gateway's logger sends the records in batches and sends a batch again when the store answers it late, and by then the store has usually written the first one: under the request id, the second copy replaces the first instead of adding a record. The index template below sends every record through this pipeline, so the pipeline is applied first. A store that holds the template and not the pipeline refuses every record while the gateway's bulk request still answers 200; g8 checks that the store holds this version of the pipeline and does not turn traffic logging on without it. The store needs at least one node with the ingest role, which a node carries by default.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_ingest/pipeline/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
  "version": 1,
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{request_id}}}",
        "if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
      }
    }
  ]
}
EOF

{"acknowledged":true}

The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_index_template/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "index_patterns": [
    "g8-*-access-*"
  ],
  "priority": 200,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "codec": "best_compression",
        "auto_expand_replicas": "0-1",
        "default_pipeline": "g8-access",
        "analysis": {
          "analyzer": {
            "g8_nodes": {
              "type": "pattern",
              "pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
              "lowercase": false
            }
          }
        }
      }
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "latency": {
          "type": "float"
        },
        "upstream_latency": {
          "type": "float"
        },
        "apisix_latency": {
          "type": "float"
        },
        "client_ip": {
          "type": "ip",
          "ignore_malformed": true
        },
        "route_id": {
          "type": "keyword"
        },
        "service_id": {
          "type": "keyword"
        },
        "upstream": {
          "type": "keyword",
          "ignore_above": 1024,
          "fields": {
            "node": {
              "type": "text",
              "analyzer": "g8_nodes",
              "norms": false,
              "index_options": "docs"
            }
          }
        },
        "consumer": {
          "properties": {
            "username": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "properties": {
            "method": {
              "type": "keyword"
            },
            "uri": {
              "type": "wildcard"
            },
            "url": {
              "type": "keyword",
              "index": false,
              "doc_values": false
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "querystring": {
              "type": "object",
              "enabled": false
            }
          }
        },
        "request_body": {
          "type": "text",
          "index": false
        },
        "response": {
          "properties": {
            "status": {
              "type": "short"
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "body": {
              "type": "text",
              "index": false
            }
          }
        },
        "server": {
          "properties": {
            "hostname": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "request_id": {
          "type": "keyword"
        },
        "route_name": {
          "type": "keyword"
        },
        "service_name": {
          "type": "keyword"
        },
        "upstream_status": {
          "type": "keyword"
        },
        "x_request_id": {
          "type": "keyword"
        },
        "logged_at": {
          "type": "date",
          "format": "epoch_second"
        }
      },
      "_meta": {
        "g8_records": 2
      },
      "_source": {
        "excludes": [
          "response.headers.set-cookie"
        ]
      }
    }
  }
}
EOF

{"acknowledged":true}

What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:

GWP=$(sudo grep '^G8_ES_GATEWAY_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
SVP=$(sudo grep '^G8_ES_SERVER_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
curl -s -o /dev/null -w 'g8-test-gateway reads the cluster      -> %{http_code}\n' -u "g8-test-gateway:$GWP" http://127.0.0.1:9200/
curl -s -o /dev/null -w 'g8-test-gateway reads records          -> %{http_code}\n' -u "g8-test-gateway:$GWP" 'http://127.0.0.1:9200/g8-test-access-*/_search'
curl -s -o /dev/null -w 'g8-test-gateway writes a day           -> %{http_code}\n' -u "g8-test-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-test-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-test-gateway writes beside its days -> %{http_code}\n' -u "g8-test-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-test-access-x-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-test-server reads the template      -> %{http_code}\n' -u "g8-test-server:$SVP" http://127.0.0.1:9200/_index_template/g8-access
curl -s -o /dev/null -w 'g8-test-server reads the pipeline      -> %{http_code}\n' -u "g8-test-server:$SVP" http://127.0.0.1:9200/_ingest/pipeline/g8-access
curl -s -o /dev/null -w 'g8-test-server reads beside its days   -> %{http_code}\n' -u "g8-test-server:$SVP" 'http://127.0.0.1:9200/g8-test-access-x-access-2000.01.01/_search'
curl -s -o /dev/null -w 'g8-test-server writes a record         -> %{http_code}\n' -u "g8-test-server:$SVP" -X POST http://127.0.0.1:9200/g8-test-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-test-server deletes the day         -> %{http_code}\n' -u "g8-test-server:$SVP" -X DELETE http://127.0.0.1:9200/g8-test-access-2000.01.01

g8-test-gateway reads the cluster      -> 403
g8-test-gateway reads records          -> 403
g8-test-gateway writes a day           -> 201
g8-test-gateway writes beside its days -> 403
g8-test-server reads the template      -> 200
g8-test-server reads the pipeline      -> 200
g8-test-server reads beside its days   -> 403
g8-test-server writes a record         -> 403
g8-test-server deletes the day         -> 200

Data streams cannot be used

elasticsearch-logger writes an index action in its bulk request; data streams require create and reject index. Hence one dated index per day.

12.7 Turning on the request records

Unverified

These steps were walked on a development stack on 2026-10-02, with the two accounts of 12.6. They have not been walked against this machine yet.

The request records are turned on from the g8 console, once this gateway is registered there as the environment test. No command on this machine writes the logger's rule.

1. Environments › test › Edit › Observability. Elasticsearch URL: the address the g8 server reaches this store at. Username g8-test-server, password the value of G8_ES_SERVER_PASSWORD in /etc/g8/es.env. Test connection, then save.
2. Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Index template Applied and Ingest pipeline Applied.
3. Turn on traffic logging. Log store address, as the gateway reaches it: http://127.0.0.1:9200. Username g8-test-gateway, password the value of G8_ES_GATEWAY_PASSWORD in /etc/g8/es.env. Turn on.

g8 writes one global rule, g8-request-records, beside rule 1: the logger with the daily index of this environment, the fields the records need and the default masks. What is recorded and masked is changed afterwards on the same Settings page. Where the gateway writes is shown on the Status card as Gateway writes to; Change address gives the logger another address and account for the same store, through the same checks of the account as turning on.

12.8 Retention

A log store does not shrink on its own. With no limit the disk fills eventually, and past 95% Elasticsearch turns its indices read-only.

g8 deletes the request records: once a day it deletes, by its name, every daily index older than the number of days the environment keeps — 30 unless it is changed under Traffic Explorer › Settings › Retention › Change — and writes each deletion to the audit log. The store carries no lifecycle policy for these indices; a second deleter would delete days g8 still shows.

A day is named by its UTC date, by the gateway that writes it and by g8 that reads and deletes it.

On a single-node Elasticsearch a replica can never be allocated. The template sets auto_expand_replicas: 0-1: an index has no replica on one node and one as soon as a second node joins. The verification below must show 0 and green.

12.9 Verification

Unverified

The output below was recorded on this machine while the rule of the earlier guide was in place; the index it shows is named by the machine's local day. With the rule g8 writes, the commands were run on a development stack on 2026-10-02: the index of the UTC day showed the requests sent, 0 and green. They have not been run on this machine since.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
for i in $(seq 1 25); do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 20
curl -s -u "elastic:$ESP" 'http://127.0.0.1:9200/_cat/indices/g8-*?v&h=index,docs.count,rep,health'
curl -s 'http://127.0.0.1:9099/api/v1/query?query=sum(apisix_http_status)'

index                     docs.count rep health
g8-test-access-2026.08.29         25   0 green

Traffic Explorer in the g8 console must list the same requests.

Analytics reads two things from these metrics: each route under its id with its name beside it, and each bucket counted exactly with the anchored range selector:

curl -sG http://127.0.0.1:9099/api/v1/query --data-urlencode 'query=count by (route, route_name) (apisix_http_status)'
curl -sG http://127.0.0.1:9099/api/v1/query --data-urlencode 'query=sum(increase(apisix_http_status[5m] anchored))'

The first query lists every route that answered with its id in route and its name in route_name; a route that carries a name means a copy of the prometheus plugin has prefer_name on, and Analytics reads that route empty. The second must answer "status":"success"; a Prometheus started without --enable-feature=promql-extended-range-selectors answers anchored modifier is experimental and not enabled, and Analytics marks its counts as estimates. Both queries were run on this machine on 2026-10-04 against Prometheus 3.15.0: every route with a name showed it in route_name, and the anchored query answered success.

Request records are LOST while Elasticsearch is unreachable

The gateway sends a batch of records once more after the first failure (max_retry_count 1 in the rule g8 writes) and then drops it. The loss was measured with the rule of the earlier guide, which tried three times: on a production installation during a password change, 20 requests that arrived while Elasticsearch was restarting appeared in the metrics but never reached the index.

Read the loss from APISIX's error log, not from a counter difference:

grep -c 'elasticsearch-logger' <error.log>

Subtracting the document count from the metric counter is not reliable and produces false alarms. A measured example: counter 88, documents 43, difference 45 — with not one delivery error in the log. The difference has three causes and none of them is loss: the counter counts from the moment APISIX started (requests from before the plugin was enabled are in the counter but not the index), a restart resets the counter while the index persists, and the two sides have different retention. The difference is a hint only within a single window — no APISIX restart, nothing deleted by retention, both plugins enabled together.

An installation that cannot afford losing records must put a buffer in between (kafka-logger, or a local file-logger plus a separate shipper).

Restarts were measured to lose nothing: after systemctl restart the document count was unchanged and Prometheus data from before the restart was still queryable. Persistence comes from /var/lib/elasticsearch and /var/lib/prometheus.

13. Diagnostics

12-18 are not sequential steps

Everything up to here is applied in order. What follows are reference sections: diagnostics, backup, uninstall, upgrade, traps. A reader who takes the numbering for steps will, at section 15, delete the system they just installed, and then run section 18 against units that no longer exist.

systemctl status apisix --no-pager -l
journalctl -u apisix -n 50 --no-pager
journalctl -u etcd -n 50 --no-pager
sudo tail -n 50 /usr/local/apisix/logs/error.log
sudo tail -n 50 /usr/local/apisix/logs/access.log

Configuration errors surface before the service starts, during the ExecStartPre stage. To see the error directly instead of waiting for systemd:

cd /usr/local/apisix && sudo env $(sudo cat /etc/g8/apisix.env) apisix init

etcd health and which member is the leader:

sudo etcdctl --endpoints=127.0.0.1:2379 endpoint health -w table
sudo etcdctl --endpoints=127.0.0.1:2379 endpoint status -w table

14. Backup and restore

The only state to back up is in etcd; APISIX holds none. On the file side, what needs backing up is config.yaml and the admin key.

sudo mkdir -p /var/backups/g8-gw
sudo etcdctl --endpoints=127.0.0.1:2379 snapshot save /var/backups/g8-gw/etcd-manual.db
sudo etcdutl snapshot status /var/backups/g8-gw/etcd-manual.db -w table

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 8f59dd47 |       43 |         48 |      25 kB |
+----------+----------+------------+------------+

Daily automated backup:

sudo tee /etc/systemd/system/g8-etcd-backup.service >/dev/null <<'EOF'
[Unit]
Description=g8 etcd snapshot backup
After=etcd.service

[Service]
Type=oneshot
TimeoutStartSec=300
ExecStart=/bin/sh -c '/usr/local/bin/etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=120s snapshot save /var/backups/g8-gw/etcd-$(date +%%Y%%m%%d-%%H%%M).db'
ExecStartPost=/bin/sh -c 'ls -1t /var/backups/g8-gw/etcd-*.db | tail -n +15 | xargs -r rm -f'
EOF
sudo tee /etc/systemd/system/g8-etcd-backup.timer >/dev/null <<'EOF'
[Unit]
Description=g8 etcd snapshot backup (daily)

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-etcd-backup.timer

That the timer is installed does not prove a backup was taken. Run the service once by hand and read its result:

sudo systemctl start g8-etcd-backup.service
systemctl show -p Result --value g8-etcd-backup.service
ls -1 /var/backups/g8-gw/etcd-*.db | wc -l

success
2

The next run time is shown by systemctl list-timers g8-etcd-backup.timer --no-pager.

14.1 Restore

The old data directory is moved aside, not deleted — if the restore does not go as expected, that is the only place to go back to. --data-dir is /var/lib/etcd directly; it must match the --data-dir in the unit file.

If you are running a drill, write one marker before and one after the backup so you can measure that the restore actually restored:

sudo etcdctl --endpoints=http://127.0.0.1:2379 put /marker-A before-backup
sudo etcdctl --endpoints=http://127.0.0.1:2379 snapshot save /var/backups/g8-gw/etcd-manual.db
sudo etcdctl --endpoints=http://127.0.0.1:2379 put /marker-B after-backup

sudo systemctl stop apisix etcd
sudo mv /var/lib/etcd /var/lib/etcd.bozuk
sudo etcdutl snapshot restore /var/backups/g8-gw/etcd-manual.db \
  --name etcd-test \
  --initial-cluster etcd-test=http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --data-dir /var/lib/etcd
sudo chown -R etcd:etcd /var/lib/etcd
sudo systemctl start etcd apisix

After the restore the first marker must be there and the second must be gone. The services coming up is not sufficient evidence: a wrong restore also produces a running etcd, just an empty one.

printf "apisix=%s etcd=%s\n" "$(systemctl is-active apisix)" "$(systemctl is-active etcd)"
sudo etcdctl --endpoints=http://127.0.0.1:2379 get /marker-A --print-value-only
sudo etcdctl --endpoints=http://127.0.0.1:2379 get /marker-B --print-value-only
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"

apisix=active etcd=active
before-backup
G8-GW-TEST-OK
status=200

The marker-B line coming back empty is the result you are looking for — it shows that everything written after the snapshot is gone. If both come back, no restore happened and etcd came up on the old data.

14.2 Defragmentation

Compaction removes old revisions logically but does not shrink the database file — the freed pages stay inside etcd and never return to the filesystem. Since the quota watches dbSize, the file keeps growing even when compaction is working perfectly, and the quota fills sooner or later. The extreme measured: dbSizeInUse at 41 kB while dbSize sat at 98 MB.

Defragmentation therefore has to be scheduled. Weekly is enough.

sudo tee /etc/systemd/system/g8-etcd-defrag.service >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation
After=etcd.service
Wants=etcd.service

[Service]
Type=oneshot
TimeoutStartSec=900
ExecStart=/usr/local/bin/etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=900s defrag
ExecStartPost=/usr/local/bin/etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=30s alarm list
EOF

sudo tee /etc/systemd/system/g8-etcd-defrag.timer >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation (weekly)

[Timer]
OnCalendar=Sun 03:00
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-etcd-defrag.timer

Verification:

sudo systemctl start g8-etcd-defrag.service
systemctl show -p Result --value g8-etcd-defrag.service

success

Defrag blocks that member briefly, which is why the members are processed one at a time. Defragmenting all three at once drops quorum.

Defrag temporarily needs free space equal to the database size; that is one reason the quota must be meaningfully smaller than the disk.

14.3 Getting backups off the node

Until now the backups sit on the same disk as the data they protect. If the disk goes, both go with it — the backup does not protect against the one event it promises to.

Direction matters: the remote machine must do the copying. Had the gateway pushed to the remote machine, the gateway would need write access there; when the gateway is compromised, the backups go too. Pulling means the gateway holds no secret at all.

On the gateway, a restricted command is defined that grants no shell and only produces the backup stream:

printf '#!/bin/sh\nexec /usr/bin/tar -C /var/backups/g8-gw -cf - .\n' \
  | sudo tee /usr/local/bin/g8-backup-stream >/dev/null
sudo chmod 0755 /usr/local/bin/g8-backup-stream

echo 'cosmonot ALL=(root) NOPASSWD: /usr/local/bin/g8-backup-stream' \
  | sudo tee /etc/sudoers.d/g8-backup-stream >/dev/null
sudo chmod 0440 /etc/sudoers.d/g8-backup-stream
sudo visudo -c

The remote machine's public key is added to authorized_keys with a command restriction; a session opened with that key gets no shell, only that command:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
cat >> ~/.ssh/authorized_keys <<'EOF'
command="sudo /usr/local/bin/g8-backup-stream",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... g8-obs-backup-pull
EOF
chmod 600 ~/.ssh/authorized_keys

On the remote machine (backup-host in this lab), the pull script:

sudo tee /usr/local/bin/g8-pull-backups.sh >/dev/null <<'SCRIPT'
#!/bin/sh
set -u
DEST=/var/backups/g8-remote
KEY=/home/cosmonot/.ssh/id_ed25519
RC=0

for spec in test-vm:192.0.2.107 prod:192.0.2.108; do
  name=${spec%%:*}
  host=${spec##*:}
  rm -rf "$DEST/$name.new"
  mkdir -p "$DEST/$name.new"
  if ssh -i "$KEY" -o BatchMode=yes -o ConnectTimeout=30 \
         -o StrictHostKeyChecking=accept-new \
         cosmonot@"$host" 2>/dev/null | tar -C "$DEST/$name.new" -xf - 2>/dev/null \
     && [ -n "$(ls -A "$DEST/$name.new")" ]; then
    rm -rf "$DEST/$name.prev"
    [ -d "$DEST/$name" ] && mv "$DEST/$name" "$DEST/$name.prev"
    mv "$DEST/$name.new" "$DEST/$name"
    echo "pulled $name: $(ls -1 "$DEST/$name" | wc -l) file(s)"
  else
    rm -rf "$DEST/$name.new"
    echo "FAILED $name" >&2
    RC=1
  fi
done

exit $RC
SCRIPT
sudo chmod 0755 /usr/local/bin/g8-pull-backups.sh
sudo mkdir -p /var/backups/g8-remote

And the timer:

sudo tee /etc/systemd/system/g8-pull-backups.service >/dev/null <<'EOF'
[Unit]
Description=g8 pull gateway etcd backups
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
TimeoutStartSec=900
ExecStart=/usr/local/bin/g8-pull-backups.sh
EOF
sudo tee /etc/systemd/system/g8-pull-backups.timer >/dev/null <<'EOF'
[Unit]
Description=g8 pull gateway etcd backups (daily)

[Timer]
OnCalendar=daily
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-pull-backups.timer

Verification. "There is a file" is not a backup; ask whether the pulled copy is identical to the source:

sudo systemctl start g8-pull-backups.service
systemctl show -p Result --value g8-pull-backups.service
sudo sha256sum /var/backups/g8-remote/test-vm/etcd-manual.db | cut -c1-16

success
27e8a4200d411a2d

The same sha256 shows the copy arrived intact. Compare it with the sha256 of the file on the source.

15. Uninstalling

Delete the environment in the g8 console first, or clear its log store there: g8 connects to the store once a day to delete the days past the retention, and records a failed cleanup every day for as long as the environment points at a store that is gone.

sudo systemctl disable --now apisix etcd g8-etcd-backup.timer
sudo rm -f /etc/systemd/system/apisix.service /etc/systemd/system/etcd.service
sudo rm -f /etc/systemd/system/g8-etcd-backup.service /etc/systemd/system/g8-etcd-backup.timer
sudo systemctl daemon-reload
sudo rm -rf /usr/local/apisix /usr/local/openresty /usr/local/bin/apisix /usr/bin/openresty
sudo rm -f /usr/local/bin/etcd /usr/local/bin/etcdctl /usr/local/bin/etcdutl
sudo rm -f /etc/logrotate.d/apisix

Deleting the data and the secrets is a separate decision; the commands above do not touch them:

sudo rm -rf /var/lib/etcd /etc/g8 /var/backups/g8-gw
sudo userdel etcd
sudo rm -f /usr/local/share/ca-certificates/g8-lab-ca.crt
sudo update-ca-certificates --fresh

Verify that the uninstall completed:

ss -lntH | awk '{print $4}' | grep -cE ':(9080|9180|9091|2379|9200|9300|9099)$'

0

16. Upgrading

This sequence moved this environment from APISIX 3.18.0 to 3.19.0 on 2026-09-30; the commands below are the ones that ran. etcd is not part of it: its version and its data stay as they are.

1. Take an etcd backup and verify it:

sudo mkdir -p /var/backups/g8-gw
sudo etcdctl --endpoints=127.0.0.1:2379 --command-timeout=60s snapshot save /var/backups/g8-gw/etcd-pre-3.19-$(date +%Y%m%d-%H%M).db
sudo etcdutl snapshot status $(ls -1t /var/backups/g8-gw/etcd-pre-3.19-*.db | head -1) -w table

2. Pull the new runtime from Harbor — the same artifact section 4 extracts, under the new tag:

T=/tmp/apisix-rt-3.19
sudo rm -rf "$T"
sudo skopeo copy docker://harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04 dir:$T
LAYER=$(python3 -c "import json;print(json.load(open('$T/manifest.json'))['layers'][0]['digest'].split(':')[1])")

3. Stop APISIX; etcd keeps running — the second command prints inactive for APISIX and active for etcd:

sudo systemctl stop apisix
systemctl is-active apisix etcd

4. Keep the old configuration and the old trees. The trees are renamed, not deleted, so the old runtime stays on the disk until the new one has passed:

sudo mkdir -p /root/g8-apisix-3.18-backup
sudo cp -a /usr/local/apisix/conf /root/g8-apisix-3.18-backup/conf
sudo mv /usr/local/apisix /usr/local/apisix.3.18
sudo mv /usr/local/openresty /usr/local/openresty.3.18

5. Extract the new tarball, put config.yaml and conf/cert/ back, and check the version:

sudo tar -xzf "$T/$LAYER" -C /
sudo install -m0644 /root/g8-apisix-3.18-backup/conf/config.yaml /usr/local/apisix/conf/config.yaml
sudo cp -a /root/g8-apisix-3.18-backup/conf/cert /usr/local/apisix/conf/
cd /usr/local/apisix && apisix version

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
3.19.0

6. Start APISIX and run the verification chain in section 11 from the top, then the shared-library sweep of section 20 with the number of files it scanned:

sudo systemctl start apisix
systemctl is-active apisix

cd / && sudo find /usr/local/openresty /usr/local/apisix -type f \
  \( -name '*.so' -o -name '*.so.*' -o -perm -u+x \) | wc -l

What the 2026-09-30 run measured: apisix version printed 3.19.0, the one route stored in etcd before the upgrade was still there and answered through the edge, and the sweep scanned 492 files with no MISSING: line. The section 11 chain was not re-run in full on that date; those four observations are what ran.

Two differences from the design this section carried before the run, both kept on purpose: the old trees are renamed (/usr/local/apisix.3.18, /usr/local/openresty.3.18) instead of deleted, and the whole conf/ directory is copied to /root/g8-apisix-3.18-backup/ before the swap, not config.yaml alone — conf/cert/ is restored from it as well.

Unverified

The rollback has not been run. Its design: stop APISIX, move the new trees aside, rename /usr/local/apisix.3.18 and /usr/local/openresty.3.18 back, start APISIX. The tarball also writes /usr/local/bin/apisix and /usr/bin/openresty outside the two trees; after a rollback those two files are still the new runtime's, and whether that matters has not been measured.

16.1 Request records of an installation made with the earlier guide

Unverified

This move was rehearsed on a development stack on 2026-10-02: an installation made as the earlier guide wrote it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

The earlier version of this guide wrote the logger into global rule 1 by hand: with the elastic account, an index named g8-test-access- and the machine's local day, and a lifecycle policy that deleted the indices. Such an installation is moved in this order:

Do not turn traffic logging on before step 6

Settings offers Turn on traffic logging as soon as the pipeline and the template are applied. Turning on over the old rule completes it in place: the account is neither asked for nor tried, and elastic stays in the rule.

1. Open the two accounts (12.6).
2. Connect the store in the g8 console with g8-test-server (step 1 of 12.7). Settings then reads Index template Outdated, Ingest pipeline Missing and the logger rule Customized. From its next daily cleanup on, g8 deletes the days past the retention.
3. Apply the ingest pipeline and then the index template (12.6). They apply to the days born after them.
4. Take the lifecycle policy off the indices and delete it:

ESP=$(sudo grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT 'http://127.0.0.1:9200/g8-*-access-*/_settings' \
  -H 'Content-Type: application/json' -d '{"index.lifecycle.name":null}'
curl -s -u "elastic:$ESP" -X DELETE http://127.0.0.1:9200/_ilm/policy/g8-access-retention

{"acknowledged":true}{"acknowledged":true}

5. The policy kept the indices 7 days. g8 keeps them 30 days unless that is changed: set the number of days this store is sized for under Traffic Explorer › Settings › Retention › Change.
6. Run the whole block of 12.5 again. It writes wire-observability.sh anew before it runs it: the file on this machine is the earlier guide's and would write the logger back. The block prints global_rule -> 200 here; rule 1 then carries the metrics alone and Settings reads the logger rule Off. Only then turn traffic logging on, with g8-test-gateway (steps 2 and 3 of 12.7).
7. Change the password of elastic: it was part of the gateway's configuration for as long as the old rule existed.

No record is written between the two halves of step 6. Every day written before the move, and the day of the move itself, was born without g8's template and without the fields its records carry: Traffic Explorer cannot list those days, and Settings counts them as written under another version. They stay in the store until the retention deletes them. The records are listed from the first day whose index is born after the move.

g8 lists and deletes the days of the environment's own slug alone. If this gateway is registered in g8 under another slug than test, the days the old rule wrote have no deleter once the policy is gone: delete them by their names when they are no longer needed.

16.2 The ingest pipeline on an installation made before it

Unverified

This move was rehearsed on 2026-10-04 against a development stack's Elasticsearch 8.15.3: an installation with the earlier template and a day born under it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

An installation made before this version of the guide holds the index template without the ingest pipeline, and its server account may not read pipelines. Settings then reads Index template Outdated and Ingest pipeline Cannot be read. The gateway keeps writing records meanwhile, copies included, and the settings can still be saved; g8 refuses only to turn logging on anew until both read Applied. The move, in this order:

1. Run the whole block that writes the accounts script again (12.6). It rewrites the saved script with the new server role and runs it, keeping the passwords; every line answers 200. Running the saved script alone writes the old role again.
2. Apply the ingest pipeline (12.6).
3. Apply the index template (12.6). The days born after it send their records through the pipeline.
4. Send the day of today, born under the earlier template, through the pipeline at once. On a store that serves several environments, run it once for each environment's slug:

ESP=$(sudo grep '^G8_ES_PASSWORD=' /etc/g8/es.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT "http://127.0.0.1:9200/g8-test-access-$(date -u +%Y.%m.%d)/_settings" \
  -H 'Content-Type: application/json' -d '{"index.default_pipeline":"g8-access"}'

{"acknowledged":true}

A store that holds no record of today yet answers 404 index_not_found_exception here: today's day will be born under the new template, and nothing more is needed. Settings then reads both Applied. A day written before the move may hold copies of a batch the gateway delivered twice; it stays as it is until the retention deletes it.

17. Traps hit during installation

17.1 make install does not install the Lua dependencies

Symptom: apisix init does not run; the first missing module is base64.

/usr/local/share/lua/5.1/apisix/cli/etcd.lua:18: module 'base64' not found:
	no field package.preload['base64']
	no file '/usr/local/apisix/deps/share/lua/5.1/base64.lua'

APISIX's Makefile installs dependencies inside the source tree with luarocks install --tree deps; the make install target never touches the deps/ directory. The runtime, however, looks for them under /usr/local/apisix/deps. The packaging step closes this gap — see section 20.

A customer installing from the tarball does not hit this trap; the gap is closed while the tarball is produced.

17.2 The wrong install prefix makes lyaml invisible

Symptom: once base64 is resolved, lyaml cannot be found instead.

/usr/local/share/lua/5.1/apisix/cli/file.lua:19: module 'lyaml' not found:
	no file '/usr/local/apisix/deps/share/lua/5.1/lyaml.lua'

lyaml is installed as lyaml/init.lua. apisix/cli/apisix.lua adds only deps/share/lua/5.1/?.lua to the Lua search path — not ?/init.lua. The missing entry is added by apisix/cli/env.lua, but only when the CLI is invoked as ./apisix/cli/apisix.lua.

If the sources are installed under /usr/local/share/lua/5.1/apisix, the wrapper picks the absolute-path branch, that branch never runs, and lyaml becomes structurally unfindable. The correct layout is the same as the official image's: sources at /usr/local/apisix/apisix, dependencies at /usr/local/apisix/deps.

17.3 WorkingDirectory is mandatory in the systemd unit

Symptom: cd /usr/local/apisix && apisix start works by hand, while systemctl start apisix fails with the lyaml error from 17.2.

Same root cause: if the systemd unit does not state a working directory, APISIX is not invoked from its own directory and the search path stays incomplete. That is what the WorkingDirectory=/usr/local/apisix line in the unit file is for.

17.4 Without libpcre3 the CLI's schema validation dies

Symptom: apisix version runs fine, apisix init dies with this line:

jsonschema.lua:54: depends on lrexlib-pcre, please install it first: error loading module 'rex_pcre' from file '/usr/local/apisix/deps/lib/lua/5.1/rex_pcre.so':
	libpcre.so.3: cannot open shared object file: No such file or directory

The jsonschema module uses ngx.re inside nginx, but in the CLI (plain LuaJIT, no ngx) it falls back to rex_pcre. rex_pcre.so links against PCRE1, and libpcre3 is not installed on the Ubuntu 24.04 cloud image.

apisix version passing does not rule this trap out: the module is loaded lazily and is only called once schema validation reaches a pattern keyword.

17.5 Do not install OpenResty from APT

Symptom: there is no direct error message. dpkg -l openresty reports one version while openresty -v prints another.

APISIX 3.19 requires a patched runtime (apisix-runtime). Building from source produces it and writes it under /usr/local/openresty — on top of the files belonging to the APT package. The package database and the disk then diverge and dead binaries such as nginx.old are left behind. This is why the procedure that produces the tarball never installs the APT OpenResty.

17.6 In logrotate, size cancels daily

Symptom: log files do not rotate for weeks; the after 1 days line is absent from logrotate -d output.

When size is given, logrotate disables time-based rotation: the file never rotates until it passes the threshold. "Daily, but sooner if it exceeds 50 MB" is written as daily + maxsize.

17.7 mocking, not echo, in the smoke test

Symptom: the body is correct but the status code is 503.

G8-GW-TEST-OK
status=503

The echo plugin writes the response body but does not short-circuit the request; with no upstream defined APISIX returns 503. mocking terminates the request itself and returns 200. The smoke test expects 200, so mocking is used.

17.8 Requires= takes the gateway down with etcd

Symptom: etcd is stopped for maintenance and the data plane goes with it. There is no error message; APISIX was stopped cleanly.

sudo systemctl stop etcd
systemctl is-active apisix

inactive

In systemd, Requires= is bidirectional: when the dependency is stopped, the dependent unit is stopped too. Writing Requires=etcd.service in apisix.service means stopping etcd shuts down the gateway — even though APISIX can serve the existing routes from cache without etcd. That is the behaviour in the Docker and OKD installations; there is no reason for the same product to behave worse on a virtual machine.

The correct directive is Wants=: together with After= it gives startup ordering without propagating the stop. The measurement is in 11.7.

17.9 The NOSPACE alarm is a one-way door

Symptom: writes are refused with etcdserver: mvcc: database space exceeded. Nobody notices, because the data plane keeps returning 200 — and once it is noticed, freeing space is not enough.

The recovery chain was measured, and all three steps are required:

159 writes         -> dbSize=16789504  alarm=NOSPACE
delete + compact   -> dbSize=16789504  write refused
defrag             -> dbSize=122880    write refused
alarm disarm       -> write OK

There are three separate misconceptions here, each verified individually:

• Deleting keys does not free space. etcd keeps the revision history; a delete writes a new revision.
• Compaction does not return disk space. dbSizeInUse drops, dbSize does not move at all — the freed pages stay free inside etcd and never return to the filesystem. Since the quota watches dbSize, the quota is still full. The extreme measured in this environment: dbSizeInUse at 41 kB while dbSize sat at 98 MB.
• Defrag returns the space but does not lower the alarm. The alarm is sticky: it survives a process restart, a container recreation and a host reboot.

Recovery:

sudo etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=900s defrag
sudo etcdctl --endpoints=http://127.0.0.1:2379 alarm list
sudo etcdctl --endpoints=http://127.0.0.1:2379 alarm disarm
sudo etcdctl --endpoints=http://127.0.0.1:2379 put /probe x

Leave alarm disarm for last: disarming without actually freeing space puts you straight back where you were on the next write.

The lasting fix is scheduled defragmentation — see long-term operation. With no defrag scheduled, dbSize keeps growing even when compaction is working correctly, and the quota fills sooner or later.

17.10 If the data directory is emptied, every check stays green

Symptom: none. That is precisely the problem.

If etcd loses its data directory (disk failure, accidental deletion, a wrong rm), then with ETCD_INITIAL_CLUSTER_STATE=new and an empty directory it bootstraps a fresh, empty cluster. The result:

• endpoint health → true
• both services active / healthy
• all four ports listening
• the smoke test… 404

The whole verification chain is green, because an empty cluster is a healthy cluster. It simply holds no configuration; every route the customer had is gone.

The only check that catches this asks about content, not health:

echo "routes=$(sudo etcdctl --endpoints=http://127.0.0.1:2379 get /apisix/routes --prefix --keys-only | grep -c 'routes/g8')"

routes=1

If you see routes=0, there is no configuration regardless of what the services say. Recovery is a restore from backup — see restore; and confirm that the backup was actually taken using the measurement in backup, because a timer being installed does not mean a backup exists.

18. Long-term operation

setting	value	why
--auto-compaction-mode=periodic + --auto-compaction-retention=1	hourly compaction	Without compaction etcd keeps the old revision of every write; the disk grows without bound
--quota-backend-bytes=2147483648	2 GB	Once the quota is exceeded etcd goes read-only with a NOSPACE alarm. The default is 2 GB; writing it explicitly stops the alarm from being a surprise
--snapshot-count=10000	10000	How often WAL files are truncated; a small value raises disk I/O, a large one lengthens recovery
LimitNOFILE=65536	in both units	The systemd default is 1024; APISIX returns 5xx with too many open files under moderate load
logrotate rotate 14 + maxsize 50M	14 files	In the worst case the log directory does not exceed ~700 MB
Restart=on-failure	in both units	systemd brings the process back if it dies; on-failure rather than always, because stopping by hand is deliberate
Wants=etcd.service	not Requires=	Requires= stops APISIX along with etcd and takes the data plane down (17.8)

That backups are actually being taken must be checked regularly; a timer being enabled does not prove a file was created:

systemctl list-timers g8-etcd-backup.timer --no-pager
ls -1t /var/backups/g8-gw/ | head -3

19. Scope limits and next steps

What this installation does not do, stated plainly:

• There is no fault tolerance. One etcd, one APISIX. If etcd stops, APISIX stays up but cannot receive configuration changes.
• There is no TLS. Traffic between APISIX and etcd, and the Admin API, are plain HTTP. Network safety rests on allow_admin and binding to 127.0.0.1. TLS is separate work.
• Authentication is the admin key only. No users or roles are configured in etcd.
• Restore is untested — 14.1. The upgrade in section 16 ran once (3.18.0 → 3.19.0, 2026-09-30); its rollback has not run.
• The tarball is verified against Ubuntu 24.04 only. libc and library compatibility on any other distribution or release is unmeasured.

20. Appendix — how the runtime tarball is produced

This section is not for the customer; it is the path followed on the g8 side when packaging a new APISIX version. The build machine is separate from the installation machine.

OpenResty is not installed from APT; make deps builds the patched runtime itself (17.5).

sudo apt-get install -y build-essential cmake curl git libldap2-dev libpcre3-dev \
  libssl-dev libyaml-dev libyaml-0-2 lua5.1 luarocks unzip zlib1g-dev
sudo git clone --depth 1 --branch 3.19.0 https://github.com/apache/apisix.git /usr/local/src/apisix
cd /usr/local/src/apisix
sudo make deps
sudo make install ENV_INST_LUADIR=/usr/local/apisix ENV_INST_BINDIR=/usr/local/bin
sudo cp -a /usr/local/src/apisix/deps /usr/local/apisix/deps

ENV_INST_LUADIR=/usr/local/apisix makes the layout identical to the official image's (17.2); the last line closes the gap make install leaves open (17.1).

The tree to be packaged is copied into a staging directory and everything that does not belong to the runtime is stripped: static libraries (.a, ~93 MB, read only at link time), documentation, generated state (logs/, nginx.conf, apisix.uid) and nginx.old if present.

S=/var/tmp/g8-apisix-runtime
sudo rm -rf "$S"; sudo mkdir -p "$S/usr/local/bin" "$S/usr/bin"
sudo cp -a /usr/local/apisix /usr/local/openresty "$S/usr/local/"
sudo cp -a /usr/local/bin/apisix "$S/usr/local/bin/"
sudo cp -a /usr/bin/openresty "$S/usr/bin/"
sudo find "$S" -name '*.a' -delete
sudo rm -rf "$S/usr/local/openresty/pod" "$S/usr/local/openresty/nginx/sbin/nginx.old"
sudo rm -f "$S/usr/local/apisix/logs/"* "$S/usr/local/apisix/conf/nginx.conf" "$S/usr/local/apisix/conf/apisix.uid"
sudo install -m0644 /usr/local/src/apisix/conf/config.yaml "$S/usr/local/apisix/conf/config.yaml"

The last line replaces the g8-specific config.yaml with the upstream default; the tarball carries no environment's settings.

The tarball is turned into a single-layer OCI artifact and pushed to Harbor. It is then extracted on a clean machine that has no compiler and the verification chain in section 11 is run from the top. This step cannot be skipped: the missing libpcre3 (17.4) and the leftover nginx.old (17.5) surfaced only in that exam — on the build machine neither is visible.

Sweep for missing shared libraries, on the clean machine:

cd / && sudo find /usr/local/openresty /usr/local/apisix -type f \
  \( -name '*.so' -o -name '*.so.*' -o -perm -u+x \) -print0 \
| xargs -0 -I{} sh -c 'ldd "$1" 2>/dev/null | grep -q "not found" && echo "MISSING: $1"' _ {} | sort -u

Make sure the sweep actually ran

The cd / is not incidental: if the shell is sitting in a deleted directory, every sh -c reports getcwd() failed, no ldd runs at all, and the sweep looks clean with empty output. Before treating empty output as evidence, print the number of files scanned as well.

20.1 The build in a clean container

The 3.19.0 tarball was built this way on 2026-09-30, on build-host, inside a fresh ubuntu:24.04 container: the same steps as above, run as root without sudo, with the packages a bare ubuntu:24.04 image does not carry added to the install line as they were run. A clean container rules out anything a long-lived build machine has collected. make deps builds the runtime the release names in its .requirements (apisix-runtime 1.3.18 for 3.19.0; OpenResty stays 1.29.2.4); the build took about 27 minutes.

mkdir -p ~/g8-rt-3.19/out
tee ~/g8-rt-3.19/build.sh >/dev/null <<'EOF'
#!/bin/bash
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y sudo build-essential cmake curl git libldap2-dev libpcre3-dev \
  libssl-dev libyaml-dev libyaml-0-2 lua5.1 luarocks unzip zlib1g-dev \
  wget lsb-release gnupg patch software-properties-common ca-certificates
git clone --depth 1 --branch 3.19.0 https://github.com/apache/apisix.git /usr/local/src/apisix
cd /usr/local/src/apisix
make deps
make install ENV_INST_LUADIR=/usr/local/apisix ENV_INST_BINDIR=/usr/local/bin
cp -a /usr/local/src/apisix/deps /usr/local/apisix/deps
S=/var/tmp/g8-apisix-runtime
rm -rf "$S"; mkdir -p "$S/usr/local/bin" "$S/usr/bin"
cp -a /usr/local/apisix /usr/local/openresty "$S/usr/local/"
cp -a /usr/local/bin/apisix "$S/usr/local/bin/"
cp -a /usr/bin/openresty "$S/usr/bin/"
find "$S" -name '*.a' -delete
rm -rf "$S/usr/local/openresty/pod" "$S/usr/local/openresty/nginx/sbin/nginx.old"
rm -f "$S/usr/local/apisix/logs/"* "$S/usr/local/apisix/conf/nginx.conf" "$S/usr/local/apisix/conf/apisix.uid"
install -m0644 /usr/local/src/apisix/conf/config.yaml "$S/usr/local/apisix/conf/config.yaml"
{ echo "apisix: $(cd /usr/local/apisix && /usr/local/bin/apisix version 2>&1 | tail -1)"; /usr/local/openresty/bin/openresty -V 2>&1 | head -1; grep APISIX_RUNTIME /usr/local/src/apisix/.requirements; } > /out/versions.txt
tar -czf /out/apisix-runtime-3.19.0-ubuntu24.04.tar.gz -C "$S" .
sha256sum /out/apisix-runtime-3.19.0-ubuntu24.04.tar.gz > /out/sha256.txt
echo BUILD_DONE
EOF
cd ~/g8-rt-3.19
docker run -d --name g8-rt-build -v ~/g8-rt-3.19/out:/out -v ~/g8-rt-3.19/build.sh:/build.sh:ro ubuntu:24.04 bash /build.sh
docker logs -f g8-rt-build

BUILD_DONE is the last log line of a finished build. out/versions.txt records the APISIX version, the OpenResty version and APISIX_RUNTIME; out/sha256.txt the tarball's digest — for 3.19.0 8a75c1ec34f1f8cc28c88e6261af5eae844852fd9d8bd533222bb682ee6c4c0f.

The single-layer artifact is an image built FROM scratch whose only instruction adds the tarball at /; Docker unpacks the tarball into that one layer, which is the layer section 4 extracts. The last command must print layers 1.

cd ~/g8-rt-3.19
printf "FROM scratch\nADD out/apisix-runtime-3.19.0-ubuntu24.04.tar.gz /\n" > Dockerfile
docker build --provenance=false --sbom=false -t harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04 .
docker push harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04
docker manifest inspect harbor.g8.dev/infra/apisix-runtime:3.19.0-ubuntu24.04 | python3 -c "import json,sys;m=json.load(sys.stdin);print('layers',len(m['layers']))"
docker rm g8-rt-build

The clean-machine exam above still applies: the 3.19.0 artifact was extracted on this document's VM by section 16 and the sweep scanned 492 files with no MISSING: line.