Skip to main content

OKD environment — 3 APISIX + 3 etcd

The host names, addresses and the registry on this page are examples. Our lab run used its own values in their place; replace them with yours, and take the registry address and its credentials from your G8 contact.

Three APISIX Deployment replicas and a three-member etcd StatefulSet on OKD/OpenShift. The installation runs under the restricted-v2 SCC with no SCC relaxation whatsoever: no anyuid, no extra grant to the ServiceAccount, no privileged.

This document was executed on OKD 4.19. The same manifests apply on OpenShift Container Platform; the difference is which registry the images come from.

A single-node cluster with three replicas does NOT prove fault tolerance

This installation was verified on a single-node OKD (SNO). Three replicas verify the shape of the manifests: StatefulSet identities, peer addresses, the quorum arithmetic, the behaviour on member loss. When the node dies, all six pods go with it.

On a multi-node cluster podAntiAffinity must be changed from preferred to required — see section 16. That change cannot be tried on a single-node cluster: with required, only one of the three replicas is scheduled.

1. What we are installing​

componentversionsource
APISIX3.19.0-debianharbor.g8.dev/infra/apisix:3.19.0-debian
etcdv3.5.17harbor.g8.dev/infra/etcd:v3.5.17
objectkindnote
etcdStatefulSet (3) + headless ServiceEach member has its own PVC
apisixDeployment (3) + 3 ServicesAll three read the same ConfigMap
apisixRouteData plane only; the Admin API is not exposed
etcd-clients-onlyNetworkPolicyDecides who may reach etcd
etcd-backupCronJob + PVCDaily snapshot

2. Prerequisites​

requirementvaluenote
OKD / OpenShift4.19The version this was verified on
Storage classA StorageClass supporting RWOHere topolvm-provisioner
Harbor accessThe cluster must be able to pull from HarborVerified below
ocLogged in as cluster adminNamespaces and NetworkPolicies are created

Harbor is signed by our own CA; if the cluster does not trust that CA, no pod can start. Verify the trust is in place:

oc get image.config.openshift.io/cluster -o jsonpath='{.spec.additionalTrustedCA}{"\n"}'
{"name":"g8-lab-registry-ca"}

If it comes back empty the CA is not configured and image pulls fail with x509: certificate signed by unknown authority.

2.1 Ports and access​

portcomponenthow it is reached
9080APISIX data planeapisix Service + Route (from outside the cluster)
9180APISIX Admin APIapisix-admin Service, in-cluster only. No Route
9091APISIX Prometheusapisix-metrics Service, in-cluster only
9090APISIX control APIPod-local 127.0.0.1 only. The probes target this
2379 2380etcd client and peerHeadless Service, restricted by NetworkPolicy

The Admin API deliberately has no Route. If it must be managed from outside the cluster, use oc port-forward; opening a permanent Route moves the Admin API outside the cluster.

3. Namespace​

oc create namespace g8-gateway

OpenShift assigns a uid range to every namespace; pods run as a random uid from that range. The range can be read:

oc get ns g8-gateway -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.uid-range}{"\n"}'
1000770000/10000

4. Verifying that images can be pulled from Harbor​

Do not skip this step: a pull failure only surfaces as ImagePullBackOff after the StatefulSet and Deployment are applied, and diagnosing it gets confused with configuration errors.

oc -n g8-gateway create deployment pulltest \
  --image=harbor.g8.dev/infra/apisix:3.19.0-debian -- sleep 120
oc -n g8-gateway rollout status deploy/pulltest --timeout=180s
oc -n g8-gateway delete deploy pulltest
deployment "pulltest" successfully rolled out

5. The etcd cluster​

ETCD_NAME, ETCD_ADVERTISE_CLIENT_URLS and ETCD_INITIAL_ADVERTISE_PEER_URLS are derived from the pod name. The $(POD_NAME) substitution is done by kubelet, not by a shell; the etcd image contains no shell (15.4).

podManagementPolicy: Parallel is mandatory — with the default OrderedReady the cluster never forms at all (15.1).

publishNotReadyAddresses: true is mandatory too: the members must be able to resolve each other while none of them is Ready yet.

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: etcd
  ports:
    - name: client
      port: 2379
    - name: peer
      port: 2380
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  serviceName: etcd
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: etcd
                topologyKey: kubernetes.io/hostname
      containers:
        - name: etcd
          image: harbor.g8.dev/infra/etcd:v3.5.17
          command: ["/usr/local/bin/etcd"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ETCD_NAME
              value: "$(POD_NAME)"
            - name: ETCD_DATA_DIR
              value: /etcd-data
            - name: ETCD_LISTEN_PEER_URLS
              value: http://0.0.0.0:2380
            - name: ETCD_LISTEN_CLIENT_URLS
              value: http://0.0.0.0:2379
            - name: ETCD_ADVERTISE_CLIENT_URLS
              value: http://$(POD_NAME).etcd:2379
            - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
              value: http://$(POD_NAME).etcd:2380
            - name: ETCD_INITIAL_CLUSTER
              value: etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380
            - name: ETCD_INITIAL_CLUSTER_STATE
              value: new
            - name: ETCD_INITIAL_CLUSTER_TOKEN
              value: g8-okd
            - name: ETCD_AUTO_COMPACTION_MODE
              value: periodic
            - name: ETCD_AUTO_COMPACTION_RETENTION
              value: "1"
            - name: ETCD_QUOTA_BACKEND_BYTES
              value: "2147483648"
            - name: ETCD_SNAPSHOT_COUNT
              value: "10000"
            - name: ETCD_HEARTBEAT_INTERVAL
              value: "250"
            - name: ETCD_ELECTION_TIMEOUT
              value: "2500"
          ports:
            - name: client
              containerPort: 2379
            - name: peer
              containerPort: 2380
          readinessProbe:
            httpGet:
              path: /readyz
              port: 2379
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 10
          livenessProbe:
            httpGet:
              path: /livez
              port: 2379
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          volumeMounts:
            - name: data
              mountPath: /etcd-data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: topolvm-provisioner
        resources:
          requests:
            storage: 8Gi
EOF
oc -n g8-gateway rollout status sts/etcd --timeout=300s

6. Admin key​

APISIX's documented default admin key (edd1c9f0…) is public knowledge. The key is kept in a Secret and passed to the pod as an environment variable; it is never written by hand into the ConfigMap.

oc -n g8-gateway create secret generic apisix-admin-key \
  --from-literal=G8_APISIX_ADMIN_KEY="$(openssl rand -hex 24)"

7. APISIX​

In a container environment logs go to stdout/stderr, not to files: kubelet collects and rotates them (section 16). Writing to files grows the container's ephemeral layer without bound and destroys the log when the pod is recreated.

The probes target the control port 9090, not 9080; a bare TCP connection to 9080 puts a "status":"400" line into the access log.

In the manifest below, the Route's host: line carries this lab's apps domain. Change it to your own cluster's before applying; a Route created with the wrong domain answers nothing from outside:

oc get ingress.config/cluster -o jsonpath='{.spec.domain}{"\n"}'
cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-config
  namespace: g8-gateway
data:
  config.yaml: |
    apisix:
      node_listen:
        - 9080

    deployment:
      role: traditional
      role_traditional:
        config_provider: etcd
      admin:
        admin_listen:
          ip: 0.0.0.0
          port: 9180
        admin_key:
          - name: admin
            key: "${{G8_APISIX_ADMIN_KEY}}"
            role: admin
        allow_admin:
          - 10.128.0.0/14
          - 169.254.0.0/17
          - 127.0.0.0/8
      etcd:
        host:
          - "http://etcd-0.etcd:2379"
          - "http://etcd-1.etcd:2379"
          - "http://etcd-2.etcd:2379"
        prefix: /apisix
        timeout: 30

    nginx_config:
      error_log: /dev/stderr
      error_log_level: warn
      http:
        access_log: /dev/stdout
        access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","status":"$status","method":"$request_method","uri":"$uri","rt":"$request_time","ua":"$http_user_agent"}'
        access_log_format_escape: json

    plugin_attr:
      prometheus:
        export_addr:
          ip: 0.0.0.0
          port: 9091
        metrics:
          http_status:
            extra_labels:
              - route_name: $route_name
              - service_name: $service_name
          http_latency:
            extra_labels:
              - route_name: $route_name
              - service_name: $service_name
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: apisix
  template:
    metadata:
      labels:
        app: apisix
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: apisix
                topologyKey: kubernetes.io/hostname
      containers:
        - name: apisix
          image: harbor.g8.dev/infra/apisix:3.19.0-debian
          envFrom:
            - secretRef:
                name: apisix-admin-key
          ports:
            - name: http
              containerPort: 9080
            - name: admin
              containerPort: 9180
            - name: metrics
              containerPort: 9091
          readinessProbe:
            exec:
              command: ["bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            exec:
              command: ["bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "2"
              memory: 1Gi
          volumeMounts:
            - name: config
              mountPath: /usr/local/apisix/conf/config.yaml
              subPath: config.yaml
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: apisix-config
---
apiVersion: v1
kind: Service
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: http
      port: 9080
      targetPort: 9080
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-admin
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: admin
      port: 9180
      targetPort: 9180
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-metrics
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: metrics
      port: 9091
      targetPort: 9091
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  host: gw.apps.okd.example.com
  to:
    kind: Service
    name: apisix
  port:
    targetPort: http
EOF
oc -n g8-gateway rollout status deploy/apisix --timeout=300s

7.1 PodDisruptionBudget​

When Kubernetes drains a node (maintenance, upgrade, autoscaling) it evicts the pods. By default there is no limit: when measured, all three etcd pods could be evicted back to back and each returned 201 Success — meaning a single maintenance operation can take quorum with it.

A PodDisruptionBudget stops that at the API level:

cat <<'EOF' | oc apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: etcd
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: apisix
EOF

Verification:

oc -n g8-gateway get pdb
NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
apisix   2               N/A               1
etcd     2               N/A               1

The ALLOWED DISRUPTIONS column is the real answer: one pod may be evicted, a second may not. Measured:

1. tahliye (etcd-0): 201 Success        -> izinli kesinti 1 -> 0
2. tahliye (etcd-1): TooManyRequests: Cannot evict pod as it
                     would violate the pod's disruption budget

This blocks voluntary evictions; if a node crashes, the PDB does not come into play. What it protects against is maintenance-induced outage.

8. NetworkPolicy​

etcd has no authentication. Without a NetworkPolicy, any pod in the cluster can connect to etcd-0.etcd:2379 and read or rewrite the gateway's entire configuration. This policy narrows access down to three labels.

cat <<'EOF' | oc apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: etcd-clients-only
  namespace: g8-gateway
spec:
  podSelector:
    matchLabels:
      app: etcd
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: apisix
        - podSelector:
            matchLabels:
              app: etcd
        - podSelector:
            matchLabels:
              app: etcd-backup
      ports:
        - protocol: TCP
          port: 2379
        - protocol: TCP
          port: 2380
EOF
allow_admin is not access control in Kubernetes

APISIX's allow_admin list looks at the source IP, but the SDN may rewrite that address: behind a Service the same client appears sometimes with its own pod IP and sometimes with the OVN masquerade address (15.3). The real control is the NetworkPolicy; allow_admin is only a coarse second door.

9. Backup​

The snapshot is written to a separate PVC. The filename is made unique with $(POD_NAME) — date cannot be called because the etcd image has no shell (15.4). Pruning old files falls to an initContainer that does have a shell.

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup
  namespace: g8-gateway
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: topolvm-provisioner
  resources:
    requests:
      storage: 5Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: g8-gateway
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        metadata:
          labels:
            app: etcd-backup
        spec:
          restartPolicy: Never
          initContainers:
            - name: prune
              image: harbor.g8.dev/infra/apisix:3.19.0-debian
              command:
                - bash
                - -c
                - "ls -1t /backups/etcd-*.db 2>/dev/null | tail -n +15 | xargs -r rm -f; echo pruned"
              volumeMounts:
                - name: backups
                  mountPath: /backups
          containers:
            - name: snapshot
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args:
                - "--endpoints=http://etcd-0.etcd:2379"
                - "snapshot"
                - "save"
                - "/backups/etcd-$(POD_NAME).db"
              env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              volumeMounts:
                - name: backups
                  mountPath: /backups
          volumes:
            - name: backups
              persistentVolumeClaim:
                claimName: etcd-backup
EOF

snapshot save requires a single endpoint; passing several returns snapshot must be requested to one selected node.

The backups sit on the same node as the data they protect

The etcd-backup PVC is on the same node, on the same disk, as the etcd data itself. If the node or the disk goes, the backup goes with it — so this backup does not protect against the event it promises to.

This installation does not move backups off the cluster; a customer environment must. The direction matters: the remote system pulls, and the gateway should hold no write credential on it — anyone who takes the gateway would use that credential to delete the backups too. A working example for the virtual-machine installs is in 02-test-vm §13.3.

9.1 Defragmentation​

Compaction removes old revisions logically but does not shrink the database file — the freed pages stay inside etcd and never return to the filesystem. Since the quota watches dbSize, the file keeps growing even when compaction is working perfectly, and the quota fills sooner or later. The extreme measured: dbSizeInUse at 41 kB while dbSize sat at 98 MB.

Defragmentation therefore has to be scheduled. Weekly is enough.

cat <<'EOF' | oc apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-defrag
  namespace: g8-gateway
spec:
  schedule: "0 3 * * 0"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 3600
      template:
        metadata:
          labels:
            app: etcd-backup
        spec:
          restartPolicy: Never
          initContainers:
            - name: defrag-0
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-0.etcd:2379", "--command-timeout=900s", "defrag"]
            - name: defrag-1
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-1.etcd:2379", "--command-timeout=900s", "defrag"]
            - name: defrag-2
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-2.etcd:2379", "--command-timeout=900s", "defrag"]
          containers:
            - name: alarms
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379", "--command-timeout=30s", "alarm", "list"]
EOF

Verification:

oc -n g8-gateway create job defrag-now --from=cronjob/etcd-defrag
oc -n g8-gateway wait --for=condition=complete job/defrag-now --timeout=600s
POD=$(oc -n g8-gateway get pod -l job-name=defrag-now -o jsonpath='{.items[0].metadata.name}')
for c in defrag-0 defrag-1 defrag-2; do oc -n g8-gateway logs $POD -c $c | tail -1; done
oc -n g8-gateway delete job defrag-now
Finished defragmenting etcd member[http://etcd-0.etcd:2379]
Finished defragmenting etcd member[http://etcd-1.etcd:2379]
Finished defragmenting etcd member[http://etcd-2.etcd:2379]

Defrag blocks that member briefly, which is why the members are processed one at a time. Defragmenting all three at once drops quorum.

Defrag temporarily needs free space equal to the database size; that is one reason the quota must be meaningfully smaller than the disk.

10. Verification​

10.1 SCC and the assigned uid​

That the installation runs without SCC relaxation is read from the annotation on the pod. Any value other than restricted-v2 means the installation has not been verified in a restricted environment.

oc -n g8-gateway get pod -o custom-columns=\
'POD:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc,UID:.spec.containers[0].securityContext.runAsUser' \
  --no-headers | sort
apisix-799d857b7f-6lt62   restricted-v2   1000770000
apisix-799d857b7f-9gp6h   restricted-v2   1000770000
apisix-799d857b7f-v7tqv   restricted-v2   1000770000
etcd-0                    restricted-v2   1000770000
etcd-1                    restricted-v2   1000770000
etcd-2                    restricted-v2   1000770000

Pod names and the uid vary per installation; the restricted-v2 column must not.

Do not test the SCC with oc run

A pod you create directly is admitted with your own privileges. As a cluster admin you get anyuid and the image runs as its own uid — which proves nothing about a restricted environment. Details: 15.2.

10.2 The etcd cluster​

The member count, the healthy-endpoint count and the leader count are each counted separately.

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
echo "members=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E member list -w simple | wc -l)"
echo "healthy=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint health -w simple 2>&1 | grep -c 'is healthy')"
echo "leaders=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint status -w simple | awk -F, '$5 ~ /true/' | wc -l)"
members=3
healthy=3
leaders=1

10.3 Admin API and key substitution​

The Admin API has no Route; access is opened with oc port-forward. Three requests: the real key, the unsubstituted raw template, and no key.

oc -n g8-gateway port-forward svc/apisix-admin 19180:9180 >/dev/null 2>&1 &
PF=$!
sleep 6
KEY=$(oc -n g8-gateway get secret apisix-admin-key -o jsonpath='{.data.G8_APISIX_ADMIN_KEY}' | base64 -d)
umask 077
printf 'header = "X-API-KEY: %s"\n' "$KEY" > /tmp/g8adm.conf
printf 'header = "X-API-KEY: ${{G8_APISIX_ADMIN_KEY}}"\n' > /tmp/g8raw.conf
curl -s -o /dev/null -w "real=%{http_code}\n" -K /tmp/g8adm.conf http://127.0.0.1:19180/apisix/admin/routes
curl -s -o /dev/null -w "raw=%{http_code}\n"  -K /tmp/g8raw.conf http://127.0.0.1:19180/apisix/admin/routes
curl -s -o /dev/null -w "none=%{http_code}\n"                    http://127.0.0.1:19180/apisix/admin/routes
real=200
raw=401
none=401
10.4 continues in this shell session

The next section uses the $PF variable set here, the /tmp/g8adm.conf file, and the port-forward that is still running. Open a new terminal and 10.4 fails with a connection error. Stay in the same shell.

10.4 Does the configuration propagate to all three pods​

The route is written once and read back from each of the three pods separately. Reading through the Service is not enough: if load balancing lands on the same pod, the other two are never tested.

curl -s -o /dev/null -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
  -X PUT http://127.0.0.1:19180/apisix/admin/routes/g8-smoke \
  -H "Content-Type: application/json" \
  -d '{"uri":"/smoke","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"G8-GW-OKD-OK"}}}'
kill $PF 2>/dev/null
sleep 3
for p in $(oc -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[*].metadata.name}'); do
  oc -n g8-gateway port-forward pod/$p 19081:9080 >/dev/null 2>&1 &
  PF=$!
  sleep 4
  echo "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:19081/smoke) $(curl -s http://127.0.0.1:19081/smoke)"
  kill $PF 2>/dev/null
  sleep 1
done
PUT=201
200 G8-GW-OKD-OK
200 G8-GW-OKD-OK
200 G8-GW-OKD-OK

10.5 The Route and the source IP​

curl -s http://gw.apps.okd.example.com/smoke -w "\nstatus=%{http_code}\n"
G8-GW-OKD-OK
status=200

Which address a request through the Route arrives with is read from the access log. This is the only truth that matters when configuring IP-based plugins (ip-restriction, real-ip):

oc -n g8-gateway logs -l app=apisix --tail=50 --prefix=false \
  | grep '"uri":"/smoke"' | sort | tail -1
{"time":"2026-08-24T21:00:48+00:00","remote_addr":"10.128.0.2","status":"200","method":"GET","uri":"/smoke","rt":"0.000","ua":"curl/8.14.1"}

The sort is mandatory: oc logs -l concatenates the logs of several pods and the order is not chronological; taking tail -1 without sorting gives an old line from a different pod. The JSON line starts with the time field, so lexical order is chronological order.

remote_addr is not the client's address; it is the OVN gateway's. If the real client IP is needed, add haproxy.router.openshift.io/set-forwarded-headers to the Route and configure the real-ip plugin in APISIX — this document does not cover that.

10.6 NetworkPolicy​

That the policy works is measured with both controls. Seeing "blocked" alone is not enough: the pod may simply never have started.

This step prints a PodSecurity warning — it is not a rejection

The Pods here and in 10.9 are created directly, not by a controller. The namespace's restricted PodSecurity profile inspects them too, and since the manifest carries no security context it prints a long warning:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
pod/npdeny created

The Warning: line is followed by created and the pod reaches Succeeded. PodSecurity is in warn mode here; the warning is not a refusal. Do not add a securityContext to the manifest — these are throwaway test pods, not part of the installation.

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: npdeny
  namespace: g8-gateway
  labels:
    app: outsider
spec:
  restartPolicy: Never
  containers:
    - name: t
      image: harbor.g8.dev/infra/apisix:3.19.0-debian
      command: ["bash","-c","timeout 8 bash -c 'echo > /dev/tcp/etcd-0.etcd/2379' && echo REACHABLE || echo BLOCKED"]
EOF
oc -n g8-gateway wait --for=jsonpath='{.status.phase}'=Succeeded pod/npdeny --timeout=120s >/dev/null
echo "unlabelled=$(oc -n g8-gateway logs npdeny)"
oc -n g8-gateway delete pod npdeny
A=$(oc -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[0].metadata.name}')
echo "apisix=$(oc -n g8-gateway exec $A -- bash -c "timeout 8 bash -c 'echo > /dev/tcp/etcd-0.etcd/2379' && echo REACHABLE || echo BLOCKED")"
unlabelled=BLOCKED
apisix=REACHABLE

10.7 Quorum: one member down​

The leader pod is deleted. Expected: the cluster elects a new leader, writes continue, the data plane is unaffected, and the StatefulSet brings the pod back.

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
LEADER=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint status -w simple \
  | awk -F, '$5 ~ /true/ {print $1}' | sed 's|http://||;s|\.etcd:2379||')
oc -n g8-gateway delete pod $LEADER --wait=false
sleep 15
SURVIVOR=$(oc -n g8-gateway get pod -l app=etcd --no-headers | grep '1/1' | head -1 | awk '{print $1}')
oc -n g8-gateway exec $SURVIVOR -- etcdctl --endpoints=http://127.0.0.1:2379 put /g8-quorum-probe ok
curl -s http://gw.apps.okd.example.com/smoke -w "\nstatus=%{http_code}\n"
oc -n g8-gateway rollout status sts/etcd --timeout=300s
oc -n g8-gateway exec etcd-2 -- etcdctl --endpoints=$E get /g8-quorum-probe --print-value-only
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E del /g8-quorum-probe
OK
G8-GW-OKD-OK
status=200
partitioned roll out complete: 3 new pods have been updated...
ok
1

10.8 Quorum loss​

Scaling the members down to one leaves no quorum. The expected behaviour has two parts and both must be measured: the write fails, and the data plane keeps serving the existing routes.

oc -n g8-gateway scale sts etcd --replicas=1
sleep 40
if oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=15s put /g8-noquorum x >/dev/null 2>&1
then echo "write=UNEXPECTED-SUCCESS"; else echo "write=FAILED"; fi
curl -s -o /dev/null --max-time 10 -w "route=%{http_code}\n" http://gw.apps.okd.example.com/smoke
echo "apisix_ready=$(oc -n g8-gateway get pod -l app=apisix --no-headers | grep -c '1/1')"
echo "etcd_restarts=$(oc -n g8-gateway get pod etcd-0 -o jsonpath='{.status.containerStatuses[0].restartCount}')"
write=FAILED
route=200
apisix_ready=3
etcd_restarts=0

No error string is printed here: the string etcd returns varies from run to run — three independent measurements produced three different ones. What is asserted instead is that the write failed, which is the actual question.

etcd_restarts=0 is a regression sentinel. Anything above zero means the liveness probe is watching cluster health and killing the last member on quorum loss (15.8).

The APISIX pods stay ready because their probes target the control port, not etcd.

The cluster is brought back:

oc -n g8-gateway scale sts etcd --replicas=3
oc -n g8-gateway rollout status sts/etcd --timeout=300s
E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
echo "healthy=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint health -w simple 2>&1 | grep -c 'is healthy')"
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E put /g8-recovered ok
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E del /g8-recovered
partitioned roll out complete: 3 new pods have been updated...
healthy=3
OK
1

10.9 Is the backup valid​

That the CronJob exists does not prove a backup was taken. It is run once by hand and the file it produces is verified.

oc -n g8-gateway create job manual-backup --from=cronjob/etcd-backup
oc -n g8-gateway wait --for=condition=complete job/manual-backup --timeout=300s
POD=$(oc -n g8-gateway get pod -l job-name=manual-backup -o jsonpath='{.items[0].metadata.name}')
oc -n g8-gateway logs $POD -c snapshot 2>&1 | grep -o '/backups/[^"]*\.db' | tail -1
oc -n g8-gateway delete job manual-backup
/backups/etcd-manual-backup-64wnx.db

The filename varies per job run. That the file really is a readable snapshot is tested separately — replace the name below with the one from your own output:

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: snapcheck
  namespace: g8-gateway
  labels:
    app: etcd-backup
spec:
  restartPolicy: Never
  containers:
    - name: c
      image: harbor.g8.dev/infra/etcd:v3.5.17
      command: ["/usr/local/bin/etcdutl"]
      args: ["snapshot","status","/backups/etcd-manual-backup-64wnx.db","-w","table"]
      volumeMounts:
        - name: backups
          mountPath: /backups
  volumes:
    - name: backups
      persistentVolumeClaim:
        claimName: etcd-backup
EOF
oc -n g8-gateway wait --for=jsonpath='{.status.phase}'=Succeeded pod/snapcheck --timeout=180s >/dev/null
oc -n g8-gateway logs snapcheck
oc -n g8-gateway delete pod snapcheck
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| ab038d9e |       86 |         93 |      37 kB |
+----------+----------+------------+------------+

HASH, REVISION and the key count vary per installation; the result being looked for is that the command prints a table without erroring.

10.10 Cleaning up verification leftovers​

10.3 writes the admin key into /tmp/g8adm.conf in cleartext. If the file is not deleted the key stays on disk, and the care section 6 takes to keep the key in a Secret is wasted here.

rm -f /tmp/g8adm.conf /tmp/g8raw.conf
ls /tmp/g8*.conf 2>/dev/null | wc -l
0

The g8-smoke route is left in place deliberately: it is lasting proof that the installation works, and the uninstall in section 13 removes it along with the namespace. On a cluster that will carry production traffic, delete it through the Admin API if you do not want it there.

11. Observability​

This environment gets its own Elasticsearch and Prometheus, both running inside the cluster. Request records go to Elasticsearch, request metrics to Prometheus.

componentversionsource
Elasticsearch9.5.2harbor.g8.dev/infra/elasticsearch:9.5.2
Prometheus3.15.0harbor.g8.dev/infra/prometheus:v3.15.0

Prometheus starts with --enable-feature=promql-extended-range-selectors: Analytics counts each bucket exactly with the anchored range selector, which Prometheus 3.15 takes only behind that flag. From Prometheus 3.16 the selector is on by default and the line can go; without it, Analytics shows its counts as estimates.

Prometheus finds the APISIX pods through Kubernetes service discovery, so the configuration does not change when the pod count does. external_labels marks the environment.

11.1 Elasticsearch​

observability-es.yaml
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  selector:
    app: elasticsearch
  ports:
    - name: http
      port: 9200
      targetPort: 9200
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: elasticsearch
          image: harbor.g8.dev/infra/elasticsearch:9.5.2
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          env:
            - name: node.name
              value: es-okd
            - name: cluster.name
              value: g8-okd
            - name: discovery.type
              value: single-node
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.security.enrollment.enabled
              value: "false"
            - name: xpack.security.http.ssl.enabled
              value: "false"
            - name: xpack.security.transport.ssl.enabled
              value: "false"
            - name: ingest.geoip.downloader.enabled
              value: "false"
            - name: bootstrap.memory_lock
              value: "false"
            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: g8-es-credentials
                  key: password
          ports:
            - name: http
              containerPort: 9200
          resources:
            requests:
              cpu: 300m
              memory: 1600Mi
            limits:
              cpu: "2"
              memory: 2200Mi
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - curl -sf -u "elastic:$ELASTIC_PASSWORD" http://127.0.0.1:9200/_cluster/health?local=true >/dev/null
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 20
          livenessProbe:
            tcpSocket:
              port: 9200
            initialDelaySeconds: 120
            periodSeconds: 20
            failureThreshold: 6
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: topolvm-provisioner
        resources:
          requests:
            storage: 20Gi

11.2 Prometheus and NetworkPolicy​

observability-prometheus.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: g8-gateway
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
      external_labels:
        g8_environment: okd
    scrape_configs:
      - job_name: apisix
        metrics_path: /apisix/prometheus/metrics
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names: [g8-gateway]
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: apisix
          - source_labels: [__address__]
            action: replace
            regex: '([^:]+)(:\d+)?'
            replacement: '$1:9091'
            target_label: __address__
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - target_label: g8_environment
            replacement: okd
      - job_name: prometheus
        static_configs:
          - targets: ["127.0.0.1:9090"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: g8-gateway
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus
  namespace: g8-gateway
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "endpoints"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus
  namespace: g8-gateway
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: g8-gateway
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: g8-gateway
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: topolvm-provisioner
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: prometheus
          image: harbor.g8.dev/infra/prometheus:v3.15.0
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus
            - --storage.tsdb.retention.time=40d
            - --storage.tsdb.retention.size=5GB
            - --enable-feature=promql-extended-range-selectors
          ports:
            - name: http
              containerPort: 9090
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: data
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: data
          persistentVolumeClaim:
            claimName: prometheus-data
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: elasticsearch-clients-only
  namespace: g8-gateway
spec:
  podSelector:
    matchLabels:
      app: elasticsearch
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: apisix
        - podSelector:
            matchLabels:
              app: prometheus
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-ingress
      ports:
        - protocol: TCP
          port: 9200

The NetworkPolicy lets only APISIX, Prometheus and the ingress controller reach Elasticsearch.

The ingress controller's namespace must be in the NetworkPolicy

Leave openshift-ingress out of the list and the Route/Ingress looks healthy while requests never reach Elasticsearch. The symptom is a timeout, not a clear rejection.

11.3 Exposing​

observability-expose.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  host: es.apps.okd.example.com
  to:
    kind: Service
    name: elasticsearch
  port:
    targetPort: http
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  host: prom.apps.okd.example.com
  to:
    kind: Service
    name: prometheus
  port:
    targetPort: http

11.4 Install​

The password is generated in the cluster and reaches neither this document nor the repository. It is the store administrator's password and no rule of the gateway carries it: the gateway writes with an account of its own (11.6).

oc -n g8-gateway create secret generic g8-es-credentials \
  --from-literal=password="$(openssl rand -base64 24 | tr -d /+= | head -c 24)"

oc apply -f observability-es.yaml
oc apply -f observability-prometheus.yaml
oc apply -f observability-expose.yaml

oc -n g8-gateway wait --for=condition=Ready pod/elasticsearch-0 --timeout=300s
oc -n g8-gateway rollout status deploy/prometheus --timeout=300s
Wait for readiness with wait

After delete pod --wait=false a loop that polls readiness can read the stale pod object, call it ready and exit early. One verification reported data loss because of exactly this; the real cause was that the pod had not come up yet.

11.5 Wiring APISIX​

The metrics are enabled through a global rule, which leaves prefer_name off: route and service on the metrics carry the ids Analytics counts by, and metrics.*.extra_labels in config.yaml put each route's and service's name beside them as route_name and service_name. The request records are not wired here: g8 writes their rule itself when traffic logging is turned on (11.7).

KEY=$(oc -n g8-gateway get secret apisix-admin-key -o jsonpath='{.data.G8_APISIX_ADMIN_KEY}' | base64 -d)

oc -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  "curl -s -o /dev/null -w '%{http_code}\n' -H 'X-API-KEY: $KEY' \
   -X PUT http://apisix-admin:9180/apisix/admin/global_rules/1 \
   -H 'Content-Type: application/json' --data-binary @-" <<'EOF'
{"plugins":{"prometheus":{"prefer_name":false}}}
EOF
201
The prometheus plugin must be enabled; plugin_attr is not enough

plugin_attr.prometheus in config.yaml only sets the address the metrics are published on. Unless the plugin is enabled on a global rule, no request metrics are collected at all. The symptom is treacherous: the Prometheus target reads up, /apisix/prometheus/metrics returns 200, and yet a query for apisix_http_status comes back with zero series.

11.6 Accounts, the ingest pipeline and the index template in the log store​

Unverified

The commands of this section were run on a development stack on 2026-10-02 (APISIX 3.19.0, Elasticsearch 8.15.3) with the outputs shown. On Elasticsearch 9.5.2 the privileges were measured with roles that name the days by a wildcard, and without the line that reads the template; the roles below name the days by a regular expression, which was measured on 8.15.3 alone. The commands have not been run on this machine yet.

The ingest pipeline, the line of the server's role that reads it and the move of 14.2 were run on 2026-10-04 against a development stack's Elasticsearch 8.15.3 with the outputs shown; that a batch sent twice is stored once was also measured in the lab on Elasticsearch 9.5.2 and OpenSearch 3.9.0.

Two accounts are opened in Elasticsearch, and neither of them is elastic:

accountused bymay
g8-okd-gatewaythe gateway, to deliver the request recordswrite records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster
g8-okd-serverthe g8 server, to show the records and to clean them upread the state of the cluster and g8's ingest pipeline, read the daily indices of this environment and delete a day by its name

The daily indices carry the slug of the environment in g8 (g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is registered under in the g8 console. The roles name the days by a regular expression and not by g8-okd-access-*: a wildcard would also reach the days of an environment whose slug begins with okd-access.

The passwords of the two accounts are generated into a Secret; they reach neither this document nor the repository:

oc -n g8-gateway create secret generic g8-es-accounts \
  --from-literal=gateway="$(openssl rand -base64 24 | tr -d /+= | head -c 24)" \
  --from-literal=server="$(openssl rand -base64 24 | tr -d /+= | head -c 24)"
ENV=okd
GWP=$(oc -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.gateway}' | base64 -d)
SVP=$(oc -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.server}' | base64 -d)
put() {
  oc -n g8-gateway exec -i elasticsearch-0 -- sh -c \
    "curl -s -o /dev/null -w '$1 -> %{http_code}\n' -u elastic:\$ELASTIC_PASSWORD \
     -X PUT http://127.0.0.1:9200$1 -H 'Content-Type: application/json' --data-binary @-"
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_security/role/g8-$ENV-gateway <<EOF
{"indices":[{"names":["$DAYS"],"privileges":["index","auto_configure"]}]}
EOF
put /_security/role/g8-$ENV-server <<EOF
{"cluster":["monitor","read_pipeline"],"indices":[{"names":["$DAYS"],
 "privileges":["read","view_index_metadata","monitor","delete_index"]}]}
EOF
put /_security/user/g8-$ENV-gateway <<EOF
{"password":"$GWP","roles":["g8-$ENV-gateway"]}
EOF
put /_security/user/g8-$ENV-server <<EOF
{"password":"$SVP","roles":["g8-$ENV-server"]}
EOF
/_security/role/g8-okd-gateway -> 200
/_security/role/g8-okd-server -> 200
/_security/user/g8-okd-gateway -> 200
/_security/user/g8-okd-server -> 200
The gateway's account must not be able to read

The password of g8-okd-gateway is part of the gateway's configuration: whoever can read the global rules of the gateway can obtain it. That is why this account can only add records. When traffic logging is turned on with a plain password, g8 tries the account and refuses one that can read the cluster or a record; a password given as a secret reference is resolved on the gateway and is not tried. create_doc is not enough for the account either: the plugin sends an index action, and with create_doc every record is refused while the request itself answers 200.

The ingest pipeline stores each record under its request id. The gateway's logger sends the records in batches and sends a batch again when the store answers it late, and by then the store has usually written the first one: under the request id, the second copy replaces the first instead of adding a record. The index template below sends every record through this pipeline, so the pipeline is applied first. A store that holds the template and not the pipeline refuses every record while the gateway's bulk request still answers 200; g8 checks that the store holds this version of the pipeline and does not turn traffic logging on without it. The store needs at least one node with the ingest role, which a node carries by default.

oc -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  'curl -s -u elastic:$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/_ingest/pipeline/g8-access \
   -H "Content-Type: application/json" --data-binary @-' <<'EOF'
{
  "description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
  "version": 1,
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{request_id}}}",
        "if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
      }
    }
  ]
}
EOF
{"acknowledged":true}

The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.

oc -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  'curl -s -u elastic:$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/_index_template/g8-access \
   -H "Content-Type: application/json" --data-binary @-' <<'EOF'
{
  "index_patterns": [
    "g8-*-access-*"
  ],
  "priority": 200,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "codec": "best_compression",
        "auto_expand_replicas": "0-1",
        "default_pipeline": "g8-access",
        "analysis": {
          "analyzer": {
            "g8_nodes": {
              "type": "pattern",
              "pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
              "lowercase": false
            }
          }
        }
      }
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "latency": {
          "type": "float"
        },
        "upstream_latency": {
          "type": "float"
        },
        "apisix_latency": {
          "type": "float"
        },
        "client_ip": {
          "type": "ip",
          "ignore_malformed": true
        },
        "route_id": {
          "type": "keyword"
        },
        "service_id": {
          "type": "keyword"
        },
        "upstream": {
          "type": "keyword",
          "ignore_above": 1024,
          "fields": {
            "node": {
              "type": "text",
              "analyzer": "g8_nodes",
              "norms": false,
              "index_options": "docs"
            }
          }
        },
        "consumer": {
          "properties": {
            "username": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "properties": {
            "method": {
              "type": "keyword"
            },
            "uri": {
              "type": "wildcard"
            },
            "url": {
              "type": "keyword",
              "index": false,
              "doc_values": false
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "querystring": {
              "type": "object",
              "enabled": false
            }
          }
        },
        "request_body": {
          "type": "text",
          "index": false
        },
        "response": {
          "properties": {
            "status": {
              "type": "short"
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "body": {
              "type": "text",
              "index": false
            }
          }
        },
        "server": {
          "properties": {
            "hostname": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "request_id": {
          "type": "keyword"
        },
        "route_name": {
          "type": "keyword"
        },
        "service_name": {
          "type": "keyword"
        },
        "upstream_status": {
          "type": "keyword"
        },
        "x_request_id": {
          "type": "keyword"
        },
        "logged_at": {
          "type": "date",
          "format": "epoch_second"
        }
      },
      "_meta": {
        "g8_records": 2
      },
      "_source": {
        "excludes": [
          "response.headers.set-cookie"
        ]
      }
    }
  }
}
EOF
{"acknowledged":true}

What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:

GWP=$(oc -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.gateway}' | base64 -d)
SVP=$(oc -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.server}' | base64 -d)
oc -n g8-gateway exec -i elasticsearch-0 -- sh -s "$GWP" "$SVP" <<'EOF'
curl -s -o /dev/null -w 'g8-okd-gateway reads the cluster      -> %{http_code}\n' -u "g8-okd-gateway:$1" http://127.0.0.1:9200/
curl -s -o /dev/null -w 'g8-okd-gateway reads records          -> %{http_code}\n' -u "g8-okd-gateway:$1" 'http://127.0.0.1:9200/g8-okd-access-*/_search'
curl -s -o /dev/null -w 'g8-okd-gateway writes a day           -> %{http_code}\n' -u "g8-okd-gateway:$1" -X POST http://127.0.0.1:9200/g8-okd-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-okd-gateway writes beside its days -> %{http_code}\n' -u "g8-okd-gateway:$1" -X POST http://127.0.0.1:9200/g8-okd-access-x-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-okd-server reads the template      -> %{http_code}\n' -u "g8-okd-server:$2" http://127.0.0.1:9200/_index_template/g8-access
curl -s -o /dev/null -w 'g8-okd-server reads the pipeline      -> %{http_code}\n' -u "g8-okd-server:$2" http://127.0.0.1:9200/_ingest/pipeline/g8-access
curl -s -o /dev/null -w 'g8-okd-server reads beside its days   -> %{http_code}\n' -u "g8-okd-server:$2" 'http://127.0.0.1:9200/g8-okd-access-x-access-2000.01.01/_search'
curl -s -o /dev/null -w 'g8-okd-server writes a record         -> %{http_code}\n' -u "g8-okd-server:$2" -X POST http://127.0.0.1:9200/g8-okd-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-okd-server deletes the day         -> %{http_code}\n' -u "g8-okd-server:$2" -X DELETE http://127.0.0.1:9200/g8-okd-access-2000.01.01
EOF
g8-okd-gateway reads the cluster      -> 403
g8-okd-gateway reads records          -> 403
g8-okd-gateway writes a day           -> 201
g8-okd-gateway writes beside its days -> 403
g8-okd-server reads the template      -> 200
g8-okd-server reads the pipeline      -> 200
g8-okd-server reads beside its days   -> 403
g8-okd-server writes a record         -> 403
g8-okd-server deletes the day         -> 200
Data streams cannot be used

elasticsearch-logger writes an index action in its bulk request; data streams require create and reject index. Hence one dated index per day.

11.7 Turning on the request records​

Unverified

These steps were walked on a development stack on 2026-10-02, with the two accounts of 11.6. They have not been walked against this machine yet.

The request records are turned on from the g8 console, once this gateway is registered there as the environment okd. No command on this machine writes the logger's rule.

  1. Environments › okd › Edit › Observability. Elasticsearch URL: the address the store is exposed at in 11.3; the NetworkPolicy of this namespace admits the g8 server through that address and no other. Username g8-okd-server, password the server key of the g8-es-accounts Secret. Test connection, then save.
  2. Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Index template Applied and Ingest pipeline Applied.
  3. Turn on traffic logging. Log store address, as the gateway reaches it: http://elasticsearch:9200. Username g8-okd-gateway, password the gateway key of the g8-es-accounts Secret. Turn on.

g8 writes one global rule, g8-request-records, beside rule 1: the logger with the daily index of this environment, the fields the records need and the default masks. What is recorded and masked is changed afterwards on the same Settings page. Where the gateway writes is shown on the Status card as Gateway writes to; Change address gives the logger another address and account for the same store, through the same checks of the account as turning on.

11.8 Retention​

A log store does not shrink on its own. With no limit the disk fills eventually, and past 95% Elasticsearch turns its indices read-only.

g8 deletes the request records: once a day it deletes, by its name, every daily index older than the number of days the environment keeps — 30 unless it is changed under Traffic Explorer › Settings › Retention › Change — and writes each deletion to the audit log. The store carries no lifecycle policy for these indices; a second deleter would delete days g8 still shows.

A day is named by its UTC date, by the gateway that writes it and by g8 that reads and deletes it.

On a single-node Elasticsearch a replica can never be allocated. The template sets auto_expand_replicas: 0-1: an index has no replica on one node and one as soon as a second node joins. The verification below must show 0 and green.

Prometheus is given two limits at once: retention.time=40d and retention.size=5GB. Whichever fills first does the cutting. Forty days keep the whole 30-day window of Analytics.

11.9 Verification​

Unverified

The output below was recorded on this machine while the rule of the earlier guide was in place; the index it shows is named by the machine's local day. With the rule g8 writes, the commands were run on a development stack on 2026-10-02: the index of the UTC day showed the requests sent, 0 and green. They have not been run on this machine since.

oc -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  "http://127.0.0.1:9200/_cat/indices/g8-*?v&h=index,docs.count,rep,health"'

oc -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -s "http://prometheus:9090/api/v1/targets?state=active"'
index                    docs.count rep health
g8-okd-access-2026.08.29         30   0 green

Every APISIX pod must appear as up in the Prometheus targets.

Traffic Explorer in the g8 console must list the same requests.

Analytics reads two things from these metrics: each route under its id with its name beside it, and each bucket counted exactly with the anchored range selector:

oc -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -sG http://prometheus:9090/api/v1/query --data-urlencode "query=count by (route, route_name) (apisix_http_status)"'
oc -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -sG http://prometheus:9090/api/v1/query --data-urlencode "query=sum(increase(apisix_http_status[5m] anchored))"'

The first query lists every route that answered with its id in route and its name in route_name; a route that carries a name means a copy of the prometheus plugin has prefer_name on, and Analytics reads that route empty. The second must answer "status":"success"; a Prometheus started without --enable-feature=promql-extended-range-selectors answers anchored modifier is experimental and not enabled, and Analytics marks its counts as estimates. Both queries were run on this machine on 2026-10-04 against Prometheus 3.15.0: every route with a name showed it in route_name, and the anchored query answered success.

Request records are LOST while Elasticsearch is unreachable

The gateway sends a batch of records once more after the first failure (max_retry_count 1 in the rule g8 writes) and then drops it. The loss was measured with the rule of the earlier guide, which tried three times: on a production installation during a password change, 20 requests that arrived while Elasticsearch was restarting appeared in the metrics but never reached the index.

Read the loss from APISIX's error log, not from a counter difference:

grep -c 'elasticsearch-logger' <error.log>

Subtracting the document count from the metric counter is not reliable and produces false alarms. A measured example: counter 88, documents 43, difference 45 — with not one delivery error in the log. The difference has three causes and none of them is loss: the counter counts from the moment APISIX started (requests from before the plugin was enabled are in the counter but not the index), a restart resets the counter while the index persists, and the two sides have different retention. The difference is a hint only within a single window — no APISIX restart, nothing deleted by retention, both plugins enabled together.

An installation that cannot afford losing records must put a buffer in between (kafka-logger, or a local file-logger plus a separate shipper).

Deleting pods was measured to lose nothing: after elasticsearch-0 and the Prometheus pod were deleted and came back, the document count was unchanged. Persistence comes from the PVCs; delete a PVC and the data is gone.

12. Diagnostics​

You will see error lines on a healthy installation too

logs output is cumulative: it holds everything since the container or pod was born. APISIX prints connection refused while waiting for etcd at startup, and the same lines land again for members stopped during the quorum drill. Those lines describe the past, not the present state.

Measured on a healthy installation in this environment:

request etcd endpoint 'http://etcd-1:2379/version' error, connection refused
error.log lines: 0

An empty error.log is normal too — APISIX's own error log is a separate file, and the startup messages go to stdout rather than there.

The present state is what the verification section's counts tell you; logs is for diagnosis, not proof of health. Decide whether a line is current from its timestamp.

oc -n g8-gateway get pod -o wide
oc -n g8-gateway logs -l app=apisix --tail=50 --prefix
oc -n g8-gateway logs etcd-0 --tail=50
oc -n g8-gateway describe pod etcd-0
oc -n g8-gateway get events --sort-by=.lastTimestamp | tail -20

If a pod has restarted, read the previous container's log; the current log looks healthy while the reason for the crash is in there:

oc -n g8-gateway logs <pod> --previous
oc -n g8-gateway get pod <pod> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

Cluster tables — member IDs and timings differ on every installation:

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E member list -w table
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint status -w table

Prometheus is tested separately on each pod:

for p in $(oc -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[*].metadata.name}'); do
  oc -n g8-gateway port-forward pod/$p 19091:9091 >/dev/null 2>&1 &
  PF=$!
  sleep 4
  echo "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:19091/apisix/prometheus/metrics) $(curl -s http://127.0.0.1:19091/apisix/prometheus/metrics | grep -c '^apisix_')"
  kill $PF 2>/dev/null
  sleep 1
done
200 87
200 87
200 87

13. Uninstalling​

Delete the environment in the g8 console first, or clear its log store there: g8 connects to the store once a day to delete the days past the retention, and records a failed cleanup every day for as long as the environment points at a store that is gone.

oc delete namespace g8-gateway
namespace "g8-gateway" deleted

Deleting the namespace deletes the PVCs too; because topolvm-provisioner uses the Delete reclaim policy, the etcd data and the backups are gone permanently. If a backup is to be kept, move it out first.

Verify that the deletion completed:

oc get ns g8-gateway --ignore-not-found | wc -l
oc get pv --no-headers | grep -c g8-gateway || true
0
0

14. Upgrading​

Unverified

A version upgrade was not attempted in this environment. The sequence below is a design, not evidence.

  1. Take the backup from 10.9 and verify it.
  2. APISIX: update the Deployment's image tag. The default RollingUpdate replaces pods one at a time; run 10.4 at every step.
  3. etcd: the StatefulSet's default RollingUpdate starts from the highest ordinal and waits for each pod to become Ready. Do not skip versions; verify endpoint health returns three at every step.

Rollback path: the old image tags are in Harbor; oc rollout undo returns the Deployment to its previous ReplicaSet.

On 2026-09-30 the APISIX Deployment moved from 3.18.0 to 3.19.0 by step 2 (oc set image deploy/apisix, rolling) and the data plane answered Server: APISIX/3.19.0; etcd was not touched. Whether step 1's backup and the 10.4 verification at every step ran for that change is not recorded, so the sequence stays unverified.

14.1 Request records of an installation made with the earlier guide​

Unverified

This move was rehearsed on a development stack on 2026-10-02: an installation made as the earlier guide wrote it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

The earlier version of this guide wrote the logger into global rule 1 by hand: with the elastic account, an index named g8-okd-access- and the machine's local day, and a lifecycle policy that deleted the indices. Such an installation is moved in this order:

Do not turn traffic logging on before step 6

Settings offers Turn on traffic logging as soon as the pipeline and the template are applied. Turning on over the old rule completes it in place: the account is neither asked for nor tried, and elastic stays in the rule.

  1. Open the two accounts (11.6).
  2. Connect the store in the g8 console with g8-okd-server (step 1 of 11.7). Settings then reads Index template Outdated, Ingest pipeline Missing and the logger rule Customized. From its next daily cleanup on, g8 deletes the days past the retention.
  3. Apply the ingest pipeline and then the index template (11.6). They apply to the days born after them.
  4. Take the lifecycle policy off the indices and delete it:
oc -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  -X PUT "http://127.0.0.1:9200/g8-*-access-*/_settings" \
  -H "Content-Type: application/json" -d "{\"index.lifecycle.name\":null}"'
oc -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  -X DELETE http://127.0.0.1:9200/_ilm/policy/g8-access-retention'
{"acknowledged":true}{"acknowledged":true}
  1. The policy kept the indices 14 days. g8 keeps them 30 days unless that is changed: set the number of days this store is sized for under Traffic Explorer › Settings › Retention › Change.
  2. Run the command of 11.5 again. It prints 200 here; rule 1 then carries the metrics alone and Settings reads the logger rule Off. Only then turn traffic logging on, with g8-okd-gateway (steps 2 and 3 of 11.7).
  3. Change the password of elastic: it was part of the gateway's configuration for as long as the old rule existed.

No record is written between the two halves of step 6. Every day written before the move, and the day of the move itself, was born without g8's template and without the fields its records carry: Traffic Explorer cannot list those days, and Settings counts them as written under another version. They stay in the store until the retention deletes them. The records are listed from the first day whose index is born after the move.

g8 lists and deletes the days of the environment's own slug alone. If this gateway is registered in g8 under another slug than okd, the days the old rule wrote have no deleter once the policy is gone: delete them by their names when they are no longer needed.

14.2 The ingest pipeline on an installation made before it​

Unverified

This move was rehearsed on 2026-10-04 against a development stack's Elasticsearch 8.15.3: an installation with the earlier template and a day born under it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

An installation made before this version of the guide holds the index template without the ingest pipeline, and its server account may not read pipelines. Settings then reads Index template Outdated and Ingest pipeline Cannot be read. The gateway keeps writing records meanwhile, copies included, and the settings can still be saved; g8 refuses only to turn logging on anew until both read Applied. The move, in this order:

  1. Run the block that writes the two roles and the two users again (11.6). It keeps the passwords and gives the server's role the line that reads the pipeline; every line answers 200.
  2. Apply the ingest pipeline (11.6).
  3. Apply the index template (11.6). The days born after it send their records through the pipeline.
  4. Send the day of today, born under the earlier template, through the pipeline at once. On a store that serves several environments, run it once for each environment's slug:
DAY=$(date -u +%Y.%m.%d)
oc -n g8-gateway exec elasticsearch-0 -- sh -c \
  "curl -s -u elastic:\$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/g8-okd-access-$DAY/_settings \
   -H 'Content-Type: application/json' -d '{\"index.default_pipeline\":\"g8-access\"}'"
{"acknowledged":true}

A store that holds no record of today yet answers 404 index_not_found_exception here: today's day will be born under the new template, and nothing more is needed. Settings then reads both Applied. A day written before the move may hold copies of a batch the gateway delivered twice; it stays as it is until the retention deletes it.

15. Traps hit during installation​

15.1 With OrderedReady, the etcd bootstrap deadlocks​

Symptom: three replicas are requested but only one pod is created, and it stays at 0/1.

etcd-0 0/1 Running

The default podManagementPolicy is OrderedReady: the StatefulSet does not create etcd-1 until etcd-0 is Ready. etcd-0's readiness probe is /readyz, and in a three-member cluster it can never be ready alone because it cannot form a quorum. The log loops on raft elections:

msg":"4130acc46e67d483 became pre-candidate at term 1

The fix is podManagementPolicy: Parallel. That field is immutable; it cannot be applied to an existing StatefulSet:

The StatefulSet "etcd" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

The StatefulSet is deleted and recreated. The PVCs left over from the failed bootstrap must be deleted too; a half-written member record breaks the new cluster.

15.2 Testing the SCC with oc run gives a false measurement​

Symptom: no error — a wrong "success". The pod runs as the image's own uid and is believed to work in a restricted environment.

scc: anyuid
uid=636(apisix) gid=636(apisix)

The SCC is chosen based on who created the pod. If you create a pod directly with oc run, that is you; a cluster admin has anyuid through system:cluster-admins. A pod created by a controller (Deployment, StatefulSet) is evaluated against the namespace's ServiceAccount and gets restricted-v2.

The correct test is to create it the way the real workload is created: apply a Deployment and read the pod's openshift.io/scc annotation.

Impersonating with --as=system:serviceaccount:... does not work either: the ServiceAccount has no RBAC to create pods, so the request is denied at RBAC before it ever reaches SCC.

15.3 allow_admin produces intermittent 403s behind a Service​

Symptom: the same client, with the same command, gets 200 sometimes and 403 other times.

{"remote_addr":"10.128.0.215","status":"401"}
{"remote_addr":"169.254.0.5","status":"403"}

When a request through the Service is routed back to its own pod (hairpin), OVN translates the source address to the masquerade address. When the same request lands on a different pod, the source is preserved. The result: if allow_admin contains only the pod network, requests are rejected at random.

The masquerade network is read from the cluster:

oc -n openshift-ovn-kubernetes exec ds/ovnkube-node -c ovnkube-controller -- ip -4 route | grep 169.254
169.254.0.0/17 dev br-ex proto kernel scope link src 169.254.0.2

That network is added to allow_admin. But the real lesson is this: in Kubernetes the source IP is not access control; the control is the NetworkPolicy in section 8.

15.4 The etcd image contains no shell​

Symptom:

executable file `sh` not found in $PATH: No such file or directory

The etcd image carries only the etcd binaries. oc exec … -- sh -c does not work; commands are given directly (oc exec etcd-0 -- etcdctl …).

The same constraint shows up in the manifest: shell substitutions such as $(date) cannot be used in command/args. Kubernetes' own $(VAR) substitution is used instead — that one is done by kubelet, not a shell. Steps that do need a shell (pruning files) are run as an initContainer from an image that has one.

15.5 APISIX does not wait for etcd; it exits​

Symptom: during an etcd outage, new APISIX pods enter CrashLoopBackOff. The operator thinks APISIX is broken.

exitCode=1 reason=Error
request etcd endpoint 'http://etcd-0.etcd:2379/version' error, connection refused

If APISIX cannot reach etcd during its init_etcd startup step, it exits with 1; it does not retry. In Kubernetes this heals itself: kubelet restarts the pod and it comes up once etcd returns. Measured in this environment: Running after two restarts.

Pods that are already running are unaffected — they keep serving from cache throughout the etcd outage (10.8).

15.6 A restart wait fires too early​

Symptom: the wait loop reports "ready" on its first iteration, when nothing has changed yet.

Immediately after rollout restart, the old pods are still 1/1 Running. A loop counting grep -c 1/1 counts those and its condition is satisfied; the real state, while the new pods are coming up, is something else entirely.

Instead of a loop counting pods, wait on the restart itself:

oc -n g8-gateway rollout status deploy/apisix --timeout=300s
oc -n g8-gateway rollout status sts/etcd --timeout=300s

15.7 On quorum loss the first write hangs, later ones fail instantly​

Symptom: the first Admin API write after etcd loses quorum hangs for tens of seconds. Running the same command immediately afterwards returns in milliseconds. The operator thinks "the server locked up" on the first attempt and "something recovered" on the second; both readings are wrong.

healthy: PUT=201 time=0.095456s
req1:    PUT=503 time=42.382359s
req2:    PUT=503 time=0.024547s
req3:    PUT=503 time=0.029827s

The first request tries the three endpoints in turn and times out on each. After that APISIX marks the endpoints unhealthy and answers from its own record without going to etcd at all:

{"error_msg":"has no healthy etcd endpoint available"}

Two consequences: (1) an automation client's timeout must be chosen for the first request, and (2) a fast 503 does not mean etcd is back — the state must be asked for with endpoint health.

The measurement differs in the prod (Docker) installation: there the requests do not get faster and each takes tens of seconds (03-prod-docker, 17.2). The two measurements are not interchangeable.

15.8 A liveness probe on cluster health kills the last member​

Symptom: during quorum loss the one surviving etcd member is restarted over and over. oc describe shows:

Killing … (x6 over 10m) Container etcd failed liveness probe, will be restarted

etcdctl endpoint health is a cluster health check: it performs a read that requires quorum. Put the same command on both readiness and liveness and, on quorum loss, readiness failing is correct (the pod should leave the Service) while liveness failing is a disaster — kubelet kills the last pod holding the data roughly every 90 seconds, and the loop does not stop until quorum returns.

Since 3.5.12 etcd exposes two endpoints precisely to separate the two. Measured:

no quorum:
  /livez  -> HTTP/1.1 200 OK
  /readyz -> HTTP/1.1 503 Service Unavailable

That is why the manifest in section 5 points liveness at /livez and readiness at /readyz. Before and after the fix, across three minutes of quorum loss:

etcdctl endpoint health  ->  restartCount 1 → 5 → 6
/livez + /readyz         ->  restartCount 0 → 0 → 0

The rule: liveness must ask "is the process running", not "is the cluster healthy". The second question belongs to readiness.

15.9 The NOSPACE alarm is a one-way door​

Symptom: writes are refused with etcdserver: mvcc: database space exceeded. Nobody notices, because the data plane keeps returning 200 — and once it is noticed, freeing space is not enough.

The recovery chain was measured, and all three steps are required:

159 writes         -> dbSize=16789504  alarm=NOSPACE
delete + compact   -> dbSize=16789504  write refused
defrag             -> dbSize=122880    write refused
alarm disarm       -> write OK

There are three separate misconceptions here, each verified individually:

  • Deleting keys does not free space. etcd keeps the revision history; a delete writes a new revision.
  • Compaction does not return disk space. dbSizeInUse drops, dbSize does not move at all — the freed pages stay free inside etcd and never return to the filesystem. Since the quota watches dbSize, the quota is still full. The extreme measured in this environment: dbSizeInUse at 41 kB while dbSize sat at 98 MB.
  • Defrag returns the space but does not lower the alarm. The alarm is sticky: it survives a process restart, a container recreation and a host reboot.

Recovery:

for n in 0 1 2; do
  oc -n g8-gateway exec etcd-$n -- etcdctl --command-timeout=900s defrag
done
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379 alarm disarm

Leave alarm disarm for last: disarming without actually freeing space puts you straight back where you were on the next write.

The lasting fix is scheduled defragmentation — see long-term operation. With no defrag scheduled, dbSize keeps growing even when compaction is working correctly, and the quota fills sooner or later.

15.10 A member that loses its disk cannot rejoin on its own​

Symptom: a member loses its data directory (disk failure, accidental deletion, a lost PVC). When it comes back it loops forever, logging cluster ID mismatch or member has already been bootstrapped.

What makes it insidious: member list still shows that member as started and the data plane keeps returning 200. The cluster silently drops to 2/3, and losing one more member takes quorum with it.

The cause is ETCD_INITIAL_CLUSTER_STATE: new hardcoded in the manifest. Seeing an empty data directory, etcd tries to bootstrap a new cluster rather than join the existing one.

The recovery chain was measured:

STATE=new + bos veri dizini    -> cluster ID mismatch (uye asla katilmaz)
member remove + olcek dusur    -> veri dizini gercekten bos
member add (saglikli kume)     -> ETCD_INITIAL_CLUSTER=...
STATE=existing, tek acilis     -> healthy=3/3
STATE=new'e donus              -> healthy=3/3, veri yerinde

The critical detail: STATE is only read while the data directory is empty. So existing is needed for exactly one start; once the member has joined it runs indefinitely under its normal definition.

Two steps are the easiest to miss and both are mandatory:

  • Scale down. Deleting the pod is not enough; the StatefulSet recreates it immediately and etcd rebuilds the data directory. The member has to be gone before you wipe the directory.
  • Verify the remaining cluster is healthy. member add refuses with etcdserver: unhealthy cluster while the cluster is unhealthy.

When wiping the data directory, delete hidden entries too; the pattern rm -rf dir/* skips entries beginning with ..

Recovery:

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
E2=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379

ID=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E member list -w simple | grep 'etcd-2' | cut -d, -f1)
oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 member remove "$ID"
oc -n g8-gateway scale sts etcd --replicas=2
oc -n g8-gateway delete pvc data-etcd-2

oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 endpoint health
ADD=$(oc -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 \
  member add etcd-2 --peer-urls=http://etcd-2.etcd:2380)
IC=$(echo "$ADD" | grep -oE 'ETCD_INITIAL_CLUSTER="[^"]*"' | sed 's/ETCD_INITIAL_CLUSTER="//; s/"$//')

oc -n g8-gateway set env sts/etcd ETCD_INITIAL_CLUSTER_STATE=existing "ETCD_INITIAL_CLUSTER=$IC"
oc -n g8-gateway scale sts etcd --replicas=3

Once the member has joined, return to the normal definition:

oc -n g8-gateway set env sts/etcd ETCD_INITIAL_CLUSTER_STATE=new \
  "ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"

15.11 ConfigMap and Secret changes never reach the pods​

Symptom: you changed config.yaml, the ConfigMap was updated, and nothing changed. There is no error; the pods keep reading the old content.

The ConfigMap is mounted with subPath — a single file rather than the whole directory. Kubernetes does not update files mounted via subPath; this is documented behaviour, not a defect. The same applies to the Secret holding the admin key.

Measured:

ConfigMap d04c459e -> pod d04c459e
ConfigMap 6c4bb814 -> 90 s sonra pod hala d04c459e
rollout restart    -> pod 6c4bb814

The way to put a change into effect is to recreate the pods:

oc -n g8-gateway rollout restart deploy/apisix
oc -n g8-gateway rollout status deploy/apisix --timeout=300s

This is also how the admin key is rotated: update the Secret, then rollout restart. The restart is not disorderly — the Deployment replaces pods one at a time, and the PodDisruptionBudget in 7.1 guarantees at least two pods stay up.

16. Long-term operation​

topicsettingwhy
etcd growthETCD_AUTO_COMPACTION_MODE=periodic, RETENTION=1Without compaction etcd keeps the old revision of every write
etcd quotaETCD_QUOTA_BACKEND_BYTES=2147483648The PVC is 2Gi; the quota must be smaller than the disk so etcd alarms before the disk fills
Logsaccess_log: /dev/stdout, error_log: /dev/stderrWriting to the container's filesystem is not durable and is not rotated
Log rotationkubelet: containerLogMaxSize=50Mi, containerLogMaxFiles=5At most 250 MB per container. No logrotate is installed on top
File descriptorsno setting neededThe CRI-O default measured 1048576 on this cluster; the 1024 problem from Docker does not exist here
PlacementpodAntiAffinity: preferredOn a single-node cluster required leaves two of the three replicas unschedulable. Make it required on a multi-node cluster
Probe targetcontrol port 9090A bare TCP connection to 9080 puts a 400 line into the access log
BackupDaily CronJob, 14 filesSection 9
etcd accessNetworkPolicyetcd has no authentication; this is the only real control

That backups are actually being produced must be checked regularly:

oc -n g8-gateway get cronjob etcd-backup
oc -n g8-gateway get job --sort-by=.metadata.creationTimestamp | tail -3

17. Scope limits and next steps​

What this installation does not do, stated plainly:

  • It was verified on a single-node cluster. Three replicas verify the shape of the manifests, not fault tolerance. On a multi-node cluster podAntiAffinity must be required, and that change was not tried here.
  • There is no TLS. Traffic between APISIX and etcd, and the Admin API, are plain HTTP. The Route does not terminate TLS either. TLS is separate work.
  • The real client IP does not reach APISIX — 10.5.
  • etcd has no authentication. Access is restricted only by NetworkPolicy.
  • Restore was not attempted. A backup is taken and its validity verified, but bringing a three-member cluster up from a snapshot was not drilled.
  • The upgrade was not attempted — section 14.
  • Behaviour under load was not measured. Every verification is a single request.
  • There is no HorizontalPodAutoscaler or resource quota. A PodDisruptionBudget is created in 7.1.
G8 dev · Apache APISIX® 3.19.0
Apache, Apache APISIX and APISIX are trademarks of The Apache Software Foundation.