Skip to main content

Back up and restore

A G8 installation is two things: one PostgreSQL database and the key that seals the secrets stored in it. Save both, and a new server started on them is the installation again, with every user, role, environment and alert rule in place. Save the database alone and the server refuses to start on it. This page says what the database holds, what is not in it, and the steps to save and bring back an installation.

Who may do it​

Whoever runs the server and holds the database and the place its settings are kept. Nothing on this page happens in the console; the console is only where the result is checked.

What the database holds​

The server creates and keeps its own schema in the database named by G8_PG_DSN. The tables carry:

  • The users, their sessions and preferences, the roles with their permissions, the assignments of roles to people, and the service accounts with their tokens.
  • The environments, each with the address of its cluster's Admin API, its sealed admin key and the sealed password of its log store.
  • The Active Directory connection, with its sealed bind password, and the group mappings.
  • The audit events and the snapshots a change recorded.
  • The alert rules, the mail relay with its sealed password, the webhook channels with their sealed signing keys, and the alert history.
  • The installation's own record, which holds the setup token until the first administrator is created.

Every value named sealed above is encrypted with the key in G8_SECRETS_KEY before it is written. The server does not store that key; it reads it from its settings at every start, as Server configuration describes.

Why the key is part of the backup​

At start, and on every readiness check, the server opens every sealed row. A row sealed under a key the process does not hold stops the start, with the table and the row named in the error. On a running server the readiness address answers that a sealed secret needs a key this instance does not hold. A restored database without its key is a database the server will not serve. Keep the key with the dump, apart from the database, in the place your team keeps secrets.

If the key was changed with Rotate the secrets key, a dump taken before the rotation needs the key that sealed its rows at that time. Keep each key as long as a dump sealed under it is kept.

What is not in the database​

  • The gateway's own configuration. Routes, services, upstreams, consumers, certificates and plugins live in the cluster's configuration store, not in G8. G8 reads and writes them there through the Admin API. Back them up as the installation guide of the cluster says: OKD, k3s, Prod on Docker, Test on a virtual machine or Dev on Docker.
  • The request records. They live in the environment's log store, under names derived from the environment's slug; see Where request records go.
  • The counts. Prometheus holds the gateway's traffic numbers; see Where the numbers come from.

A G8 backup brings back the console's own data: who may sign in, which environments exist and how to reach them, and what was changed by whom. It brings back no route and no request record.

Where the key is kept​

On OKD and OpenShift the Helm chart reads G8_SECRETS_KEY from the Secret named by secrets.existingSecret, g8-secrets by default, and G8_PG_DSN from the Secret named by db.existingSecret. The chart's install notes list both. The chart's own PostgreSQL, switched on with db.internal.enabled, is the lab database, with one replica and no backup. The notes say to point G8_PG_DSN at a real PostgreSQL for anything else, and that database is where your dump is taken.

Back up​

The dump tool is your choice; the commands below use PostgreSQL's own, and your database team may have another.

  1. Read the database connection the server uses from the g8-db Secret.

    G8_PG_DSN=$(oc -n g8 get secret g8-db -o jsonpath='{.data.G8_PG_DSN}' | base64 -d)

  2. Dump the database to a file.

    pg_dump --format=custom --file=g8-$(date +%Y%m%d).dump "$G8_PG_DSN"

  3. Copy the current value of G8_SECRETS_KEY from the g8-secrets Secret to the place your team keeps secrets, next to a note of which dump it belongs to. During a rotation, copy G8_SECRETS_KEY_PREVIOUS as well.

    oc -n g8 get secret g8-secrets -o jsonpath='{.data.G8_SECRETS_KEY}' | base64 -d

  4. Store the dump where the database host's loss does not take it along.

Run in the lab on a fresh installation with one administrator, one environment and three audit rows. The dump was taken on the virtual machine that runs the database, with PostgreSQL's own tool, and the key was read from the Secret.

Restore​

The target is an empty database; the dump brings the schema and the rows with it.

  1. Stop every G8 server that points at the database you are replacing, so none of them writes while you restore.

    oc -n g8 scale deploy/g8 --replicas=0

  2. Create an empty database and a user that owns it, with the tools of your PostgreSQL.

  3. Load the dump into it.

    pg_restore --no-owner --dbname="$NEW_G8_PG_DSN" g8-<date>.dump

  4. Set G8_PG_DSN to the new database and G8_SECRETS_KEY to the key that was saved with the dump, by updating the g8-db and g8-secrets Secrets.

  5. Start the server. Its log prints schema ready with the schema version it found and the one it moved to, then listening.

    oc -n g8 scale deploy/g8 --replicas=1
    oc -n g8 rollout status deploy/g8

  6. Check readiness: /readyz answers with the status ready once the database answers and every sealed row opens.

    curl -sS https://<console host>/readyz

  7. Sign in to the console. Under Environments, every environment is listed and shows that its cluster answers; see Environments.

Run in the lab: the Deployment was scaled to zero so nothing held the database, the database was dropped and created again empty, the dump was loaded with the command above, and the Deployment was scaled back to one. The new pod's log read as follows, /readyz answered 200 and the administrator signed in with the password from before the loss.

"msg":"schema ready","version_from":16,"version_to":16,"applied":null
"msg":"listening","addr":"[::]:8080","tls":false

If the server stops at start with the message sealed with a key this server does not hold, the key does not match the dump. Restore the key, never the row: the message names the table and row, and deleting them loses the admin key or password they hold.

G8 dev