Skip to main content

Upgrade G8

An upgrade of G8 is a new image and nothing else. The image holds the server and the console together, and the server brings its database schema to the version it needs at start. There is no console to deploy on its own and no migration tool to run by hand. This page says what the server does when the new image starts, the order to keep when the gateway moves to a new version too, the steps on each platform, and what a rollback means.

Who may do it​

Whoever runs the server, which on OKD and OpenShift is whoever may change the Helm release. Nothing in the console starts an upgrade, and no role in the console is needed.

What the new image does at start​

The console is built into the image: the build copies the console's files into the server, and the server serves them from its own binary. A new image is therefore a new console and a new server at once, and a browser that reloads the page gets the new console.

Before it serves anything, the server takes a lock in the database so that one process at a time applies the schema, applies every migration the database does not hold yet, and releases the lock. The migrations are embedded in the binary. The log line schema ready carries version_from, the schema version it found, version_to, the one it reached, and applied, the list of migration names it ran. A start that found the schema current logs the same line with the two versions equal and nothing under applied.

Behind the same lock the server also moves sealed rows to the current key, seeds the permission dictionary and the built-in roles, and checks that the roles in the tables match the ones in code. A mismatch stops the start. Only then does it print listening and answer requests. What to watch lists these lines.

The moment the console is unavailable​

The chart's Deployment replaces its pods with the strategy Recreate: the old pods stop before the new ones start, whatever the number of replicas. Between the two the console answers nothing and the API answers nothing. The strategy is chosen for the secrets key, so that two key generations never serve at once; Rotate the secrets key explains why. A session that was open before the upgrade continues after it, because sessions live in the database.

When the gateway moves to a new version too​

A G8 release is verified against one gateway version: the schema snapshot, the version folders and every form the console draws are taken from that version. A cluster running another version keeps working, is marked Unverified wherever the console shows the gateway, and is drawn with the forms of the verified version. A new gateway version is therefore matched by a new G8 release verified against it; until that release runs, the console marks the environment and keeps working.

The order is the gateways first, then G8. Each environment's cluster moves as its installation guide says, and the console marks the environment Unverified while it runs the newer gateway. Then G8 moves to the release verified against that version, and the mark goes. The other way round, G8 first, leaves the console drawing forms for a version the clusters do not run yet; it works, but every environment is marked until the clusters follow.

Before you start​

  • A dump of the database and the current secrets key, as Back up and restore describes. The dump is the way back; see Rolling back.
  • The version that runs now: open /version and keep its answer.
  • The release's artifacts: the packaged chart and the image digest.
  • A moment when the console may be unavailable.

On OKD and OpenShift​

The chart is released as a package named after the release, with the release's version stamped by the pipeline. The pods run the image by digest: the chart requires image.digest and refuses to render without it. The package, its checksum and the image digest come from G8 with each release, the same way the first install got them.

  1. Take the packaged chart of the new release and compare its checksum with the one that came with it.

  2. Upgrade the release with the package and the new digest. Keep every other value as it is; the chart's values.schema.json refuses a misspelled value instead of rendering a default.

    helm upgrade <release> ./g8-<version>.tgz -n <namespace> \
      --reuse-values --set image.digest=<sha256 digest of the new image>

  3. Wait until the new pods are ready. Readiness is /readyz, which the Deployment probes itself.

    oc rollout status deployment/g8 -n <namespace>

  4. Read the log of a new pod for schema ready, with the migrations it applied, and then listening.

  5. Open /version. Its version is the new release and its git_commit the commit the release names.

    curl -sS https://<console host>/version

  6. Open /readyz. It answers with the status ready.

  7. Sign in to the console. The version at the bottom of the menu is the new one. Under Environments, every environment is listed and its cluster answers; see Environments.

Run in the lab, from the release before the current one to the current one, with the command of step 2. The new pod applied the two migrations the release added, the session opened before the upgrade still answered after it, and /version named the new release. For a few seconds after the rollout the Route still showed the router's own page saying the application is not available; the next request reached the new pod.

"msg":"schema ready","version_from":14,"version_to":16,"applied":["015_alerts.sql","016_alerts_retire_certificate_and_version_kinds.sql"]
"msg":"listening","addr":"[::]:8080","tls":false

Rolling back​

The server moves the schema forward only. Each migration file carries a drop half below a marker, but the server has no step that runs it; at start it applies what is missing and stops at the newest version it embeds. An older image started on a database a newer one migrated stops at start with a migration error, because the schema version it finds is above the highest one it knows. It runs no drop half and undoes nothing.

Run in the lab: the previous release's image, pointed at the database the current one had migrated, stopped at every start with the error below and never became ready.

"level":"ERROR","msg":"server exited","error":"migrate apply: current version 16 is outside the valid versions of 0 to 14"

A rollback is therefore your own procedure, outside the server: restore the dump taken before the upgrade, as Back up and restore describes, with the key that was current then, and start the previous image on it. Every change made in the console between the dump and the rollback is lost with the database; the gateway's own configuration is not touched by either, because it lives in the cluster's configuration store.

G8 dev