Skip to main content

Stop calling a failing backend

When a backend falls over, every client that keeps calling it waits for an error, and the calls keep it down. The plugin api-breaker on a route counts the backend's failures. After enough of them, the gateway stops forwarding for a while and answers the client itself. This guide does that for orders-api after five failures. At the end, once the backend has failed five times, the clients of orders-api get 503 and your own message from the gateway, and the backend is not asked.

The breaker has two states. Closed, requests reach the backend and the gateway counts its failures. Open, the gateway answers every request with 503 and a message of your own, and the backend is not asked at all.

The time the breaker stays open grows with each round of failures. Beside Longest time the breaker stays open, the form says: The breaker first opens for two seconds and doubles that with every further round of failures, up to this many seconds. Successes clear the failure count again. Beside Successes, the form says: How many successes in a row clear the failure count, so the next opening starts again at the shortest time. The check below shows the breaker opening only.

Before you start​

  • This guide starts from orders-api as Route requests to a backend leaves it, with key-auth from Require an API key. The requests below call /orders/42. A plugin from another guide that this page does not name can change the answers below; remove it first or work on a copy of the route.
  • A way to see the breaker open without harming clients. The check below needs a backend that cannot be reached. To try it, give a test route one backend node at an address and port where nothing listens, and put the same api-breaker on it.
  • The host name of your gateway. In the console it is the first part of the Address line on a route's Overview tab, api.example.com in this page's examples; replace it in every request below.
  • The Editor or Environment administrator role in this environment.

Steps​

1. Choose api-breaker​

Open orders-api under Routes, select its Plugins tab, then Add plugin at the top right of the tab. Open the Traffic category, or type api-breaker in the search box. Select api-breaker, then Configure.

2. Say what a failure is and what the client gets​

Fill in these fields and leave the rest as they are.

FieldValueWhy
Break response code503The status a client gets while the breaker is open.
Unhealthy › Failures5How many failures open the breaker.
Unhealthy › HTTP statuses500, 502, 503 and 504An answer with one of these codes counts as a failure.
Healthy › Successes2Successes in a row that clear the failure count.
Longest time the breaker stays open120The longest opening, in seconds. Each further round of failures doubles the time up to this ceiling.
Break response bodyThe orders service is temporarily unavailable.The whole body of the gateway's answer, sent exactly as written.

Select Add plugin at the bottom of the panel. The plugin is saved like this:

The Add api-breaker panel with the status codes that count as failures, the number that opens the breaker and the answer while it is open.
api-breaker · as the console saves it
{
  "break_response_code": 503,
  "break_response_body": "The orders service is temporarily unavailable.",
  "unhealthy": {
    "failures": 5,
    "http_statuses": [
      500,
      502,
      503,
      504
    ]
  },
  "healthy": {
    "successes": 2
  },
  "max_breaker_sec": 120
}

Check it​

While the backend cannot be reached, send the same request six times. The first five count as failures and are answered alike, and the sixth meets the open breaker. On orders-api, which carries key-auth, add the key of mobile-app in the x-api-key header to the request below.

curl
curl -i -L 'https://<gateway host>/orders/42'
First request
While the backend refuses connections, the first request is answered 502 by the gateway, which still tried the backend.
Second request
The second request is answered 502 the same way.
Sixth request
The sixth request is answered 503 with The orders service is temporarily unavailable. by the gateway itself, without reaching for the backend.

What to know​

  • A connection the backend refuses is counted as a 502, as the first answer above shows. A status the gateway produces itself never opens the breaker.
  • An answer with a code under Healthy › HTTP statuses counts as a success once any failure has been counted. An answer in neither list leaves the count where it stands.
  • The count is kept per host and request path. A route that takes /orders/* counts each path under it on its own, so one failing path does not open the breaker for the others.
  • Only codes from 500 up count as failures. A code below 500 under Unhealthy is refused when you save.
  • A Longest time the breaker stays open below three seconds is refused when you save.
  • Headers under Break response headers are sent only when a break response body is set. Without a body, only the code is sent.
  • An open breaker keeps a path's requests from every backend node at once. To take one failing backend node out of rotation while the others serve, give the backend a health check, as Share a backend between routes shows.
G8 dev