🔥Let’s Do DevOps: Building an API Token Expired Circuit Breaker

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I recently had to create 60k auto-link references in Jira (link to the story), and I immediately ran into an issue — a GitHub PAT (Personal Access Token) is given 5,000 “tokens” per hour. A “token” is a budget of the API calls that can be issued to the server which will be honored. More than that will fail.

API budgets are a concept established to help avoid DoS (Denial of Service) attacks where tens or hundreds of thousands of calls are sent to a service in order to destabilize it.

Well, sending 60k requests to GitHub means I’m spending 60k tokens, 12 times the value I get per hour. My local script runs quite a bit faster than that, uh oh. And many of the other requests I’m sending, like opening PRs and issuing comments on them, also consume tokens — so how can I tell when my API token budget is consumed? How can I tell when token budget has been refilled and I can continue?

That’s where a “circuit breaker” comes in. That concept is borrowed from electrical engineering where it means to detect when more current than is safe passes through the circuit breaker, and when that happens it immediately disconnects the circuit so nothing downstream of it can be fried. In this context, it means we’ll monitor our API token budget, and establish a wait timer until our budget is refilled.

Let’s do it!

Establish the Circuit Breaker

A circuit breaker in this context is a check that won’t continue until a condition is satisfied. We might want to call this circuit breaker lots of times, so let’s put it in a function called hold_until_rate_limit_success.

And then we build one of the “you shouldn’t do this” loops, a while true, which means our loop will continue forever until a command issues break. These are generally not advised because a misconfiguration could lead to a loop that goes on forever. We’ll keep our function concise and simple in order to be as safe as possible.