🔥Let’s Do DevOps: Shard GitHub Actions Workload over Many Concurrent Builders

Kyler Middleton
9 min readApr 24, 2023

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I’ve recently been building a tool I’m calling the GitHub Cop. It’s a tool that processes all 1.5k+ repos in my GitHub Org and sets all the settings, branch protections, auto-link rules, permissions, etc. That’s a lot of changes each night, and on every repo sequentially.

To do that, I’ve built several tools, including an API token circuit breaker to avoid running out of API tokens when doing tens of thousands of actions in a short time, and how to paginate API calls when processing thousands of attributes, but none of those processes help speed up the single-threaded processing of thousands of repos.

What would help is having several threads working concurrently. Thankfully, GitHub Actions makes this pretty easy to do! I did have to teach bash some arithmetic, but I was able to make it work.

Let’s talk about theory, then how I implemented this change in my custom bash program, and the limitations I faced and how I solved them.

GitHub Actions Theory: Matrix

--

--

Kyler Middleton

DevNetSecOps, DevRel, cloud security chick. I will teach you, it’s unavoidable. She/Her 🏳️‍🌈🏳️‍🌈, INFJ-A, support the EFF!