🔥Let’s Do DevOps: Shard GitHub Actions Workload over Many Concurrent Builders

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I’ve recently been building a tool I’m calling the GitHub Cop. It’s a tool that processes all 1.5k+ repos in my GitHub Org and sets all the settings, branch protections, auto-link rules, permissions, etc. That’s a lot of changes each night, and on every repo sequentially.

To do that, I’ve built several tools, including an API token circuit breaker to avoid running out of API tokens when doing tens of thousands of actions in a short time, and how to paginate API calls when processing thousands of attributes, but none of those processes help speed up the single-threaded processing of thousands of repos.

What would help is having several threads working concurrently. Thankfully, GitHub Actions makes this pretty easy to do! I did have to teach bash some arithmetic, but I was able to make it work.

Let’s talk about theory, then how I implemented this change in my custom bash program, and the limitations I faced and how I solved them.

I’ve Moved To Lets Do DevOps

From here on out, all content will be published on Lets Do DevOps for free. I want to give back to the community as much as I’m able. Please subscribe there to avoid missing any content. Thanks all ❤.

You can find this article here:

🔥Let's Do DevOps: Shard GitHub Actions Workload over Many Concurrent Builders