🔥Let’s Do DevOps: GitHub API Paginated Calls (More than 1K Repos!)

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I previously wrote a story about setting permissions and branch protections on hundreds of repos. That tool works fantastically, and is a standard at my work. However, we have now crossed over 1k repos!

Which is amazing, but also caused the tool I wrote to no longer get all of them. As I came to find out, the gh cli tool I used to get all the repo names uses the GitHub Search API, which is limited to 1k results, and isn’t paginated. That means you simply can’t get more repos using it (without tricks like calling repos since a certain date, which isn’t very reliable).

This snuck up on us — when I wrote the tool, we had fewer than 1k repos, so I saw no error messages, and the number of repos the gh repo list command printed matched the number we had! It was only through examining the logs after we passed 1k repos that I saw we were only getting the first 1k of them.

I rewrote the Action to use the REST API, which only provides 100 repos at a time, but is paginated, so you can simply keep calling the API until all repos are returned. Let’s walk through how I implemented a…