🔥Let’s Do DevOps: GitHub API Paginated Calls (More than 1K Repos!)

Kyler Middleton
6 min readMar 13

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I previously wrote a story about setting permissions and branch protections on hundreds of repos. That tool works fantastically, and is a standard at my work. However, we have now crossed over 1k repos!

Which is amazing, but also caused the tool I wrote to no longer get all of them. As I came to find out, the gh cli tool I used to get all the repo names uses the GitHub Search API, which is limited to 1k results, and isn’t paginated. That means you simply can’t get more repos using it (without tricks like calling repos since a certain date, which isn’t very reliable).

This snuck up on us — when I wrote the tool, we had fewer than 1k repos, so I saw no error messages, and the number of repos the gh repo list command printed matched the number we had! It was only through examining the logs after we passed 1k repos that I saw we were only getting the first 1k of them.

I rewrote the Action to use the REST API, which only provides 100 repos at a time, but is paginated, so you can simply keep calling the API until all repos are returned. Let’s walk through how I implemented a paginated API call in bash for a GitHub Action to get more than 1k repos in our GitHub Org!

How Many Repos Do I Have?

It’s harder than it seems to find out exactly how many repos you have in your GitHub Org once you have more than 1k. Below that number, you can see the exact count in the GitHub.com UI on the Repositories tab.

But after you have more than 1k, the repository number is rounded. That makes it hard to double-check our automation and make sure we are finding all of them!

We could use the output from the gh repo list command, but it’s just bitten me with a limitation — plus, I don’t want to rely on an informational output from the gh cli that might change at any time. Rather, let’s try and find that…

Kyler Middleton

DevNetSecOps, DevRel, cloud security chick. I will teach you, it’s unavoidable. She/Her 🏳️‍🌈🏳️‍🌈, INFJ-A, support the EFF!