🔥Let’s Do DevOps: Ingesting 100s of Thousands of Records into Terraform for Fun and Profit 🚀

Kyler Middleton
6 min readJan 4, 2024

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

It’s been a while — I’ve been busy with life, and chasing a 2-year-old toddler around the house takes up a lot of time and generates a lot of snuggles.

Recently, at work I was tasked with helping establishing geo-blocking and geo-permitting for some Azure Firewalls. Unlike Azure Application Gateways, they don’t have native support for geo-based location policies, so we needed to find a list of country-specific IPs.

I was sure that finding that data would be as simple as querying IANA, the Internet Assigned Numbers Authority, for a specific country code, to get all the assigned IPs. Unfortunately, that data wasn’t as easy to get as I assumed it would be. However, our NetOps team was way ahead of me here, and was able to get that working.

MaxMind For Global Domination

This article waswritten by Shannon Ford, an incredibly talented Network and DevOps engineer I have the opportunity to work with. Without his effort to build the data ingrestion from MaxMind, and the automation to refresh it weekly, I couldn’t have built any of the rest of this.

Terraform: Ingest a Country’s Entire IPv4 Internet Range!

So now we have 250 country-level flat files, each containing 10s of thousands of CIDRs that correspond to all the IPv4 CIDR ranges that are assigned to particular companies.

And that’s awesome! We need that! But it’s not in a format that is accessible by our Azure cloud infrastructure. Thankfully, Terraform has some tricks up its sleeve that permit it to ingest external data, including very large flat text files.

--

--

Kyler Middleton

DevNetSecOps, DevRel, cloud security chick. I will teach you, it’s unavoidable. She/Her 🏳️‍🌈🏳️‍🌈, INFJ-A, support the EFF!