I just spent the better part of the last few months investigating a completely non-sensical problem created by a single toggle that frankly still doesnt explain what actually happened. And I feel like writting it down for other homelabbers amusement.
The problem: Apple's App Store and the Apple Music app failed to load on any iOS, or macOS device when connected to my home network.
For the homelabbers out there, I bet the first thing that popped into your head was DNS. Somewhere, somehow, it has to be a DNS problem. And thats where I sunk a vast majority of my time investigating. For good reason too, my network is configured with some relatively strict DNS rules. First, I have 3 instances of Pi-Hole DNS which are synchronized using Orbital Sync. At the start of this I had a half a dozen lists with just under 1 million domains blocked. I may not have a good reason for the massive lists, but I do have a good reason for running 3 instances of pi-hole. At the firewall level I have banned all other devices from sending DNS requests outside of the network. Meaning if my pi-hole devices are down, we have no way of resolving DNS. This is to combat smart devices using hardcoded DNS servers to bypass my pi-hole blocklists.
With all that said I tried every combination of disabling blocking, removing lists, rebooting, tracking down published apple service domains and whitelisting them all to no avail. I even disabled all my firewall rules that touched on DNS to see if maybe Apple was doing something with hardcoded DNS. Luckily that isnt the case, and the only conculsion I could make was that it was NOT a DNS problem. That was of course until I switched my iPhone over to 1.1.1.1 for DNS as a last ditch effort and it worked. Now, Id like to think that this was definitive proof that it was a DNS problem all along, and the fact that I was using Pi-Hole for my DNS led me to the discourse forums to see if anyone else had been encountering a similar issue.
The Solution
After a bit of searching around I found this post where the user described my exact problem, albeit with a less convoluted configuration, and with the same firewall software. He also came back to share what his solution like the beautiful person that he is.
Since I expose ports on my network to the public I also have been pretty liberal with outright banning traffic to and from entire regions. Particularly regions that showed up in my logs as suspicious via the intrusion prevention functionality of my firewall. As of now I have 15 countries selected that I blocked all incoming and outgoing traffic too. Though none of them seem like countries that would break App Store and Apple Music functionality for me in the United States. The blocked countries are: Afghanistan, Brazil, China, Egypt, Israel, Iraq, Iran, North Korea, Lithuania, Pakistan, Russia, Saudi Arabia, Turkey, Ukraine, and Vietnam.
Surely it would make no sense that my connection to Apple's servers from my home in Northern California would have to route through ANY of those countries right? So I started running a traceroute for the suspected problematic domain: e673.dsce9.akamaiedge.net. And as I suspected every IP address shown was somewhere in California. So what gives? Well...I don't know. What I do know now is that setting my Unifi firewall to block regional traffic in both directions, or outgoing only causes the connection to the Apple App Store, and Apple Music to fail. Once I changed that value to block incoming connections from the counties listed above I was able to access the app store again.
This however just raises more questions. First, how is Unifi handling this geo location blocking. Second, what changed to cause this failure to start happening(since my network rules hadn't changed for years before this)? Third, is some hop in the path to these servers incorrectly geo tagged by Ubiquity, or actually in one of the banned regions?
If you've made it this far into my poorly formatted ramblings and you have some ideas as to what the hell happened, Id love to hear about it! You can find me on various social media's here.