This week, we continued to look into network routing solutions for our audio stream, plus we partook in a little bit of good old fashioned April foolin’. For more information, read on.
Following last week’s work on our radio network, we took a deeper dive into how our multi-regional network functions at the configuration level, the pros and cons of regional routing at the DNS level, and how our network impacts users on a more practical level.
Previously, we had a custom internal system which we referred to as the “director.” This system was thought to be responsible for traffic management prior to station connections, but as we found, this was mostly unused by the system. The system depends in part on a reverse proxy with a geolocation scheme to determine where to send connecting listeners. While listeners are successfully diverted to their regional server, the problem with this is that reverse proxies make upstream servers transparent to the user by sending traffic back through the original server. This makes regions useful only for spreading load across multiple servers rather than allowing listeners to directly connect to regional servers. The previous system would redirect traffic rather than proxy it which had the desired effect, but at the cost of compatibility with players and other client systems which could not gracefully handle redirects prior to completing a new connection. Previous research had suggested this type of traffic flow could be avoided while maintaining a non-region specific URL.
There are two paths we can take. One is to revert to the previous system where listeners would be redirected (and not proxied) to the region server closest to them. This is the cheapest and easiest solution to this problem. The other option is to utilize a DNS-based geolocation service to divert listeners to regional servers at the DNS level. This is a solution that we would need to pay for at $5 base cost + $.50 per 500,000 requests. Last month, we received about 45 million requests, so the total cost for us to utilize this service would be about $50 per month (variable based on traffic). While this might not sound like much to people familiar with server administration costs, this would be cost prohibitive for us. Combined with our existing server costs, this would consume over 100% of our Indigo memberships for what would benefit a very small number of users.
On the practical front, regional servers appear to impact our listeners in Asia the most. We ran cross-regional tests to find out more about the practical benefits of utilizing this system. Asia’s latency jumps higher than the max audio buffer when streaming from North America most of the time, and from Europe some of the time, so they appear to benefit the most from having a server in their region. Luckily, audio streams don’t need super low latency because audio is buffered ahead of time. We could get around this by increasing the size of the buffer server-side, but that would place listeners in this region farther behind “real-time,” or when audio is initially broadcast from the source. For interactive events like our Live shows, we try to minimize this delay as much as possible.
In summary, we’re likely going to move back to a regional redirect system which will continue to work fine for the majority of listeners, but will require some to connect to a specific region for players and clients that can’t handle redirects. Most listeners won’t notice a difference, but we’ll have some additional information posted on our website to help those affected with certain edge cases.
At last, we threw a dance party on our website for April 1st. DMJ, Xara, and General Coolman teleported from 2017 to show us what they’re capable of. Also, Reimu transformed into Nano with his shutter shades from a prior year.
That’s all for now, thanks for your support this week, and thanks for listening!
[Knowledge #111]