This week, we’ll dive into a new failover mechanism we implemented which will help to keep the radio station available globally when a regional streaming server becomes unavailable. For more details, read on.
We know that not everyone is deeply technical or particularly interested in how things work behind the scenes, so we’ll break this down into two parts. First, we’ll go through how the radio station is currently set up at a high level, from broadcast to listener and the major steps in-between; for our more regular listeners, much of this will be review. Second, we’ll dive into how the new update functions on a more technical level and walk through how this gets us closer to a more highly available system overall.
How the Station’s Audio Stream Works
The radio station begins with audio files like the ones that a person might put on their phone to listen to while on an airplane or a place with no internet. These audio files are placed in storage and made available to our broadcast server. The broadcast server has a couple main roles: it determines what plays and when by looking at song play history and the current time, and it merges different pieces of audio such as songs and trailers together by using cut-ins and crossfades as appropriate for each piece of media. This merging of media is output as the radio station’s broadcast audio stream.
From here, Gensokyo Radio uses a small network of servers located in three different regions around the world. Each server receives the broadcast audio stream which is a high-quality, “lossless” stream and converts that stream into the four qualities that are available to listeners (64kbps, 128kbps, 256kbps, and lossless). Each audio stream is then handed off to Icecast which is the software that handles listener connections, and when a listener connects, the audio stream is sent to them from a server that is usually geographically closest to them.
How the System Determines the Closest Server
By making our radio network geographically diverse, we have the option of sending listeners to different servers based on where the system thinks listeners are connecting from, but this doesn’t just happen automatically; something needs to figure out where the system thinks the listener is, map out where the three regional servers are known to be, do some math, and send the listener to the server the system thinks is closest to them. In the past, we handled this logic more manually by having stream.gensokyoradio.net handle incoming requests, used a database to figure out which continent the listener was on, and then redirected them to a regional subdomain that started with na1, eu1, or as1. While this worked for most listeners, there were some applications and integrations that didn’t like seeing the “302 redirect” being returned when the application expected a direct audio stream. To solve this, we looked at ways to map regional servers without requiring redirects, and the answer lies in a core piece of how the modern internet works: DNS.
In a nutshell, DNS helps computers convert domain names like “station.moe” into an IP address which can be used to allow your computer or device to talk to the server that a website is hosted on. In most cases, DNS is a simple mapping that turns “station.moe” into a single IP address like answering the question “Where is station.moe?”, and the DNS server will respond with an IP address. However, it is also possible to have DNS respond differently depending on where the request is coming from, and this is where GeoDNS comes in. To do this, the DNS server keeps a table of IP ranges and their known approximate locations to get an answer for where it thinks the request is coming from, and based on that, it answers with one of three different IP addresses that correspond to the regional streaming servers. As a sidenote, it’s entirely possible for a listener in one region to be sent to an entirely different region because the listener’s IP might be listed in a different range or region, but it should get it right about 99% of the time.
Using GeoDNS with Failover
Up to now, this post has really just been a more in-depth look at a change we made way back in March 2023. At that time, we removed regional subdomains in favor of sending all listener traffic directly to stream.gensokyoradio.net, and here we’ve covered how that system still works without the use of regional subdomains for geo-based redirection; these days, it works at the DNS-level. However, in order to turn this distributed network into one which is also considered a “high availability” network, we need to make the system more resilient by implementing redundancy. When thinking about how to implement redundancy, the first thought might be to spin up secondary servers for each geographic region, but this would require twice as much upstream bandwidth at the broadcast server, it depends on each region’s datacenter to maintain system-wide uptime, and obviously it would cost more to pay for servers that we don’t normally need if everything’s working.
That brings us to this week’s update: we’re introducing a new system which handles the scenario where one region’s server becomes unavailable. Normally, this would effectively lock out the listeners in that region without the need for something like a VPN to connect to a different region. To address this, we’ve updated our DNS configuration to handle failover. Let’s go over what this does:
- This new system constantly monitors every region for issues.
- If a server becomes unavailable for over a minute, DNS will change the way it answers the question “Where is stream.gensokyoradio.net?" so that the listener is directed to a different region which is still available.
- This has the benefit of costing zero additional dollars per month for us to implement and use while improving global stream availability and uptime.
- Although listeners may be temporarily pointed to a server that’s farther away, latency testing reveals that this introduces 50-100ms more in the worst case scenario, and this is generally negligible for audio streaming.
This solves a specific case where, if one regional server becomes unavailable, the other regional servers can pick up the additional traffic. This helps us get closer to a “high availability” system, but it still only solves one particular problem. In the case where the website / auth server becomes unavailable, the station’s endpoints will still “fail closed,” meaning that they become unavailable in the event of a web failure. There are multiple solutions to address this and we’ll look at these next alongside the continued system stability improvements that are happening behind the scenes.
That’s all for this update, stay tuned for the announcement of our next broadcast soon as well as more news and updates like this one, and thanks for listening!
[Knowledge #230]