Over the past few weeks, we’ve been monitoring the server system, continued to work on the broadcast deployment, and have been splitting attention between projects and new work. For more details, read on.
Way back in January of 2026, we started looking into a new hosting system for websites, apps, and pretty much everything else we end up serving out to our users and listeners. This system is designed in a way that is supposed to provide increased resistance to service interruptions and enable easier migrations (after the most recent one) when it’s time to swap over to new hardware. This system has also been extensively tested in a local environment on physical hardware where we were able to refine configurations and ensure we had a working and stable system before moving into production.
About 3 months prior to the migration of GensokyoRadio.net, we started testing this new system in a production environment where everything would eventually be migrated to. While testing the production system, we realized that there were some limitations related to one of the key systems involved in how availability is detected and how traffic is automatically switched from one system to another. This is generally related to the topic of failover; when a system becomes unavailable, there’s another system running that can pick up where the other system left off. With the old system, if anything happened to the one server, everything would become unavailable until an admin could intervene.
With the new system, ideally the failover process would automatically handle things and redirect traffic, but one of the limitations of the production infrastructure we’re hosted on prevents us from implementing failover in the way it was done in the local test environment. This led to a bit of a hackish solution that *technically* works, but clearly has some drawbacks against the intended method: it’s slower, it sometimes fails to run so the switchover never happens, and sometimes the switchover is too aggressive so the system ends up split over what the configuration says and how the traffic actually gets routed. This has resulted in a similar-feeling system as what we had before, except potentially worse because this failover is more aggressive about trying to direct traffic to a working node.
After making many adjustments to the custom script that is meant to work in place of the infrastructure not supporting the intended failover method, we ended up adjusting the system so that there is only one eligible node available for failover. This means that network blips won’t redirect traffic like before, but users will need to ride out those periods of (hopefully brief) instability. There are a couple of other solutions we might be able to implement which involve externally monitoring response codes or redirecting traffic at the DNS level, but for the moment, it appears that using a single node to handle traffic has fixed the network issues we’ve been seeing over the past few weeks.
With networking improved, we then turn our attention to stability issues with the radio station’s website and stream connectivity. At the time of writing, whenever a listener connects to the radio station via any of its endpoints, there is an authentication script that runs in order to check for ban status and log the listener before handing out an auth flag which tells the Icecast endpoint server that the connection may proceed. If a user has a temp ban or the authentication script is unavailable, the endpoint server never gets the flag, and the connection is rejected which results in a 401 authentication required error (which is why a login box may appear when listening on non-Indigo endpoints). In this case, the station stream can be operating normally, but new connections can’t be made until the authentication script is available again.
Furthermore, the endpoint servers fail in a “closed” state by default. Making this distinction is important because the authentication script is currently hosted within the same space as the radio station’s website, so when the website becomes unavailable, the authentication process also becomes unavailable.
There are two solutions we’ll look into for this as well, with the first being to implement a self-monitoring and healing system. As a radio station that’s been in operation for over 15 years and with regular listeners around the world, the website receives constant and regular requests in the form of page and media retrieval, API requests, connection authentication, and other automated traffic such as scanner spam and live info scraping. Even with multiple replicas and multiple servers handling these requests, we’ve found that the new system eventually becomes stuck in a state where requests aren’t immediately handled and legitimate users start to see 504 timeout errors. One of the things the new system can do natively is run a self-healing loop where, when an instance of the webserver doesn’t return a normal response code, it can make that instance unavailable and redirect traffic to other working instances while it resolves or restarts the malfunctioning instance. The exact implementation of this may take some adjustment, but we’ve already implemented this and will be monitoring the results.
The other solution in terms of continued stream availability is to split out the auth process from the rest of the webserver so that when one goes down, the other is not affected. This seems like a logical next step to improve the connection experience for listeners. Additionally, we could also implement a system which allows the endpoints to fail open instead of closed, but this has other issues such as ignoring temp bans and listener logging (which would affect the ability for listeners to rate songs).
On the topic of rating songs and listener logging, the system that logs listeners has been very flaky since migration; listeners will sometimes be able to rate songs, and sometimes not. Listener counts have been unreliable, the system which ties together IPs and locations has been slower than in the pre-migration system, and listener exits haven’t always been recorded. In general, this is a system which is looking like it needs to be revamped because it’s not behaving the way it was before, and it’s not immediately clear why. This is yet another high-priority sidequest that we’ll need to address before returning to our longer-term work related to the store and Station.
Finally, to add to the pile we’ve already discussed, there has been an increase in work external to any of LSM’s projects. This has resulted in some overdue upgrades, with the most notable one being the replacement of hardware which contributed to the delayed starts during the previous two Live broadcasts. We also needed to replace a storage drive which in part holds historic Gensokyo Radio audio and photos. Both of these were unexpected expenses, and together they would have created a much more difficult situation if not for our listener supporters, so thank you for that. Beyond the immediate benefits, external work has begun to occupy more time which will result in less time being dedicated to LSM’s projects. This actually more closely resembles a “return to normal” in the sense that we’ve had an abnormally high amount of solo dev time available to dedicate to LSM’s projects for the past few years.
Regardless of the amount of time available, our project list remains and we will work through each item in due time and with due care. As always, thank you for your continued support, and thanks for listening!
[Knowledge #227]