If you visited music4dance.net in the last week of October (between about October 23rd and 26th), you probably noticed it was very sluggish, possibly even completely unresponsive. I’m very sorry if that affected you. The timing was particularly unfortunate for a couple of reasons. First, this happened while I was on vacation overseas, traveling with only an iPad to diagnose and resolve issues. Second, it was the week leading up to Halloween, so DJs using music4dance to build or enhance their Halloween Playlists were justifiably frustrated. Thanks for bearing with me. All the emails I got reporting the outage were very kind, and I appreciate that you took the outage in stride.
What Happened?
While I was on vacation overseas, I noticed the server was rebooting several times a day, which was unusual. But whenever I checked, things were working. Since the server is in the western US and I was in South Korea, I attributed the latency to geographic issues.
It wasn’t until you started emailing me that I took some time out to log in and investigate.
Despite the title of this post (which I couldn’t resist), the proximate cause of this outage wasn’t bot traffic like the ones I’ve blogged about before, or the attempts to hack the site, which I didn’t end up blogging about. It was due to memory pressure on the very small virtual computer running the website in the cloud. It happened because my hosting provider replaced the operating system/libraries underneath me with a version that changed behavior (possibly due to a bug) and caused some kind of memory leak.
But it would not have been as severe, or may not have happened at all, if over 90% of the traffic coming to the site (and causing pressure) wasn’t bot traffic of some sort. The direct human traffic probably wouldn’t have stressed the server enough to cause downtime, even with the bug.
This is part of a larger trend for small, independent websites. Here are a few articles about that trend, if you’re interested:
- AI crawler wars threaten to make the web more closed for everyone | MIT Technology Review
- Please stop externalizing your costs directly into my face
- Rise of AI crawlers and bots causing web traffic havoc – Computerworld
- Meta web crawlers crash small business websites as AI bots surge – Fast Company
- You cannot have our user’s data
What did I do?
The immediate fix, which I could perform from an iPad (ironically – because I had AI to do the heavy lifting), was to scale up to the next biggest virtual machine that my host supports. At the very low end of the cost spectrum, that meant I ended up doubling the available resources. That immediately mitigated the issue, and the site stopped crashing. But double the resources means double the cost, which I would prefer not to sustain indefinitely.
When I got back home, I spent some time looking into other ways to mitigate the issue. I’d already invested quite a bit in recognizing and ignoring malicious bots and improving my code’s resilience. Because of that work, the server could restart and reconnect reliably without human intervention, and even while on vacation, I could diagnose and mitigate an issue like this.
I also discovered during the investigation that there were several cheap things I could do to reduce the size of the web pages my server returns, which helped quite a bit.
But I hadn’t taken the next obvious step: adding an edge network between the internet and my website to filter traffic, mostly because the lowest-cost version my hosting provider offered came with a minimum monthly fee greater than my server cost (even the bigger version I’m running right now). On further investigation, Cloudflare has a free offering that looks like it will fit the bill. I’ve put that in place and am tuning up their bot protections and other methods to mitigate many of the issues that have been plaguing the site recently. If that works, I might even feel comfortable scaling back to the smallest available server and returning to the monthly cost I was shelling out before this incident.
What this costs me
As I said in March, music4dance has never paid for itself. I cover the costs, and I’ve been happy to, because the site is a labor of love. But I want to be transparent about where the time and money go:
- Server cost doubled. The bigger server costs about twice as much as the old one. This isn’t a full doubling of the hosting cost, since I didn’t have to increase the size/capacity of the resources beyond the server instance, but it’s still significant.
- Bandwidth is billed by the gigabyte, and the site was sending out several gigabytes a day, mostly to robots. The changes above cut that substantially, so I hope this won’t be an issue moving forward.
- The biggest cost is time. I’ve been spending well over half the time I devote to music4dance resolving issues like these or building out infrastructure to (hopefully) avoid them coming up in the first place. This is time I’d rather spend adding new features, curating the catalog, and responding to all the interesting questions I get from you. Or writing posts about the relationship between music and dance, rather than posts like this one.
How you can help
- If you use music4dance regularly and haven’t subscribed, please consider it. Subscriptions should be what keep the lights on, and more subscriptions would directly offset costs like these.
- Tell me when the site is slow or broken, and what page you were on. Reports from real people are still the best early warning I have.
- Keep sharing the site. As I said in March: the last thing I want is for usage to go down as the solution. I’d just rather the visitors be dancers than robots.
I appreciate your patience and your support in this adventure.
Step of the Month
The “Step of the Month” for November is to please spread the word – let a fellow dancer or DJ know how much you get out of music4dance and encourage them to give it a try. Or pass along your favorite post (hopefully one about a feature or something fun rather than this one).
Is music4dance becoming too popular?
I first published music4dance in 2013 and, within a year or so, started seeing traffic measurable in the thousands of active users per month. Usage plateaued there for over a decade, but last year, the numbers started climbing rapidly – into ranges better measured in tens of thousands. Now these are very rough measures from…
Bug Report
Bug Bounty: We would like to provide a reward to people who take the time to report bugs. So for each bug that you report that hasn’t been reported by others, we will give you a week of our premium service for free (or extend an existing premium membership by a week). We must be…








