Hacker Takes Down Millions of Websites
Most of my clients have heard me complain about Mondays before -- they're terrible. Not in a mythical way, or an "ugh! I have to go back to work" way, but for many reasons that are real and provable.
But today took the cake.
Just after lunchtime, as I was settling into a groove and starting to work with Matt on a particular project, I received an email from a client. "Our website seems to be down. What's up?" Then immediately, another similar email from another client. But these two sites were on completely different servers. I stopped what I was doing and started looking the status of ours servers and panic struck: "Oh my God! The entire server farm is down!" And just as that deer-in-the-headlights look hit me, the phone started lighting up with incoming calls. With almost no warning I went from zero-to-sixty, in mere seconds. I jumped up and bolted out the door, telling Matt "I've gotta run to the datacenter -- something bad is going on with the servers!" as I blew past. My phone was starting to explode from constant incoming calls -- everyone at once.
Interestingly, my *very first* gut reaction was that it was a DNS issue -- exactly what it turned out to be. Except I quickly dismissed it as implausible -- not on this scale. And as it turned out, there were a couple red herrings tossed in for good measure that made it particularly tough to separate out the wheat from the chaff. I ended up wasting 5 hours of troubleshooting and diagnostics trying to figure this out and trace it down. I had a tech from the upstream fiber provider on the line with me for several hours as we traced through step by step, whittling it down. It just didn't make sense...
If you haven't heard yet, a member of the hacker group Anonymous has taken credit for taking down millions of websites today.
As it turned out, it had nothing to do with our servers at all. The attack took out not only sites GoDaddy hosts directly, but also any and all sites that use GoDaddy for DNS services. I'll explain.
DNS is like a card catalog system in a public library. Imagine someone stole the card catalog system when you weren't looking. Now there are millions of books in the library, but you have no idea how or where to find them. To draw and even better analogy to websites, imagine the books in the library are just randomly placed on shelves, not in any particular order nor by section -- they're just one giant mish-mash of never-ending rows of books. Without the card catalog system to tell you where to find a particular book, there's no way you'll ever find it. So even if the book is there, nobody can find it.
Now you understand the concept behind DNS. It's a very simple, basic, yet critical function. If DNS is down, nobody can find your website. It all happens behind the scenes -- your web browser (Firefox, Chrome, Safari, Internet Explorer, etc.) will do DNS lookups hundreds or even thousands of times a day, behind the scenes. It all just works. Until it doesn't.
The hacker knew this too (it's not a secret in the IT world). You don't have to take down a million servers/sites -- just take out DNS.
GoDaddy has grown to become one of the largest Internet domain name registrars in the U.S., handling millions of domains. That makes it a juicy target -- a big payday for a hacker.
Now, all of the forensics aren't "in" yet, but from what we can tell so far, it looks like it was a DDoS ("Distributed Denial of Service") attack, a brute force attack that is akin to opening up a fire hose on someone -- just pummeling a server/website with data requests so fast and furious that it overwhelms the servers. This may or may not have necessarily been GoDaddy's "fault" or lapse in security. Culpability remains to be seen just yet, and I'm sure it will be debated for weeks and months to come. But, my gut tells me so far that this probably shouldn't be a black mark on GoDaddy.
However, it is a black mark on Anonymous. Millions of sites went down, resulting in untold loss of revenues (think of all the eCommerce sites that were offline and not selling today), untold loss of productivity from the millions of website owners chasing down why their site isn't working, etc. Anonymous claims to be a "hacktivist" group targeting government agencies and those involved in Internet censorship. Unfortunately, by far, most of these sites are run by small businesses, mom-and-pop shops, non-profits, freelancers and individuals. They didn't stick it to "the man", they stuck it to you and me.
By dinner time service was restored, and we were all feeling a big sigh of relief that there wasn't some catastrophic issue with our servers or infrastructure -- it was a GoDaddy issue, and for them to deal with (and they have the resources and means to do so). As much as it stunk, in the grand scheme of things it was a contained issue. I lost a day of work. Tomorrow we're back on track. As my dad used to say, "I guess in our misfortune, we were fortunate."