A couple weeks ago, I got a call on my cell phone from a number in the United Kingdom that I didn’t recognize. Usually, these unrecognized numbers are spam calls, so I ignored it. But this time, the person calling left what looked like an urgent message as I glanced at the garbled transcript. It was around 6 AM. I wasn’t fully awake. I listened to the voicemail. The voice had a British accent. I didn’t fully understand what he was saying. He claimed to be calling on behalf of the server provider called Cogeco/Peer1 which is the company we use to supply our hosting servers. So I listened again. Then I ran to my computer and confirmed what he was telling me.
One of our servers, where we host many websites, had been taken offline. The official, technical term the provider used in a support ticket was that the IP address (much like a street address for the internet) of the server had been “null routed”, essentially removing it from the network. The server was under a distributed denial of service (DDoS) attack from a swarm of zombie computers.
What?! I have been running servers for over 15 years and have never had such a thing happen. All our servers run layers of protective systems and the software firewall dutifully fends off hundreds of attacks per day. But, under a true DDoS attack, software solutions like these are not enough. The server provider sent a document that illustrated that the server was being hit by requests from thousands of locations every second. The goal is not to break in to the server but simply to overwhelm its resources, denying legitimate access. Not only was this attack affecting our server, but it was causing other devices on the same network to slow down. They had no choice but to shelter the server until the storm passed.
Typically, there are 3 reasons for a server to be hit by a DDoS.
- Money. The attacker is hoping to extract some kind of payment to call off the attack.
- Revenge. The attacker feels slighted or somehow offended by an action taken by the server owner, or some other user on the server.
- Randomness. The attacker unleashes the mob and points it at a randomly selected IP address.
Unfortunately, it’s nearly impossible to track down the source of such an attack. The huge numbers of requests come from thousands of sources. Each unwittingly made part of the attack by malware. The ultimate source of the attack is hidden behind layers of distributed and obfuscated control nodes in the botnet.
So it’s impossible to truly know the motive for the attack. However, reason tells me to eliminate the first two possibilities. If money or revenge had been the goal, I would have been contacted by the perpetrator(s) to either pay up to call off the attack, or to tell me why they were seeking revenge. Plus, I had relatively little they could extract from me financially, and I couldn’t think of anything that my company or any of our hosting clients could possibly have done to become a target of revenge.
Protecting the Server
Most likely, the attacker was someone testing or demonstrating their abilities. While it was a true DDoS it was small compared to those perpetrated against tech giants. In our case, Cogeco/Peer1’s usual policy is to keep the server off the network for 24 hours. However, after much communication I was able to convince them to return the server to the network after about 8 hours. They did so, with the caveat that if the attack had not stopped or the attack resumed later that they would have to null route the server’s IP again. Fortunately, the attack had disappeared and has not returned. This added to my sense that our server was randomly chosen from millions of network connected devices.
It’s one thing to have your website become inaccessible. However, many of our clients hosted on that server also rely on it for email. Suddenly, as the work day dawned, hundreds of people were without email.
While there was little I could do to fix the situation, I spent hours communicating as best i could with clients, explaining the situation over text, the phone and through social media. I am very grateful to everyone with the patience and understanding they showed during the ordeal. Fortunately, once the server was brought back, email held by the sending servers flooded in and was delivered.
It was a terrible day that I hope never recurs. I continue to explore possible ways to prevent or at least ameliorate the effects of a future, rather unlikely, attack. However, no matter what defensive measures are applied, it’s impossible to guarantee stability. Successful attacks are regularly made against much larger organizations with far more resources than we could ever hope to deploy. When bad things happen, the power goes out, the water gets cut off, a terrorist blows up a building, we all just have to pick up and move on. We are doing so.