What to do if you find yourself being DDoSed

The absolute first thing you should do is to call your upstream ISP and see what assistance they can afford.

If they are unhelpful or unable to help, or you just want to start this process in parallel, read on. In fact, it’s worth thinking about responding to potential incidents before they actually happen.

Initially, try to find out how many bots are hitting your server? Is there any pattern to the requests?

If you can’t answer this, then you won’t get very far. You need to install some monitoring tools; personally I like to use syslog to transfer everything useful to a central server where I can examine it all. If you can arrange for all the different machines to have the same system clock using the NTP protocol, so much the better, it will save you headaches later on.

At this stage, you’ll need to be able to recognise bad requests from valid ones. If you can’t, you won’t be able to teach the computer how to. In one incident, malicious requests stood out as distinctive because they were only ever requesting two pages, and had malformed HTTP headers. In other words, great fail2ban fodder. (Fail2ban is an automatic blocking tool, which is typically configured to block people attempting to do something too many times in a particular timeframe.)

So, can you use fail2ban to drop them? If it’s a lot of bots, you’re likely to need to use ipset rather than individual iptables rules, but this is fairly straightforward as long as you’re below 65,000 odd attacking IPs. Ipset will deal with a hash table of up to 65,535 unique IP addresses, and is vastly quicker than the same 65,000 separate rules using iptables for each. (If you’re above this number, or your Internet pipe is just full, you’d probably best get on the phone to your upstream.)

If you can get a working fail2ban / ipset solution to kick these bots off, that is more than half the battle. If the attacker is not being effective, they’ll probably get bored and turn to an easier target soon.

Otherwise, can you use tools like mod_security to drop as soon as possible? mod_security’s “deny” target takes the trouble to reply with a 403, or whatever you customise it to. Don’t be polite, just use the “drop” target and kill the TCP connection right away, you’ll save yourself an apache worker process. mod_security will do all sorts of pattern matching, or even geographical IP lookups if you know where your customers are.

While you’re doing this, tune the timeouts down through your TCP/IP stack, so half-open connections are relinquished sooner, and set TCP retries lower than the default.

Basically, try to drop every bad connection as soon as you can – iptables is best, if not, with mod_security. If you can identify malicious hosts, feed an ipset rule to reject connections at your boundary. Implement something even if it’s not perfect, make sure it’s not affecting your real users and then check the bandwidth and CPU you’re using to see if it’s helped matters at all.

The strategy outline above was effective against an attacker deploying around 12,000 bots against a particular target; it took maybe a day to implement a reasonable filter which made the difference between the site being usable and not, and soon after this the attacker gave up.

NB:This was a fairly naïve attack – a sophisticated attacker would have done better, or maybe even simply filled the upstream internet connection with packets, and there’s not a lot you can do about that as an end-user of the bandwidth.