Some language and math that come in handy when you talk about or fight Distributed Denial of Service…

Distributed Denial of Service: an attack that uses a number of attacking nodes that overwhelm the target with network, web, or application traffic.  DDoS implies 100 or more nodes attacking the same target, although just about everybody has their own threshold for what consitutes “distributed”.

Command and Control (C2): how the attackers get instructions to the attacking nodes.  This could be automated in the case of a botnet (and probably what defines a botnet if you think about it too hard) or done manually as in the case of some booter scripts, or, as in the case of hacktivists, done with IRC, flyers, manifestos, and forums.  Different C2 has different strengths and weaknesses.

Node: a unique IP address that is participating in a DDoS. Not to be confused with node.js.  =)

Lethality: The lethality of a DDoS is a function of the number of attacking nodes times the average bandwidth per node with efficiency multipliers for how high in the technology stack that the attack goes (layer 3/4 versus 5-7) and if the nodes all attack at the same time (determined by the quality of the attacker’s command and control (C2) ).  It really is a brute force numbers game for the most part.

Average Bandwidth:

• Home users in US: 1-2 Mbps
• Home users in South America, Africa, South Asia: .5 Mbps
• Home users in South Korea, Japan: 5Mbps
• Virtual Private Server: 100Mbps
• Core Routers: 1000Mbps and up

Number of Nodes: divide the total bandwidth of attack traffic received by the average node bandwidth to determine how many attacking nodes there are.  So, for example, a hacktivist army attacking a site and bringing 2Gbps of attack traffic has around 2,000-4,000 participants.

Recruitment: how fast the attackers (botnet via malware, hacktivists, homebrew botnet, etc) can add nodes to the attack.  This could also be correlated with rates of infection for botnets consisting of home PC users, rates of exploits for servers, number of hacktivists joining in the campaign, etc.

Attrition: how fast the attackers lose nodes.  This could be due to ISPs blocking node access due to detection of attack traffic or bandwidth caps, hacktivists headed off to work during the week, the end of a significant campaign, or the general lack of interest in the attack.

Rate of Growth or Decay of an Attack: total size of attacking nodes plus recruitment minus attrition.

Cute Bot Couple photo by Jenn and Tony Bot

• DDoS Planning: Business Continuity with a Twist
• DDoS and Elections
• Keeping Up With the DDoS Kids
• My Month of Entertainment
• The Rise of the Slow Denial of Service

I’ve noticed a trend over the past 6 months: DDoS traffic associated with elections.  A quick sampling of news will show the following:

• http://www.opendemocracy.net/od-russia/irina-borogan-andrei-soldatov/kremlin-and-hackers-partners-in-crime <-Russia
• http://www.theregister.co.uk/2012/03/26/hong_kong_vote_hack/ <-Hong Kong
• http://www.theregister.co.uk/2011/12/07/korean_election_ddos_row/ <-Korea
• http://www.cbc.ca/news/politics/story/2012/03/27/pol-ndp-voting-disruption-deliberate.html <-Canada (rybolov: yikes!)
• http://www.csoonline.com/article/700523/ddos-attackers-target-russian-election-webcams <-Russia again

Last week it picked up again with the re-inauguration of Vladimir Putin.

• http://en.ria.ru/russia/20120506/173265737.html
• http://www.washingtonpost.com/world/europe/peaceful-protest-turns-violent-in-moscow/2012/05/06/gIQAFti35T_story.html
• http://english.ruvr.ru/2012_05_06/73954450/
• http://www.rferl.org/content/independent_russian_daily_hit_by_dos_attack_kommersant/24571463.html

And then yesterday, Ustream and their awesome response: which, in the Rybolov-paraphrased version read something like: “We shall go on to the end. We shall fight in France, we shall fight on the Interblagosphere, we shall fight with growing confidence and growing strength in our blocking capabilities, we shall defend our videostreams, whatever the cost may be. We shall fight on the routers, we shall fight on the load balancers, we shall fight in the applications and in the databases, we shall fight by building our own Russian subsite; we shall never surrender!!!!1111” (Ref)

Afghanistan Presidential Elections 2004 photo by rybolov.

So why all this political activity?  A couple of reasons that I can point to:

• Elections are a point-in-time.  It’s critical for one day.  Anything that has a short window of time is a good DDoS target.
• DDoS is easy to do.  Especially for the Russians.  Some of them already have big botnets they’re using for other things.
• Other DDoS campaigns.  Chaotic Actors (Anonymous and their offshoots and factions) have demonstrated that DDoS has at a minimum PR value and at the maximum financial and political value.
• Campaign sites are usually put up very quickly.  They don’t have much supporting infrastructure and full/paid/professional staffing.
• Elections are IRL Flash Mobs.  Traffic to a campaign site increases slowly at first then exponentially the closer you get to the day of the election.  This stresses what infrastructure is in place and design ideas that seemed good at the time but that don’t scale with the increased load.

So is this the future of political campaigns?  I definitely think it is.  Just like any other type of web traffic, as soon as somebody figures out how to use the technology for their benefit (information sharing => eCommerce => online banking => political fundraising), a generation later somebody else figures out how to deny that benefit.

How to combat election DDoS:

• Have a plan.  You know that the site is going to get flooded the week of the election.  Prepare accordingly.  *ahem* Expect them.
• Tune applications and do caching at the database, application, webserver, load balancer, content delivery network, etc.
• Throw out the dynamic site.  On election day, people just want to know a handful of things.  Put those on a static version of the site and switch to that.  Even if you have to publish by hand every 30 minutes, it’s better than taking a huge outage.
• Manage the non-web traffic.  SYN and UDP floods have been around for years and years and still work in some cases.  For these attacks, you need lots of bandwidth and something that does blocking: these point to a service provider that offers DDoS protection.

It’s going to be an interesting November.

• Keeping Up With the DDoS Kids
• DDoS Planning: Business Continuity with a Twist
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target

Ah yes, you now know how I spend my Saturday mornings lately.

• DDoS Planning: Business Continuity with a Twist
• The Rise of the Slow Denial of Service
• Training the Apache Killers
• Oh to be a Program Manager
• A Little Story About a Tool Named #RefRef

Let me tell you a little story.

So September 17th was Constitution Day and was celebrated by protestors in most major cities across the US with a sizable percentage of folks on Wall Street in NYC.  In conjunction with this protest, a new Denial-of-Service tool, #RefRef, was supposed to be released.  It supposedly used some SQL Injection techniques to put a file (originally listed as a JavaScript but Java is more believable) on application or database servers that then created massive amounts of OS load, thereby crippling the server.  The press coverage of the tool does have the quote of the year: ““Imagine giving a large beast a simple carrot, [and then] watching the best choke itself to death.”  Seriously?

Then came the 17th.  I checked the site, whoa, there is some perl code there.  Then I read it and it sounded nothing like the tool as described.  Rumor around the Intertubes was that #RefRef was/is a hoax and that the people responsible were collecting donations for R&D.

This is what we actually have for the tool that was released on the RefRef site does:

GET /%20and%20(select+benchmark(99999999999,0x70726f62616e646f70726f62616e646f70726f62616e646f)) HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: localhost
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.1.12) Gecko/20080201Firefox/2.0.0.12

The way this works is that it requests a large amount of benchmark queries against the database.  This is very similar to SQL Injection in that the request contains database commands which are then passed by the application server to the database.  In this case, the SQL command is “benchmark” which executes the query multiple times to build test performance of the query.  As you would guess, it generates a ton of database server load.  However, it’s only applicable to MySQL.

• Apache Killer Effects on Target
• Keeping Up With the DDoS Kids
• #RefRef the Vaporware DoS Tool
• DDoS Planning: Business Continuity with a Twist
• The Rise of the Slow Denial of Service

Here at IKANHAZFIZMA, we’re training the next generation of Apache webserver Denial-of-Service gurus.  It involves punching bags, some nomz for the troops, and lots of requests for kibble.

• The Rise of the Slow Denial of Service
• A Little Story About a Tool Named #RefRef
• DDoS and Elections
• DDoS Planning: Business Continuity with a Twist
• no rly, iz protest

Usually when you think about Denial of Service attacks nowadays, most people think up images of the Anonymous kids running their copy of LOIC in a hivemind or Russian Gangsters building a botnet to run an online protection racket.  Now there is a new-ish type of attack technique floating around which I believe will become more important over the next year or two: the slow http attacks.

Refs:

• Slowloris
• Slow HTTP POST
• RU Dead Yet (RUDY)
• Tor’s Hammer
• XerXes

How Slow DOS Works

Webservers run an interesting version of process management.  When you start an Apache server, it starts a master process that spawns a number of listener processes (or threads) as defined by StartServers (5-10 is a good starting number).  Each listener serves a number of requests, defined by MaxRequestsPerChild (1000 is a good number here), and then dies to be replaced by another process/thread by the master server.  This is done so that if there are any applications that leak memory, they won’t hang.  As more requests are received, more processes/threads are spawned up to the MaxClients setting.  MaxClients is designed to throttle the number of processes so that Apache doesn’t forkbomb and the OS become unmanageable because it’s thrashing to swap.  There are also some rules for weaning off idle processes but those are immaterial to what we’re trying to do today.

Go read my previous post on Apache tuning and stress testing for the background on server pool management.

What happens in a slow DOS is that the attack tools sends an HTTP request that never finishes.  As a result, each listener process never finishes its quota of MaxRequestsPerChild so that it can die.  By sending a small amount of never-complete requests, Apache gladly spawns new processes/threads up to MaxClients at which point it fails to answer requests and the site is DOS’ed.  The higher the rate of listener process turnover, the faster the server stops answering requests.  For a poorly tuned webserver configuration with MaxClients set too high, the server starts thrashing to swap before it hits MaxClients and to top it off, the server is unresponsive even to ssh connections and needs a hard boot.

The beauty of this is that the theoretical minimum number of requests to make a server hang for a well-tuned Apache is equal to MaxClients.  This attack can also take out web boundary devices: reverse proxies, Web Application Firewalls, Load Balancers, Content Switches, and anything else that receives HTTP(S).

Post photo by Salim Virji.

Advantages to Slow DOS Attacks

There are a couple of reasons why slow DOS tools are getting research and development this year and I see them growing in popularity.

• Speed and Simplicity:  Slow DOS attacks are quick to take down a server.  One attacker can take down a website without trying to build a botnet or cooordinate attack times and targets with 3000 college students and young professionals.
• TOR:  With volume-based attacks like the Low Orbit Ion Cannon, it doesn’t make sense to route attack traffic through TOR.  TOR adds latency, throttles the amount of requests that the attacker can send, and might eventually fail before the target’s network does.  Using TOR keeps the defender from tracking you back to your real location.
• Server Logging: Because the request is never completed, most servers don’t make a log.  This makes it very hard to detect or troubleshoot which means it takes longer to mitigate.  I’m interested in exceptions if you know specifics on which webserver/tool combinations make webtraffic logs.
• IDS Evasion: Most DOS tools are volume-based attack.  There are IDS rules to detect these: usually by counting the number of TCP SYN traffic coming from each IP address in a particular span of time and flagging the traffic when a threshold is exceeded.  By using a slow DOS tool that sends requests via SSL, IDS has no idea that you’re sending it slow DOS traffic.
• Stay out of the “Crowbar Hotel”:  Use the Ion Cannon, make logs on the target system, go to jail.  Use slow DOS with TOR and SSL, leave less traces, avoid having friends that will trade you for a pack of cigarettes.

Defenses

This part is fun, and by that I mean “it sucks”.  There are some things that help, but there isn’t a single solution that makes the problem go away.

• Know how to detect it.  This is the hard one.  What you’re looking for is Apache spawned out to MaxClients but not logging a comparative volume of traffic.  IE, the servers are hung up waiting for that one last request to finish and shucking all other requests.
• “ps aux | grep apache2 | grep start | wc -l” is equal to MaxClients +2.
• Your webserver isn’t logging the normal amount of requests.  Use some grep-foo and “wc -l” to compare traffic from: a month ago, a day ago, an hour ago, and the last 5 minutes.
• Disable POST as a method if you don’t need it.  Some of the more advanced techniques rely on the fact that POST can contain more headers and more body data.
• Use an astronomically high number of servers.  If your server processes can timeout and respawn faster than the slow DOS can hang them, you win.  If you had maybe 3000 servers, you wouldn’t have to worry about this.  Don’t have 3000 servers, I might have some you could use.
• Set a lower connection timeout.  Something like 15-30 seconds will keep Apache humming along.
• Limit the request size.  1500 bytes is pretty small, 3K is a pretty good value to set.  Note that this needs testing, it will break some things.
• Block TOR exit nodes before the traffic reaches your webservers (IE, at layer 3/4).  TOR has a list of these.

• Apache Killer Effects on Target
• Stress-Test Apache with Intent to Tune: BSOFH Tip for the Software Masochist
• Keeping Up With the DDoS Kids
• Training the Apache Killers
• A Little Story About a Tool Named #RefRef