Some language and math that come in handy when you talk about or fight Distributed Denial of Service…

Distributed Denial of Service: an attack that uses a number of attacking nodes that overwhelm the target with network, web, or application traffic.  DDoS implies 100 or more nodes attacking the same target, although just about everybody has their own threshold for what consitutes “distributed”.

Command and Control (C2): how the attackers get instructions to the attacking nodes.  This could be automated in the case of a botnet (and probably what defines a botnet if you think about it too hard) or done manually as in the case of some booter scripts, or, as in the case of hacktivists, done with IRC, flyers, manifestos, and forums.  Different C2 has different strengths and weaknesses.

Node: a unique IP address that is participating in a DDoS. Not to be confused with node.js.  =)

Lethality: The lethality of a DDoS is a function of the number of attacking nodes times the average bandwidth per node with efficiency multipliers for how high in the technology stack that the attack goes (layer 3/4 versus 5-7) and if the nodes all attack at the same time (determined by the quality of the attacker’s command and control (C2) ).  It really is a brute force numbers game for the most part.

Average Bandwidth:

• Home users in US: 1-2 Mbps
• Home users in South America, Africa, South Asia: .5 Mbps
• Home users in South Korea, Japan: 5Mbps
• Virtual Private Server: 100Mbps
• Core Routers: 1000Mbps and up

Number of Nodes: divide the total bandwidth of attack traffic received by the average node bandwidth to determine how many attacking nodes there are.  So, for example, a hacktivist army attacking a site and bringing 2Gbps of attack traffic has around 2,000-4,000 participants.

Recruitment: how fast the attackers (botnet via malware, hacktivists, homebrew botnet, etc) can add nodes to the attack.  This could also be correlated with rates of infection for botnets consisting of home PC users, rates of exploits for servers, number of hacktivists joining in the campaign, etc.

Attrition: how fast the attackers lose nodes.  This could be due to ISPs blocking node access due to detection of attack traffic or bandwidth caps, hacktivists headed off to work during the week, the end of a significant campaign, or the general lack of interest in the attack.

Rate of Growth or Decay of an Attack: total size of attacking nodes plus recruitment minus attrition.

Cute Bot Couple photo by Jenn and Tony Bot

• DDoS Planning: Business Continuity with a Twist
• DDoS and Elections
• Keeping Up With the DDoS Kids
• My Month of Entertainment
• The Rise of the Slow Denial of Service

I’ve noticed a trend over the past 6 months: DDoS traffic associated with elections.  A quick sampling of news will show the following:

• http://www.opendemocracy.net/od-russia/irina-borogan-andrei-soldatov/kremlin-and-hackers-partners-in-crime <-Russia
• http://www.theregister.co.uk/2012/03/26/hong_kong_vote_hack/ <-Hong Kong
• http://www.theregister.co.uk/2011/12/07/korean_election_ddos_row/ <-Korea
• http://www.cbc.ca/news/politics/story/2012/03/27/pol-ndp-voting-disruption-deliberate.html <-Canada (rybolov: yikes!)
• http://www.csoonline.com/article/700523/ddos-attackers-target-russian-election-webcams <-Russia again

Last week it picked up again with the re-inauguration of Vladimir Putin.

• http://en.ria.ru/russia/20120506/173265737.html
• http://www.washingtonpost.com/world/europe/peaceful-protest-turns-violent-in-moscow/2012/05/06/gIQAFti35T_story.html
• http://english.ruvr.ru/2012_05_06/73954450/
• http://www.rferl.org/content/independent_russian_daily_hit_by_dos_attack_kommersant/24571463.html

And then yesterday, Ustream and their awesome response: which, in the Rybolov-paraphrased version read something like: “We shall go on to the end. We shall fight in France, we shall fight on the Interblagosphere, we shall fight with growing confidence and growing strength in our blocking capabilities, we shall defend our videostreams, whatever the cost may be. We shall fight on the routers, we shall fight on the load balancers, we shall fight in the applications and in the databases, we shall fight by building our own Russian subsite; we shall never surrender!!!!1111” (Ref)

Afghanistan Presidential Elections 2004 photo by rybolov.

So why all this political activity?  A couple of reasons that I can point to:

• Elections are a point-in-time.  It’s critical for one day.  Anything that has a short window of time is a good DDoS target.
• DDoS is easy to do.  Especially for the Russians.  Some of them already have big botnets they’re using for other things.
• Other DDoS campaigns.  Chaotic Actors (Anonymous and their offshoots and factions) have demonstrated that DDoS has at a minimum PR value and at the maximum financial and political value.
• Campaign sites are usually put up very quickly.  They don’t have much supporting infrastructure and full/paid/professional staffing.
• Elections are IRL Flash Mobs.  Traffic to a campaign site increases slowly at first then exponentially the closer you get to the day of the election.  This stresses what infrastructure is in place and design ideas that seemed good at the time but that don’t scale with the increased load.

So is this the future of political campaigns?  I definitely think it is.  Just like any other type of web traffic, as soon as somebody figures out how to use the technology for their benefit (information sharing => eCommerce => online banking => political fundraising), a generation later somebody else figures out how to deny that benefit.

How to combat election DDoS:

• Have a plan.  You know that the site is going to get flooded the week of the election.  Prepare accordingly.  *ahem* Expect them.
• Tune applications and do caching at the database, application, webserver, load balancer, content delivery network, etc.
• Throw out the dynamic site.  On election day, people just want to know a handful of things.  Put those on a static version of the site and switch to that.  Even if you have to publish by hand every 30 minutes, it’s better than taking a huge outage.
• Manage the non-web traffic.  SYN and UDP floods have been around for years and years and still work in some cases.  For these attacks, you need lots of bandwidth and something that does blocking: these point to a service provider that offers DDoS protection.

It’s going to be an interesting November.

• Keeping Up With the DDoS Kids
• DDoS Planning: Business Continuity with a Twist
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target

Usually when you think about Denial of Service attacks nowadays, most people think up images of the Anonymous kids running their copy of LOIC in a hivemind or Russian Gangsters building a botnet to run an online protection racket.  Now there is a new-ish type of attack technique floating around which I believe will become more important over the next year or two: the slow http attacks.

Refs:

• Slowloris
• Slow HTTP POST
• RU Dead Yet (RUDY)
• Tor’s Hammer
• XerXes

How Slow DOS Works

Webservers run an interesting version of process management.  When you start an Apache server, it starts a master process that spawns a number of listener processes (or threads) as defined by StartServers (5-10 is a good starting number).  Each listener serves a number of requests, defined by MaxRequestsPerChild (1000 is a good number here), and then dies to be replaced by another process/thread by the master server.  This is done so that if there are any applications that leak memory, they won’t hang.  As more requests are received, more processes/threads are spawned up to the MaxClients setting.  MaxClients is designed to throttle the number of processes so that Apache doesn’t forkbomb and the OS become unmanageable because it’s thrashing to swap.  There are also some rules for weaning off idle processes but those are immaterial to what we’re trying to do today.

Go read my previous post on Apache tuning and stress testing for the background on server pool management.

What happens in a slow DOS is that the attack tools sends an HTTP request that never finishes.  As a result, each listener process never finishes its quota of MaxRequestsPerChild so that it can die.  By sending a small amount of never-complete requests, Apache gladly spawns new processes/threads up to MaxClients at which point it fails to answer requests and the site is DOS’ed.  The higher the rate of listener process turnover, the faster the server stops answering requests.  For a poorly tuned webserver configuration with MaxClients set too high, the server starts thrashing to swap before it hits MaxClients and to top it off, the server is unresponsive even to ssh connections and needs a hard boot.

The beauty of this is that the theoretical minimum number of requests to make a server hang for a well-tuned Apache is equal to MaxClients.  This attack can also take out web boundary devices: reverse proxies, Web Application Firewalls, Load Balancers, Content Switches, and anything else that receives HTTP(S).

Post photo by Salim Virji.

Advantages to Slow DOS Attacks

There are a couple of reasons why slow DOS tools are getting research and development this year and I see them growing in popularity.

• Speed and Simplicity:  Slow DOS attacks are quick to take down a server.  One attacker can take down a website without trying to build a botnet or cooordinate attack times and targets with 3000 college students and young professionals.
• TOR:  With volume-based attacks like the Low Orbit Ion Cannon, it doesn’t make sense to route attack traffic through TOR.  TOR adds latency, throttles the amount of requests that the attacker can send, and might eventually fail before the target’s network does.  Using TOR keeps the defender from tracking you back to your real location.
• Server Logging: Because the request is never completed, most servers don’t make a log.  This makes it very hard to detect or troubleshoot which means it takes longer to mitigate.  I’m interested in exceptions if you know specifics on which webserver/tool combinations make webtraffic logs.
• IDS Evasion: Most DOS tools are volume-based attack.  There are IDS rules to detect these: usually by counting the number of TCP SYN traffic coming from each IP address in a particular span of time and flagging the traffic when a threshold is exceeded.  By using a slow DOS tool that sends requests via SSL, IDS has no idea that you’re sending it slow DOS traffic.
• Stay out of the “Crowbar Hotel”:  Use the Ion Cannon, make logs on the target system, go to jail.  Use slow DOS with TOR and SSL, leave less traces, avoid having friends that will trade you for a pack of cigarettes.

Defenses

This part is fun, and by that I mean “it sucks”.  There are some things that help, but there isn’t a single solution that makes the problem go away.

• Know how to detect it.  This is the hard one.  What you’re looking for is Apache spawned out to MaxClients but not logging a comparative volume of traffic.  IE, the servers are hung up waiting for that one last request to finish and shucking all other requests.
• “ps aux | grep apache2 | grep start | wc -l” is equal to MaxClients +2.
• Your webserver isn’t logging the normal amount of requests.  Use some grep-foo and “wc -l” to compare traffic from: a month ago, a day ago, an hour ago, and the last 5 minutes.
• Disable POST as a method if you don’t need it.  Some of the more advanced techniques rely on the fact that POST can contain more headers and more body data.
• Use an astronomically high number of servers.  If your server processes can timeout and respawn faster than the slow DOS can hang them, you win.  If you had maybe 3000 servers, you wouldn’t have to worry about this.  Don’t have 3000 servers, I might have some you could use.
• Set a lower connection timeout.  Something like 15-30 seconds will keep Apache humming along.
• Limit the request size.  1500 bytes is pretty small, 3K is a pretty good value to set.  Note that this needs testing, it will break some things.
• Block TOR exit nodes before the traffic reaches your webservers (IE, at layer 3/4).  TOR has a list of these.

• Apache Killer Effects on Target
• Stress-Test Apache with Intent to Tune: BSOFH Tip for the Software Masochist
• Keeping Up With the DDoS Kids
• Training the Apache Killers
• A Little Story About a Tool Named #RefRef

My DDoS presentation at DojoCon on Sunday.  A big thanks to Marcus J Carey for organizing the con and Adrian Crenshaw for doing the recording.

• DojoCon 2009 Presentation
• Barcode Hacking
• DDoS and Elections
• Massively Scaled Security Solutions for Massively Scaled IT
• DDoS Planning: Business Continuity with a Twist

So for those of you keeping track at home:

• Indian firm Aiplex announced that they were launching Distribute Denial-of-Service (DDoS) attacks against The Pirate Bay. (the attack)
• The collective wisdom (Anonymous) of the Internet decided that in turn it would DDoS Aiplex, the MPAA, the RIAA, and their international counterparts in Operation Payback. (the counter-attack)
• Somebody has DDoS’ed the sites coordinating the attacks. (the counter-counter-attack)
• I’m popping up some popcorn to wait for the counter-counter-counter-attack and to watch the backscatter.

May we all live in interesting times, to say the least.  Some random thoughts I’m having about the DDoS campaigns:

• If people hate you enough to show up with signs outside your office to protest, they hate you enough to flood your network.
• Activist/vigilante/mob rule/protest has evolved to a very viable DDoS platform using a wide variety of operating systems.
• The DDoS campaign in 2008 against the Church of Scientology was called off by activist leaders, so now we’re seeing the unbridled fury of the Intertubez unleashed.
• On the tools side of things, I’ve seen some good development and some really creative methods to let non-technical folks to participate in the DDoS.
• Coordinating an activist army seems like the weak point in the model.

• Keeping Up With the DDoS Kids
• DDoS and Elections
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target

So let me give you a hypothetical job:

• You have to give up your high-paying private-sector job to be a Government employee
• You have tons of responsibility
• You have no real authority
• You have no dedicated budget
• You have no staffers
• The job has had half a dozen people filling it in the last 7 years
• The job has been open longer than it’s been staffed over the past 7 years

And yet this is what we’re asking candidates to do in order to even be a candidate for the Cybersecurity Coordinator.  Yes, this is the exact same problem that all CISOs have with having a huge helping of responsibility and none of the authority to get things done, only we scaled it up and out to a national-level CISO position.

Somebody’s even gone as far to say that the lack of candidates for the job is the security field’s way of sending the message that you didn’t scope the job right.  I think this opinion has much merit.  CISOs being what they are, they’re usually pretty astute at walking into an ambush, and this job has all the makings of a good one.

I’ll even turn it around the other way and say that the security industry has yet to produce a CISO’s CISO–somebody who can do politics, budget, security, IT, and consensus-building all in one person.  We have lots of people who can manage the enterprise and below, but it’s that additional little bit of political intrigue that is what we’re missing.  Security people usually avoid politics like the bubonic plague because we’re an industry full of people who say it like it really is.  This is a detriment in sales and politics.

So in true Guerilla-CISO fashion of not pointing out problems without offering something as a fix (no matter how much of a strawman arguement it really is), this is what we need to do to get people interested in being the Cybersecurity Czar^wCoordinator:

• A really well-defined scope.  One person cannot do everything that we are asking for at this price (or any price for that matter).
• A budget for an operating staff where the number is more than than 8 digits.
• Statutory authority over the various departments and agencies responsible for cybersecurity: NCSD, S&T, DoJ, FBI, Commerce.  Indirect influence doesn’t work here, never has.
• The direct ear of the President.  Councils are OK, but puhlease, you want to get the job done, this is what it will take.

Then I read back through my list and realized that we really do need a law to create the Cybersecurity Czar position with everything that I just mentioned.  But here’s the rub: legislation is slow, the bills to make the Cybersecurity Czar aren’t even going to be looked at until the next congressional session because we’re still trying to figure out the budget for last year.

I also think that what we’re calling the Cybersecurity Czar is really 2 jobs.  You need somebody working for the Government CIO Vivek Kundra as the executive-branch CISO and you need a more senior person who worries about the military-industrial base, the critical infrastructure, the support to American commerce, and the protection of little old grandmas who represent the end-users.

Tsar’s Cannon photo by Siyad Ma.  Now that’s some teeth for the position.

• Blow-By-Blow on S.773–The Cybersecurity Act of 2009–Part 5
• Blow-By-Blow on S.773–The Cybersecurity Act of 2009–Part 4
• Next Up in Security Legislation: S3474
• Cyber Security coming to a boil
• A Layered Model for Massively-Scaled Security Management