Downtime Through Hacker Attack!

You can find important news and current events here.
Locked
seraph0x
Administrator
Administrator
Posts: 2654
Joined: Sun Jul 23, 2006 8:58 am

Downtime Through Hacker Attack!

Post by seraph0x »

Dear Milovanians,

Today (and some report yesterday as well) the site has been running into issues that caused a lot of people not to be able to use Milovana.

At this point everything seems to indicate that we were hit by a Denial of Service attack. We deactivated the front page and that seems to have solved the problem for the moment. Obviously this is not a permanent solution, but we're working very hard to secure the site against such attacks in the future.

With a limited budget like ours, we can't afford the kind of redundancy and excess capacity that would thwart such an attack. For now, we'll keep fighting with the means available to us, but it seems we're at a point where we will need to raise some money or we won't be able to sustain the site. The increase in user numbers brings with it a higher profile and that means more work for the admins and more risk that there's somebody out there who doesn't have our best interest at heart. :closedeyes:

There might be more downtime in the next few days if the attackers adopt to our countermeasures, but we'll definitely keep fighting. :boxing:

Edit: The exploited vulnerability has been fixed, more on that in this post. This means we should be safe now!

Still going strong,

Seraph0x :-)
User avatar
cumhardy
Experimentor
Experimentor
Posts: 1139
Joined: Wed May 28, 2008 10:54 pm
Gender: Male
I am a: None of the above
Location: UK

Re: Downtime Through Hacker Attack!

Post by cumhardy »

Denial of Service? Topical! I dont even know what that is...

Is there a donate button? I would hapilly some of my laughable earnings to keep this site going... not much because im poor... but its the thought that counts! I feel like I use this site enough to warrant me giving something back

What happened to the idea of putting a few ads on the site?
seraph0x
Administrator
Administrator
Posts: 2654
Joined: Sun Jul 23, 2006 8:58 am

Re: Downtime Through Hacker Attack!

Post by seraph0x »

cumhardy wrote:What happened to the idea of putting a few ads on the site?
I temporarily put a banner next to the "Got Stamina" video as a test. The idea is very much alive, although I lack the time to really put it into action. The pressure to get my ass up is definitely rising though, so you can expect us to start trying to raise money through some means soon. (Whether it be donations, ads or whatever.)
User avatar
dark
Co-Admin
Co-Admin
Posts: 1355
Joined: Mon Aug 14, 2006 3:53 pm
Location: Germany

Re: Downtime Through Hacker Attack!

Post by dark »

Good job my friend. Nice idea to disable the front page :smile:

And cumhardy "Denial of Service" means that you send so much requests to a server until the server denies it's service. You can also name it flooding.
Try or die
omssx
Explorer At Heart
Explorer At Heart
Posts: 177
Joined: Sat Dec 02, 2006 9:08 pm

Re: Downtime Through Hacker Attack!

Post by omssx »

did the DOS come from a single IP or a multitude of IPs or even just one range?

If its one range or a single IP you should be able to block that IP, of course if its more than one IP or range then theres problems!
enoch
Explorer
Explorer
Posts: 46
Joined: Wed Jan 24, 2007 8:31 am
Gender: Male
Sexual Orientation: Straight
I am a: Slave
Dom/me(s): Goddess Maya Loux
Contact:

Re: Downtime Through Hacker Attack!

Post by enoch »

I haven't had much time to contribute to teases and the like, so let me contribute here with my knowledge on the subject. For the last two years I've run a site of my own that has been the subject of constant attacks, both denial of service attacks and attacks intended to compromise the server. So, here are a few tips you might want to think about. I know several people come here who also run their own sites, and this might apply to them, so I'm making most of what I have to say on this public. If you'd like to talk further Seraphox, feel free to PM me.


IPtables tricks
There are a few things you can do with iptables that help with this sort of thing a great deal. Namely, you can limit connections based on network blocks.

Code: Select all

iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 4 -m limit --limit 4/minute -j LOG --log-prefix "HTTP-DOS: "
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 4 -j DROP
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 16 --connlimit-mask 24 -m limit --limit 4/minute -j LOG --log-prefix "HTTP-DOS: "
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 16 --connlimit-mask 24 -j DROP
iptables -A INPUT -p tcp --sport 1024:65535 -m multiport --destination-ports 80,443 -m state --state NEW -j ACCEPT
The above rules ensure that any given IP address cannot have more than 4 open connections to port 80 at any given time, and that a any given class C subnet will only be allowed to have 16 open connections at a time. The last rule allows in new traffic not blocked by those rules, and tells the connection tracking feature in the kernel to handle them (performance increase). The log rules there will only trigger a max of 4 times per minute, which lets you log that an attack has happened, but without filling up all of your logs. You can then later parse the logs for abusive IP's and block them specifically.

The above tactic would also work for just about any port, but you'll have to play with the limits if it's something intensive like mail or dns.

It can help sometimes to use a stateful firewall, which you can do about like this:

Code: Select all

iptables -F
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
#===========================
iptables -A INPUT -i lo -s 127.0.0.0/8 -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED, RELATED -j ACCEPT
#===========================
iptables -A OUTPUT -o lo -d 127.0.0.0/8 -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
And then the rest of your rules go after that, but you change any of your rules which would just be -j ACCEPT to -m state --state NEW -j ACCEPT. The effect this has is that a new connection only traverses your full set of rules one time. If the first packet meets all the rules to be allowed through then the firewall keeps track of that so it doesn't have to fully inspect the rest of the packets. This leads to much higher performance and mitigates a minor DoS condition from someone just trying to make a bunch of connections at once.

Kernel Tricks
These commands turn on some security features in the kernel which are disabled by default. There isn't any noticeable performance hit from turning them on (in my experience), and they each protect against some common network-based attacks.

Code: Select all

#enable broadcast echo protection
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

#disable source routed packets
for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do
    echo 0 > $f
done

#enable syn cookie protection
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

#disable icmp redirects
for f in /proc/sys/net/ipv4/conf/*/accept_redirects; do
    echo 0 > $f
done

#disable icmp redirect messages
for f in /proc/sys/net/ipv4/conf/*/send_redirects; do
    echo 0 > $f
done

#drop packets which would have a reply sent on a different interface
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
    echo 1 > $f
done

#log packets with impossible addresses
for f in /proc/sys/net/ipv4/conf/*/log_martians; do
    echo 1 > $f
done
Apache tricks
There are a few apache modules you might want to check out, like mod_evasive, which analyzes HTTP requests in real time and will automatically block and/or throttle offensive clients for configurable periods of time. I've rarely run into any trouble with it blocking legitimate visitors, and it helps a lot against brute force attacks and general DoS attacks against Apache. Just be aware that it does sometimes mistake download managers as attacks. Installing the suhosin patch for PHP probably isn't a bad idea either.

If you can, I highly recommend running Apache inside of a chroot and enforcing per-process resource limits (which you can do with grsecurity/pax and some other methods).


Anyway, I hope this helps out.

-ps: OpenBSD's pf firewall stands up to this kind of crap WAY better than Linux.
seraph0x
Administrator
Administrator
Posts: 2654
Joined: Sun Jul 23, 2006 8:58 am

Re: Downtime Through Hacker Attack!

Post by seraph0x »

omssx wrote:did the DOS come from a single IP or a multitude of IPs or even just one range?
It was a distributed attack. It's very hard to distinguish between legit clients and clients that are part of the attack. But since I turned off the front page I've logged about 1600 IP addresses who were accessing the front page and not any other page (i.e. whatever client they are using doesn't understand HTTP redirection, which means it's definitely not a browser). Each client seems to be doing one request per fifteen minutes approximately.

The thing is - the attack alone shouldn't have been enough to take down the site. Those were a lot of requests, but we have about 1.2 million requests per day anyway. But somehow they managed to make the front page script run extremely slow. And that's what I'm trying to investigate at the moment, how that was possible. (Could also be another reason why disabling the front page thwarted the attack - if the exploit is specific to that page.)
enoch wrote:I haven't had much time to contribute to teases and the like, so let me contribute here with my knowledge on the subject.
Great stuff! Some of this we're doing already, for instance suhosin is installed and I think most of the kernel options are set the way you recommend. It's hard to say whether any of it would have helped, since it really was a fairly well orchestrated attack. But I'll look into the changes you suggested. There is always room for improvement. Thanks!
enoch wrote:-ps: OpenBSD's pf firewall stands up to this kind of crap WAY better than Linux.
I've been wanting to try *BSD for a long time now. Never got around to it. Any tips how to get started? Do you recommend OpenBSD for hosting? (currently we're using Debian)
enoch
Explorer
Explorer
Posts: 46
Joined: Wed Jan 24, 2007 8:31 am
Gender: Male
Sexual Orientation: Straight
I am a: Slave
Dom/me(s): Goddess Maya Loux
Contact:

Re: Downtime Through Hacker Attack!

Post by enoch »

OpenBSD is kind of a double-edged sword. It's rock solid on the outside looking in, as there are very few public exploits for it. Private exploits are another story, especially local exploits. It's very good as a firewall/IDS/backup system though. It's packet filtering system is a lot more robust than iptables, and is less static. The main two differences are it's support for "tables" and in it's actions.

You can create a table of IP addresses or networks and set it up so that the table can be dynamically changed without flushing your rules or state tables, and can be dumped to and loaded from a file very easily. You can also set up rules which will add IP's to a given table automatically based on certain conditions, which makes it ideal for handling things like DoS attacks.

As far as making the front page run slowly I can only guess. If I could see the code myself, especially along with a log of what data was actually sent to it, I might be able to figure out what they did. Have you considered setting up tcpdump or something of that nature to log all of the data going to port 80? You might find more information that way. It might even be some sort of memory consumption exploit in Apache or something like that.
seraph0x
Administrator
Administrator
Posts: 2654
Joined: Sun Jul 23, 2006 8:58 am

Re: Downtime Through Hacker Attack!

Post by seraph0x »

Ok, front page is back and it's holding up to the flooding. :-)

There was indeed a bug in there that allowed somebody to artificially cause the script to open three database connections rather than just one. Obviously this is plenty of leverage for a DoS. :unsure:

I now added some memcaching and other optimizations that also make the front page load a lot faster. (From ~166k operations to ~37k; a 349% performance increase)

Some of these optimizations are site-wide and should make everything a bit faster. :icecream:

Bottom line is - unless they are able to significantly ramp up their attack we should be pretty safe now. :smile:

I'll still look into additional safety measures such as the ones enoch and others have suggested. Thanks to all those sending PMs or posting here. It was amazing to have that kind of support and to me, you guys are the real heros of this incident! :wave:
User avatar
camipco
Explorer At Heart
Explorer At Heart
Posts: 326
Joined: Sun Aug 13, 2006 6:17 pm

Re: Downtime Through Hacker Attack!

Post by camipco »

Hey, thanks for working to bring the free-stuff in the face of the meenies, Seraph0x. You're a superstar.

Fucking meenies.

In addition, I for one am more than willing to look at banner ads if it keeps the site viable.
Foxhawke
Explorer At Heart
Explorer At Heart
Posts: 457
Joined: Fri Aug 10, 2007 8:59 pm

Re: Downtime Through Hacker Attack!

Post by Foxhawke »

camipco wrote:I for one am more than willing to look at banner ads if it keeps the site viable.
Especially if it's sexy banners. Image :whistle:
Locked

Who is online

Users browsing this forum: No registered users and 33 guests