Milovana.com

Dear Milovanians,

Today (and some report yesterday as well) the site has been running into issues that caused a lot of people not to be able to use Milovana.

At this point everything seems to indicate that we were hit by a Denial of Service attack. We deactivated the front page and that seems to have solved the problem for the moment. Obviously this is not a permanent solution, but we're working very hard to secure the site against such attacks in the future.

With a limited budget like ours, we can't afford the kind of redundancy and excess capacity that would thwart such an attack. For now, we'll keep fighting with the means available to us, but it seems we're at a point where we will need to raise some money or we won't be able to sustain the site. The increase in user numbers brings with it a higher profile and that means more work for the admins and more risk that there's somebody out there who doesn't have our best interest at heart.

There might be more downtime in the next few days if the attackers adopt to our countermeasures, but we'll definitely keep fighting.

Edit: The exploited vulnerability has been fixed, more on that in this post. This means we should be safe now!

Still going strong,

Seraph0x

Denial of Service? Topical! I dont even know what that is...

Is there a donate button? I would hapilly some of my laughable earnings to keep this site going... not much because im poor... but its the thought that counts! I feel like I use this site enough to warrant me giving something back

What happened to the idea of putting a few ads on the site?

cumhardy wrote:What happened to the idea of putting a few ads on the site?

I temporarily put a banner next to the "Got Stamina" video as a test. The idea is very much alive, although I lack the time to really put it into action. The pressure to get my ass up is definitely rising though, so you can expect us to start trying to raise money through some means soon. (Whether it be donations, ads or whatever.)

Good job my friend. Nice idea to disable the front page

And cumhardy "Denial of Service" means that you send so much requests to a server until the server denies it's service. You can also name it flooding.

did the DOS come from a single IP or a multitude of IPs or even just one range?

If its one range or a single IP you should be able to block that IP, of course if its more than one IP or range then theres problems!

I haven't had much time to contribute to teases and the like, so let me contribute here with my knowledge on the subject. For the last two years I've run a site of my own that has been the subject of constant attacks, both denial of service attacks and attacks intended to compromise the server. So, here are a few tips you might want to think about. I know several people come here who also run their own sites, and this might apply to them, so I'm making most of what I have to say on this public. If you'd like to talk further Seraphox, feel free to PM me.


IPtables tricks
There are a few things you can do with iptables that help with this sort of thing a great deal. Namely, you can limit connections based on network blocks.
The above rules ensure that any given IP address cannot have more than 4 open connections to port 80 at any given time, and that a any given class C subnet will only be allowed to have 16 open connections at a time. The last rule allows in new traffic not blocked by those rules, and tells the connection tracking feature in the kernel to handle them (performance increase). The log rules there will only trigger a max of 4 times per minute, which lets you log that an attack has happened, but without filling up all of your logs. You can then later parse the logs for abusive IP's and block them specifically.

The above tactic would also work for just about any port, but you'll have to play with the limits if it's something intensive like mail or dns.

It can help sometimes to use a stateful firewall, which you can do about like this: And then the rest of your rules go after that, but you change any of your rules which would just be -j ACCEPT to -m state --state NEW -j ACCEPT. The effect this has is that a new connection only traverses your full set of rules one time. If the first packet meets all the rules to be allowed through then the firewall keeps track of that so it doesn't have to fully inspect the rest of the packets. This leads to much higher performance and mitigates a minor DoS condition from someone just trying to make a bunch of connections at once.

Kernel Tricks
These commands turn on some security features in the kernel which are disabled by default. There isn't any noticeable performance hit from turning them on (in my experience), and they each protect against some common network-based attacks. Apache tricks
There are a few apache modules you might want to check out, like mod_evasive, which analyzes HTTP requests in real time and will automatically block and/or throttle offensive clients for configurable periods of time. I've rarely run into any trouble with it blocking legitimate visitors, and it helps a lot against brute force attacks and general DoS attacks against Apache. Just be aware that it does sometimes mistake download managers as attacks. Installing the suhosin patch for PHP probably isn't a bad idea either.

If you can, I highly recommend running Apache inside of a chroot and enforcing per-process resource limits (which you can do with grsecurity/pax and some other methods).


Anyway, I hope this helps out.

-ps: OpenBSD's pf firewall stands up to this kind of crap WAY better than Linux.

omssx wrote:did the DOS come from a single IP or a multitude of IPs or even just one range?

It was a distributed attack. It's very hard to distinguish between legit clients and clients that are part of the attack. But since I turned off the front page I've logged about 1600 IP addresses who were accessing the front page and not any other page (i.e. whatever client they are using doesn't understand HTTP redirection, which means it's definitely not a browser). Each client seems to be doing one request per fifteen minutes approximately.

The thing is - the attack alone shouldn't have been enough to take down the site. Those were a lot of requests, but we have about 1.2 million requests per day anyway. But somehow they managed to make the front page script run extremely slow. And that's what I'm trying to investigate at the moment, how that was possible. (Could also be another reason why disabling the front page thwarted the attack - if the exploit is specific to that page.)

enoch wrote:I haven't had much time to contribute to teases and the like, so let me contribute here with my knowledge on the subject.

Great stuff! Some of this we're doing already, for instance suhosin is installed and I think most of the kernel options are set the way you recommend. It's hard to say whether any of it would have helped, since it really was a fairly well orchestrated attack. But I'll look into the changes you suggested. There is always room for improvement. Thanks!

enoch wrote:-ps: OpenBSD's pf firewall stands up to this kind of crap WAY better than Linux.

I've been wanting to try *BSD for a long time now. Never got around to it. Any tips how to get started? Do you recommend OpenBSD for hosting? (currently we're using Debian)

OpenBSD is kind of a double-edged sword. It's rock solid on the outside looking in, as there are very few public exploits for it. Private exploits are another story, especially local exploits. It's very good as a firewall/IDS/backup system though. It's packet filtering system is a lot more robust than iptables, and is less static. The main two differences are it's support for "tables" and in it's actions.

You can create a table of IP addresses or networks and set it up so that the table can be dynamically changed without flushing your rules or state tables, and can be dumped to and loaded from a file very easily. You can also set up rules which will add IP's to a given table automatically based on certain conditions, which makes it ideal for handling things like DoS attacks.

As far as making the front page run slowly I can only guess. If I could see the code myself, especially along with a log of what data was actually sent to it, I might be able to figure out what they did. Have you considered setting up tcpdump or something of that nature to log all of the data going to port 80? You might find more information that way. It might even be some sort of memory consumption exploit in Apache or something like that.

Ok, front page is back and it's holding up to the flooding.

There was indeed a bug in there that allowed somebody to artificially cause the script to open three database connections rather than just one. Obviously this is plenty of leverage for a DoS.

I now added some memcaching and other optimizations that also make the front page load a lot faster. (From ~166k operations to ~37k; a 349% performance increase)

Some of these optimizations are site-wide and should make everything a bit faster.

Bottom line is - unless they are able to significantly ramp up their attack we should be pretty safe now.

I'll still look into additional safety measures such as the ones enoch and others have suggested. Thanks to all those sending PMs or posting here. It was amazing to have that kind of support and to me, you guys are the real heros of this incident!

Hey, thanks for working to bring the free-stuff in the face of the meenies, Seraph0x. You're a superstar.

Fucking meenies.

In addition, I for one am more than willing to look at banner ads if it keeps the site viable.

camipco wrote:I for one am more than willing to look at banner ads if it keeps the site viable.

Especially if it's sexy banners.