Back from the Dead!
Posted: Sun Feb 12, 2017 3:10 am
Whew, what a week!
I really have been neglecting Milovana quite a bit.
Have you ever had a mission in life? Something you just knew you had to do? Suddenly, it's hard to think about anything else.
Unfortunately, Milovana fell into the "anything else" category for the last couple of years. Nothing to do with the site, I still love you all, I just felt that I was needed elsewhere.
That was until Milovana delivered a rude wake-up call in her own way.
The site was originally set up very nicely - rotating backups were generated every couple of hours and the server ran on multiple-redundant hard drives.
Nothing lasts forever, though. Over the years, the redundant hard drives failed, one by one. The backup mechanism died with an old server that turned itself off last year.
Until finally, Milovana's last hard drive ran into a write error and halted the database server. I noticed the site was down, but was on a deadline at work, so I restarted the server, not really looking into the problem.
A few days later the site was down again and when that happens, it's pretty clear that it isn't just a turn-it-off-and-on-again type of problem.
So I investigated, and found out about the dead drives, the lack of backups and the fact that the last drive was on its last leg as well.
That got my attention. Milovana is very dear to me and the fact that I let it get so close to death shocked me to my core. I was determined to resurrect it.
Assessing the damage, there was a lot of good news: The images are safely stored in the cloud and there was zero damage on that front. All other files were intact as well. But when copying the database, I got some very bad news. The database file was unreadable. Any attempt to copy it resulted in an error.
I was eventually able to recover most of it, using an awesome open-source called ddrescue. This tool is able to read a file block-by-block, sector-by-sector in a way to minimize further damage to the drive. This way, I was able to retrieve all but 615KB of the 4GB database file:
99.98% sounds pretty good until you try to boot the database and it still won't start. No database, no way to recover so much as a single tease.
Well, about 16 hours of work later, I had the data back and here we are.
A few changes
I'd like to invest some time into the site again. But I need your help to figure out on what. Here are my thoughts:
Milovana Forever!
I really have been neglecting Milovana quite a bit.
Have you ever had a mission in life? Something you just knew you had to do? Suddenly, it's hard to think about anything else.
Unfortunately, Milovana fell into the "anything else" category for the last couple of years. Nothing to do with the site, I still love you all, I just felt that I was needed elsewhere.
That was until Milovana delivered a rude wake-up call in her own way.
The site was originally set up very nicely - rotating backups were generated every couple of hours and the server ran on multiple-redundant hard drives.
Nothing lasts forever, though. Over the years, the redundant hard drives failed, one by one. The backup mechanism died with an old server that turned itself off last year.
Until finally, Milovana's last hard drive ran into a write error and halted the database server. I noticed the site was down, but was on a deadline at work, so I restarted the server, not really looking into the problem.
A few days later the site was down again and when that happens, it's pretty clear that it isn't just a turn-it-off-and-on-again type of problem.
So I investigated, and found out about the dead drives, the lack of backups and the fact that the last drive was on its last leg as well.
That got my attention. Milovana is very dear to me and the fact that I let it get so close to death shocked me to my core. I was determined to resurrect it.
Assessing the damage, there was a lot of good news: The images are safely stored in the cloud and there was zero damage on that front. All other files were intact as well. But when copying the database, I got some very bad news. The database file was unreadable. Any attempt to copy it resulted in an error.
I was eventually able to recover most of it, using an awesome open-source called ddrescue. This tool is able to read a file block-by-block, sector-by-sector in a way to minimize further damage to the drive. This way, I was able to retrieve all but 615KB of the 4GB database file:
GNU ddrescue 1.22
Press Ctrl-C to interrupt
Initial status (read from mapfile)
rescued: 4497 MB, tried: 615424 B, bad-sector: 615424 B, bad areas: 134
ipos: 0 B, non-trimmed: 0 B, current rate: 0 B/s
opos: 0 B, non-scraped: 0 B, average rate: 0 B/s
non-tried: 0 B, bad-sector: 615424 B, error rate: 0 B/s
rescued: 4497 MB, bad areas: 134, run time: 0s
pct rescued: 99.98%, read errors: 0, remaining time: n/a
time since last successful read: n/a
Finished
99.98% sounds pretty good until you try to boot the database and it still won't start. No database, no way to recover so much as a single tease.
Well, about 16 hours of work later, I had the data back and here we are.
A few changes
- Upgraded to Ubuntu 16.04, PHP 7 and phpBB 3.2
- Removed the ads (they hadn't actually been hooked up to make money anyway)
I'd like to invest some time into the site again. But I need your help to figure out on what. Here are my thoughts:
- Design a new permanent backup mechanism (we're on a provisional one right now) (non-negotiable)
- Re-enable registrations (using an invite system to limit spam)
- Set up a new chat (anyone got suggestions)
- Finish Mia (HTML5 interactive tease engine)
- ... what else? Tell me your ideas!
Milovana Forever!
