On the eve of 13th March, 2012. Team DEP finished Each and every article in the Que, there was almost nothing pending their attention. Over 700 articles were processed during that night and everyone was happy for their share of hard work. But they never knew what fate had in store for them. Later that night, suddenly hard disks of our main server crashed followed by another crash of the backup server.
Inspedium Hosting , which was our primary host for most of our websites went completely under. None of the unfortunate websites hosted with them was alive. all dead. All the data was lost. All they could come back to us was with a backup taken by them dated back in september 2011.
Unfortunately, we took our last backup on 9th March… we take weekly backups & suddenly that was not enough This was our lesson No 1.
After quickly restoring DEP on another of our dedicated servers now hosted with HostGator. we painfully waited for DNS to propogate and that takes around 72 hours. Meanwhile , we tried to mine our office hard drives , emails and notepads to make sure that all financial transactions happened during that period can be restored.
Good News is that all of the articles along with their financial information was found and quickly restored. Bad news is that everything else.. dates, approvals, rejection reasons etc. was lost for the 3 days.
We have thus decided to move on. the team is back at work. trying to recover from the situation. Your help and patience is what we are counting on. Check your dashboards. if you see something way out of common sense. please email us urgently. we will try to rectify it as soon as possible.
One more thing, we have closed down our Promotion Module. we are working on its replacement. We will notify you as soon as something worth communicating comes up.
I just finished restoring this blog from Google’s Cache(we did not backup our own blog = Lesson 2). Comments before december 2011 are lost however.
I hope you will appreciate our openness and forgive our unforgivable sin of all.. loosing data. we are sorry about that and will try to make it up to you with some really good news soon.
Last lesson for me personally is that there is no such thing as too many backups. We should have a backup of everything. then backup of backups & then their backup… We should have a hosting company of our own too. A really good and reliable one….
