How good are your backups? If you were like me, I would have said outstanding last week. I mean, our local computer systems have system state backups. The all-important data is backed up locally and remotely nightly. Our servers get either a weekly or daily incremental (depending on its functionality) continuously. Our hosted web pages and their underlying databases have nightly backups as well. Everything was running so smoothly that a wrinkle, caused by me, went unnoticed for months. Then disaster struck. Not money making, or more importantly, money losing, but still a disaster. Our sites run off four databases clusters. This site was moved to a new cluster in May. That is the wrinkle. New database, need to add it to the backup sets. As I look back at the checklist (self created), that item is on the checklist, right after creating and doing initial tests on it.

That little innocuous box is not checked. Everything else is, including the master checks saying all the little checks have been doubled checked. This was the recipe for disaster. Months of not noticing started the baking process. Then the catalyst was added. I upgraded our account. This type of upgrade causes all your servers to change, web and databases, and the host states repeatedly that your data is your responsibility. Their process attempts to transfer your data but it is far from being guaranteed. Therefore, I backup up my log files and committed the change. I felt safe, secure, and confident in that I would not only not need to restore anything, I felt that all the fuss was unnecessary.

After the change, everything was fast responsive, and looked like a million bucks. The next day, I checked my personal site, this one, only to be met by “site offline.” This was confusing at first. Then I recalled that site offline was the error message given for a wide variety of things that I wanted to insulate any casual (crackers specifically) visitor from seeing. Like the name of the database server if connection fails. I logged into the admin portion and there it was, big red X on database connectivity. I instantly realized that the name had changed and so I changed it, fully expecting to see all green. Database still showed red.

After some digging, I saw that although the server, database, username, password, and table names were all correct, the tables just did not exist. After a minor amount of troubleshooting, I said to heck with it, and proceeded to restore from backup. That is when I realized the mess I was in. I had neither database nor a backup. I stared in disbelief that somehow, that I, master of my domain, of my data, pastor of the Big Backup Book, had succumb to a not a failure of the backup to work, but a failure of my brain to realize that I had failed, so many months before.

Therefore, here now, is the first article added since restoration. A great deal of articles (12, but still a great deal!) is lost. Along with accounts, photos, etc. As the machine the articles were written on and its backup has been the victim of a format and reinstallation. Rest assured however, that this database is now being backed up, probably as you are reading this!