On Monday, September 17th, I got a call from someone asking me if I worked on servers, and what my rate was. I gave them that information, and they asked how soon I could get there to fix theirs. When I got there, I encountered a desktop server (HP Gen 6, runnning Windows Small Business Server (SBS) 2008. There was only a power cord and an Ethernet cable connected to the machine, which was located on the very top of a tall shelf. Remoting in didn’t work, so I setup a monitor, mouse, and keyboard, and got to work. The server would boot to “Applying Computer Settings”, but no further. It never gave me a login prompt, even in SafeMode, etc. I tried various methods to try to get into a UI on the server, but was unsuccessful with all of them. The Client assured me that they had very good, very complete backups, etc, so I decided to wipe and reinstall the OS. This machine had been almost entirely untouched since it was set up in 2010, so a fresh install seemed prudent anyway. I did not have install media with me, and their internet connection was slow enough that it was estimating about 6 hours to download the install iso, so I called it a night, and went back to my shop and downloaded and burned an install disk.

I went back the next day, and after a few trivial hiccups, got SBS reinstalled, and we were ready to rock. Well, almost. I tried to go to Carbonite.com to download and install the software, but I couldn’t load ANY webpages from Internet Explorer, due to “Security issues”. I looked into all of the common causes, time, date, time zone, etc, and then just decided to download Firefox. The installer downloaded fairly quickly, I opened it, and…”This version of Windows is to old for this version Firefox”. I doubt that is the exact message, but that’s what it meant. So I went to Mozilla’s archives, and downloaded Firefox version 43. Still no go. So I went back to version 40. No good. I did some research, which said that anything before version 32 should work. Nope. I finally got version 30 to install. Launched the browser, navigated to Carbonite, and clicked login. username box, password box, and… what’s this? “This browser does not support Recaptchas”. Well, bollocks. Fortunately, this one was fairly easy to work around, I just needed to manually tell Firefox to install the latest updates, which fortunately got me to a new enough version to load the Recaptcha. Onward, HO! I logged into their Carbonite account, and got all set to start a restore. However, for some reason, I could not find a “restore” button anywhere in the Carbonite window. I knew it should be there, somewhere. So I called Carbonite, and got their tech remote access, and he looked, and confirmed that it should be there, but it wasn’t. He tried a few things, adjusting display settings, etc. When nothing worked, he worked some behind the scenes magic to get the restore started manually. It estimated 14 hours for the restore, and thus ended day two of the debacle.

The following day I had another job that I needed to take care of in the morning, so I wasn’t able to get there until about 5:30 PM. I had apprised the client of the status of things, and they called me around 10 AM and stated that the server seemed to be frozen. When I got there, I checked things out, and it seemed that Carbonite had finished restoring everything, and we were ready to move onto the next step. The next step was to get the software that the client needed on the server installed. As this is a commercial software suite, which included support, I again made a phone call, got the support tech remote access, and let him do his thing. Things turned sharply for the worse, almost immediately. I told him what had happened, and what we had done. He poked around for about a minute, and asked “where are the files?”. Uh-oh. “What files?” “There should be [these_files] in [this_location].” So I got off with him, and dug around a bit myself, both on the server, and on the online Carbonite backup. The necessary files were nowhere to be found. SO I called Carbonite again, and explained my predicament to them. “Oh, that location isn’t backed up by default, that would have had to be manually setup to back up. I do see that the customer is paying for the option to back that up, but it was never setup.” Crap. The files in question were a database for all of my customers customers over the previous 8 years, as well as ALL of their scheduling, billing, and customer information. They had been blind since the server went down, and now it didn’t look like that was going to change very soon. I’ll skip the painful details of having that particular conversation with the client, but if you work in IT, you can imagine. There wasn’t much to be done from there, other than to start over. Accordingly I got the software support tech back in, and had him start the installation process. (At that point I had not totally given up on recovering the database, but either way, we would need the software installed, so I decided we may as well pursue that in the interim.) The software rep said that the installation would take him about 2 hours, and it was 8 PM by then, so I took off for the night again, planning to be back by 10 the next day.

The next morning, I got a call at about 930 saying that the software rep had not been able to finish the night before, and would need to get back on today. However, he was located in California, and didn’t get in until 10 AM EST (our time), so there was point in me coming until later. I waited impatiently for the call to come back and keep working, but it never came. Instead, around 5PM I got a call letting me know that due to the age of the server/OS, the software rep was running into problem after problem, and was warning them that the system might not end up being stable no matter what. With this recommendation, they decided that they were going to buy a new server, and they would be in touch with me to let me know when they wanted me to come set it up for them.

Thus ended the debacle of server number one. A few things to address before I close. IT guys, NEVER trust someone else’s backup. CLONE THE DRIVE. Yes, it takes time. precious time, when the server is down, and the customer wants it up RIGHT NOW. Go buy an extra drive to swap in in the meantime, if they need to get up immediately. You can always schedule a time overnight or something to swap back to the original drive if you need to. The second part is that I am sure some of you are thinking “Data recovery service” for that database and other files. We are still pursuing that option. (Shoutout to Ontrack Data recovery). However, the drive in question was an older Samsung SSD. Hopefully Ontrack can recover the stuff we need, but SSDs are notoriously pesky for that. (The drive was formatted during the installation.)