A tech savvy friend can be a great asset but they can also be a serious liability to your business. An example is a local medical practice I recently visited; the backup system was in place and working as advertised – or so they thought.
The doctors got a great deal; the office manager had a friend who was a computer science major at FSU. They figured it’d be a great way to save some money while giving the friend an opportunity to hone their computer skills. The friend installed a travan 20G/40G internal tape drive, downloaded and installed software to manage the nightly backup, configured it and went off – back to school.
Each morning the office manager came into the office to find the backup tape popped out of the server – indicating that the backup had completed. He would remove the tape and place it on the stack; grabbing the one for the next day and pushing it into the drive.
While the friend was off at college time marched on. The medical practice upgraded the Electronic Health Records (EHR) software and then upgraded it again. Business continued as usual. More and more of their patient data was entered into the database; more images of lab results and more patient scheduling. All the while the tapes kept rotating in and out of the server.
The office manager’s friend came back into town on college breaks but he was too busy and had forgotten about the doctor’s office. He had installed the system and was paid for his work. Done.
One day I show up. I talk to the office manager. They have a tape backup solution in place but he doesn’t know much about it – a buddy of his installed it for them. They put tapes in and take tapes out like they’re supposed to. “May I take a look?”.
Close Enough Only Counts With Horseshoes And Hand Grenades
This turned out to be the classic “Feel Good” backup. What I mean is the office manager had done the responsible thing; they had a backup plan in place. The “Feel Good” backup is what you have when you’re comfortable that you’ve done just enough – without worrying too much. Feels good not to worry, right?
I discovered the last backup which had completed successfully was in December of the previous year. Also, they did not set that particular tape aside so it had been overwritten by subsequent backups which had all failed. Sure, the log file shows that the backup completed successfully in December – but it’s now March and the tape has been written over many times. Effectively no backup. Their disaster recovery plan put simply; don’t have a disaster.
The plain and simple truth of it is this: data backup is an interactive process which requires human attention. The backup “job” often needs oversight and modification (what I like to call “care and feeding”) to ensure reliability. If you’re delegating that responsibility to someone else be damned sure they are reliable. Make sure they have some “skin in the game” themselves.
Ronald Reagan: IT Genius
“Trust but Verify” was Ronald Reagan’s response once, when asked if he trusted the cold war Soviet empire to dismantle their nuclear arsenal. Ronald should have worked in information technology because those are the words we live or die by.
Verification is critical to backup because a tape backup is just a stream of information laid down on magnetic tape. If the tape is worn or old, missing information is not uncommon. Tape is fast and simple but you can see where writing and overwriting again and again could wear out the magnetic properties of the tape – without running the tape back across the heads and confirming what was written by comparing it to what should have been written backup is virtually useless. I mean really, what value is a backup if you’re uncertain about it?
The medical practice wasn’t verifying their backup! The backup job was completing but because the drive/tape solution chosen was inexpensive and slow the installer opted to skip over verification. Really?? The failure was even worse than imagined: not only was the backup failing but even if it were succeeding the tapes themselves were not assured to actually contain a full or even partial backup. Amazing!
Being oblivious to the obvious.
I had noticed it right away because I’d seen it so many times before. The office staff had a hierarchy of delegation and the tape would be swapped out by the office manager unless they were on vacation or out sick in which case one of the other staff would be responsible for making sure it got done. The tapes were all arrayed right beside the server; one for Monday, two for Tuesday, three for Wednesday, etc … with a cleaning tape there in case it was needed. I asked if they kept any of the tapes off-site – they did not.
Data backup is great mitigation against software corruption, virus infections, catastrophic hardware failure and accidental deletion. A verified backup will help with any of these things but if you’re keeping your tapes on-site you’re making a huge mistake. In the case of this doctor’s office they were doing it because they were concerned about taking personally identifiable patient information (PII) off-site which for HIPAA reasons is a valid concern. However, if the data were encrypted securely that concern evaporates, tapes can leave the building for storage off-site and fire is no longer a threat. If you have a backup and it’s not off-site you should consider what you need to do right now to get it out of the building.
Disasters actually do happen. For this reason and in the hope of being useful; Penguin Computer & Telephone Solutions will provide for your business or organization a no-obligation consultation. Let us evaluate your current disaster recovery plan and suggest ways to improve your chances of surviving a challenging recovery before it’s too late.