Thursday, May 14, 2009

Dealing with Brownouts

In addition to my work for Major League Media and Loneliest Road Consulting LLC, I work for a K-12 educational organization in southeast Indiana as the assistant technology coordinator.

Day to day operations for this school district are generally tame. Even so, we utilize RAID, a strict backup regimen, UPS devices on all critical systems, and the whole nine yards.

It's amazing how something as trivial as a small Mylar balloon can cause a major electrical nightmare.

Said balloon flew into a transformer owned by our local utility company. The resulting voltage spike and brownout was mostly contained by our array of UPS devices. The switch to our backbone was spared an untimely end, as were the application servers running the school office and gradebook software.

Unfortunately, a total of five Pentium4 workstations- connected only to consumer grade surge protectors, suffered electrical damage to their motherboards. Our Linux firewall suffered a corrupted file system resulting in only repeated kernel panics with each boot attempt.

This event is a lesson in disaster preparedness. Despite your best attempts at protecting your network and attached computer systems, electrical faults can still cause damage. It is important to note:

1. Distributing your key services among several machines is ideal. Had we suffered the loss of a DNS server or a domain controller, we had at least two ancillary machines that would successfully go into failover.

2. Hot spares are essential. Our local hardware vendor did an excellent job in repairing our firewall. It still required time. In the meantime, we were without Internet while the repairs were being made. Having no hot spare for our firewall was an Achilles heel.

3. Cold spares are better than nothing. As for the destroyed workstations, we had some legacy Dell GX-270 Optiplex workstations on hand. We were able to install our key applications within minutes. We were able to utilize them despite the lack of Internet access as they depended only upon resources on our LAN to function.

This brings up another important observation.

4. If possible and practical, keep essential services on your LAN. We were able to operate because we did not remotely host core services. If you require Internet access to get to your data and your ISP or remote host throws a router or has a power outage, you don't want to be crippled as a result. It may not be possible to always avoid out-sourcing the hosting of applications and data, but at the very least you should ensure you are informed of their disaster mitigation and recovery policies.

Fortunately this story has a happy ending. We were prepared for a far worse disaster. Even so, it is good to constantly re-evaluate disaster recovery plans for flaws and potential improvements. If a small mylar balloon hitting a transformer can derail your operation, consider hiring someone to develop a plan for you if you are otherwise unable to do so yourself.

Saturday, May 2, 2009

Let's Talk About Hosts

Hosting is more affordable than ever, but when looking for a host, do not proceed with the assumption that all hosts are created equal. There are a number of factors that should be considered in selecting a host- provided you are not going to use your own hardware to host your site.

Hardware Believe it or not, there are some web hosts who are using Intel Atom processors on their web servers. This is NOT an ideal platform. Between the Atom's optimization for low-power computing, low clock speed, and dismal performance under load, if your website requires consistent and speedy performance, this is not the solution you want to pay money for.

Also determine whether you are going to be receiving shared hosting or dedicated hosting. If your site is being hosted on the same physical machine as 100 other sites, you can be assured of a resultant reduction in the performance of your website. You may save money by using a shared host, but if your web presence demands high performance in order to ensure your operation is profitable, you will lose more money than you will save.

Operating System For websites that are built with mainly open source and cross-platform tools such as MySQL, PHP, Java, and Ruby, Linux-based hosts are a sensible and economical solution.

On the other hand, if your business relies on proprietary web applications that require Microsoft's specific version of SQL, ASP, or simply are available only as Windows binaries, Windows 2003 or Windows 2008 is the only way to go.

Bandwidth If you are streaming media files or otherwise moving large amounts of data outside of your LAN, your host needs to provide you with ample bandwidth. If you are running a business, you do not want to lose clients (and potential revenue) when visitors to your website receive a notice along the lines of: "The bandwidth for www.acme-widgets.com has exceeded its alotted bandwidth. Please try again later."

Quality of service Let's face it. A host can run on the latest hardware, support all of the apps you want, and provide boundless amounts of bandwidth, but if when you call, you are stuck talking to an intern who doesn't know hosting from a hole in the ground, you have paid too much money for your hosting. If you are met with rudeness when you call for support, your host doesn't deserve your money. Most of all, if your hosting provider tries to up-sell you despite your actual needs and over your objections, bail. Hosting providers can make plenty of money without having to mistreat their potential customers.

When choosing a host, its always a good idea to do some research. Speak with colleagues in your field to find out their experiences. Utilize resources such as the Better Business Bureau and RipOffReports.com. Google any potential hosting companies. Keep in mind that even the best hosting companies have their share of customer complaints. Be sure to account for the scope of these complaints and whether they were resolved in addition to merely the number of complaints.