Software is the most transient component of any data center. This is both a blessing and a curse: when it operates...
incorrectly, it can be easily fixed (more easily and generally more quickly than a hardware component), but it also requires frequent changes.
Over time, it even requires a complete replacement by a different version that may appear and operate differently than the previous version, impacting user comfort as well as your organization's processes. Imagine what life would be like if, when you took your car to the shop, you got it back with the steering wheel moved to the other side and they said that's just how the new version of your car worked!
While this flexibility allows for constant innovation, it sometimes comes at a price even higher than simple change. Software changes put you at risk of instability and failure because of dependencies other parts of a system or the user community may have on that software. Rarely can a system administrator upgrade to the "latest and greatest" version of an application or operating system without impacting someone.
The question: how much impact can you tolerate and is it worth the new functionality you'll gain? Here are some questions to ask yourself whenever you are considering an upgrade or replacement to any production software.
1. Who needs this upgrade?
If your user community is local, this is easily definable. But if your user community is organizationally or geographically dispersed, this can be complicated. Do you support students all over the campus? Do you have sales or support teams in other cities that depend on software or data from your site? Do customers have access to your data (i.e., via a web server that would be impacted by an internal software upgrade). An upgrade to a database might change, for instance, the output of a report that a web site provides for a salesman in London who pulls it into a spreadsheet every month.
2. Why is the upgrade required?
How important is this upgrade? Many are mandatory for one reason or another, but make sure you understand why. You may want to look for another solution if the pain of even a "mandatory" upgrade becomes high enough.
New hardware often requires operating system upgrades. If old application software is not supported by the new version of the operating system, you can have a cascade effect requiring you to also upgrade the application. In the case that there is no upgrade path for the application, a choice must be made between the new functionality requiring the OS upgrade and the old functionality of the application software. As another resort, you could separate the functions onto different servers.
An operating system or application software vendor may require you to upgrade in order to maintain your support contract. Vendors cannot support old software "forever," so they will drop support after several versions.
Security-related upgrades or patches are becoming a normal part of systems management as more and more vulnerabilities are identified in both applications and operating systems. Of course, these are often mandatory and may change the operation of a system.
3. What will the upgrade cost (in terms of both time and money)?
Sometimes upgrades come as part of your support contract. You might also purchase a new version of an application. Either way, it is not the only cost. People will also spend their time installing and adjusting to the changes that result from almost any software upgrade.
This cost can be paid in small, regular increments by upgrading to each new release, or in larger lump sums less often by only upgrading to major releases. The pain of small upgrades may be smaller, but will occur more frequently. Some sites prefer infrequent but significant changes followed by complete stability for a while. Your site may be more able to endure minor hiccups on a regular basis in order to avoid a yearly major upheaval.
4. What people or components depend on the current version?
You must understand what other components of your system(s) you risk affecting and who will be impacted. If you plan an upgrade that is "backward compatible" (i.e., supports everything that came before) and only adds new functionality, then you're probably safe. Be especially careful of any "unsupported" use of an application or operating system component your site might depend on. Old, unsupported features often stay around for a long time when they aren't hurting anything. But they can disappear at a moment's notice when someone cleans up code or they cause a problem with new code.
Assess the impact of a failure of anything that depends on the parts you plan to upgrade. Know whether or not such a failure is a "deal breaker" and prepare an alternative plan if it is.
Upgrades that affect outside users (field personnel or customers) are especially important because these people will notice any problem almost immediately. If at all possible, test an upgrade on an internal version of the software before performing the upgrade on an externally visible system.
Keep in mind that upgrades sometimes require re-customization of an application. If settings and local modifications cannot be easily reproduced in a new or upgraded version of a component, there will be additional effort required to get everything set up like it was before. This may not be your pain, though. It may be up to the user, but it is rarely a show-stopper. Still, it is part of the total impact of the upgrade.
5. What problems arise if the new version doesn't work?
If the upgrade causes some type of failure, how will you find out (other than angry phone calls)? Go through your list of people and components having dependencies and try to list how a failure would manifest itself to them. This will give you an idea of what to look for after the upgrade and how to decide when to declare victory.
This also provides you with a way to choose the proper reaction to a problem. If the failures are significant inconveniences but people can work around the problems, then you will have time to investigate and determine the right solution. If the failures lead to a loss of revenue or significant productivity, a fix may be required quickly, possibly including reverting back the previous version.
6. What timeframes are available for the upgrade?
Scheduling an upgrade, especially a major upgrade, is never easy. Somebody always has a time when it will be bad, and no time block ever seems to be good for everyone. Nights and weekends are often your only choice in a production environment. You may be able to get more flexibility from the people who will benefit most from the upgrade.
However you schedule it, never (never, ever, NEVER) make a change, no matter how small, on a Friday afternoon right before you leave. These kinds of changes are pre-destined to have unforeseen effects and by Monday you will have forgotten what you did. If you make a night or weekend change, be prepared to stay for a while to make sure things are working as expected.
7. What support level do you have from the vendor?
If you do run into problems, will you be able to call the vendor's support organization for help, even if you're performing the upgrade at night or over a weekend? You might have to perform the upgrade during the day if your support contract only provides daytime support.
8. Could you back out the upgrade if you ran into problems?
If all else fails, can you throw in the towel, cancel the upgrade, and be where you were when you started? If so, then no one is worse off. Most applications, and even operating systems, provide a mechanism to uninstall simple upgrades or patches.
But what if you can't? A major new version of an application or operating system may not allow you to revert back to the previous version. In this case, you're either committed to the upgrade and to living with the consequences, or you will have to re-install from an earlier distribution (and probably also re-customize).
This is by no means an exhaustive list of everything you should consider. Every data center has its own unique issues. But if you can answer these questions to the satisfaction of everyone involved in or impacted by a software upgrade, you should avoid the most common pitfalls.
ABOUT THE AUTHOR: King Ables earned his bachelor's degree in computer science from the University of Texas at Austin in 1982. He has been involved with Unix as a user, developer, systems administrator, consultant and author since 1979.