kantver - Fotolia

Could IT change control have prevented an IT deployment failure?

An IT shop experiences performance issues in a long-running deployment that's been operational through many changes. The culprit is probably a build-up of IT configuration artifacts.

We've all experienced a failure of IT change control; some critical system or infrastructure that's been operating for extended periods of time has gone through countless changes, leaving configurations cluttered and bloated.

Stale IT configurations build up from:

  • A maintenance task that was indirectly related to a system and did not account for all connected configurations. For example, when an IT technician decommissioned a server, they did not clean up the related firewall policies and switch port configurations for connected network interface cards (NICs).
  • Confusion about downstream effects. Some infrastructure may have a single configuration for multiple downstream systems, making it hard to know exactly when to clean up the configuration. For example, if a firewall is set up to allow access to a group of servers, the team cannot clean up the firewall's configuration until they decommission all the servers.
  • Configurations left in place out of fear of removing what proves to be a necessary IT component, or due to the notion that making fewer configuration changes to accomplish a goal is the best practice.

Stale configurations have a negative impact on IT infrastructure performance. Unneeded configurations, though no longer relevant, still occupy system memory and consume processing power. Packet inspections still evaluate firewall policies regardless of the policy's current relevance to the IT estate. Configuration drift and sprawl create a death by a thousand cuts scenario: One or two stale configuration blocks won't noticeably slow performance, but the accumulation over time takes a toll on host systems.

Gratuitous IT configurations also affect operations. Engineers and IT professionals get disoriented by a setting with no perceivable purpose. In extreme cases, uncontrolled configurations increase the likelihood of human error during planned changes. The tasks of operating and maintaining an IT architecture become cumbersome as all future changes must work with or around the excess IT configuration structure.

Ward off cluttered IT configurations

Fear of an outage is no reason to avoid IT configuration upkeep.

Proactive and reactive measures can yield serious results when it comes to controlling IT changes; improving performance by as much as 30% in my experience.

Build all configuration cleanup into IT change plans. Take the necessary time to flesh out a plan to address all related configurations for a specific maintenance task. IT teams typically should remove switch port configurations for decommissioned NICs, update firewall policies and groups when changing a server IP address, and remove domain name system records that are no longer in use. Managing the issue on a daily basis is the optimal approach to prevent IT configuration build up over time.

Combat risk of larger change scope through a robust IT change control process. Teams dealing with live production systems fear the service impact of any change. A formal IT change control policy will lessen these unknowns. Thoroughly plan changes, formulate test and backout plans, have peer reviews, follow a set approval process, and schedule and communicate maintenance windows for all changes to critical systems. These steps can almost completely eliminate unexpected outages related to change, and contain outages to maintenance windows. Fear of an outage is no reason to avoid IT configuration upkeep.

Periodically review and clean up configurations on IT systems. Even with IT change control policies and processes in place, it's worthwhile to review systems and identify candidates for cleanup. Plan and execute the cleanup carefully to catch anything that was missed in previous changes.

In extreme cases of configuration sprawl and drift, schedule rebuilds for susceptible systems. Rebuilds are the IT equivalent of taking a sledgehammer to your problem, but if done right and not too frequently they guarantee clean configurations without too much effort or residual effect on the IT estate. This approach also forces staff to stay familiar with the systems and configurations that they manage. I adore and seek out opportunities to transform otherwise mundane work into learning and team enhancement experiences; all the better when it aligns with good IT configuration control practices.

The future is bright for IT change control

Software-defined IT infrastructure, such as software-defined networking, makes the configuration intrinsic to its system. Configuration management databases relate configurations of disparate systems to each other and help IT teams assess what one change will do to the rest of the IT systems it touches.

Some organizations that adopt a full DevOps or continuous deployment model eliminate superfluous configurations. IT infrastructure and application configurations are expressed as one; instances of that singular configuration can be deployed and retracted through automated processes. Once an instance is withdrawn, it leaves behind no residual configuration to generate future problems.

Next Steps

Win the configuration control battle

These management tools make change control easy

Use this guide to determine if a CMDB is right for you

Dig Deeper on Configuration Management and DevOps