This content is part of the Essential Guide: Use these DevOps examples to reimagine an IT organization

Tighten IT operations control with a DevOps feedback loop

DevOps empowers developers to shorten build time and release code to production faster, which achieves nothing without a feedback loop on how production absorbs that code.

SAN FRANCISCO -- Think DevOps is all about speed? Think again, outage-fearing IT ops control freaks.

The benefits of DevOps include the ability to move faster and align IT with business goals, but IT teams should concentrate on a lesser-known facet of the methodology: deployment to incident correlation. More small code changes being verified as production-ready should coincide with more IT operations control and fewer incidents.

DevOps feedback loops reveal how an organization's software runs in production, said Scott Prugh, chief architect and VP of software development and operations at CSG International, while presenting at the DevOps Enterprise Summit this week. CSG, headquartered in Englewood, Colo., provides business support systems for telecom companies and runs a print mail service. It has about 20 technology stacks in operation.

Prugh found that, though DevOps improved CSG's software build process, it did not benefit IT operations: 98% of IT incidents occurred outside of the software release, meaning about 92% of incidents were dumped on the production IT staff to fix. The conclusion: Developers didn't understand how their software functioned in production and how users consumed it, and no feedback came from traditional IT operations to teach them what needed to change. They needed more Ops in their DevOps.

DevOps reveals to developers that live production is a dangerous, high-stakes place. "Ops is hard!" admitted Erica Morrison, director of software development for CSG, who copresented with Prugh.

Operations teams at CSR dealt with competing priorities from their customers and with too much work in flight at any given time. Work took place across multiple tools, with poorly standardized communication and overworked team members, Morrison noted. Developers handed off code to operations teams, which handed it off to platform teams -- code releases historically went to QA once and then straight to production. The teams had different schedules, visions and goals. In addition, a lack of engineering skills in the operations group prevented improvements and encouraged "duct tape engineering," where binaries get thrown over the wall with monitoring slapped on top and a lifetime of babysitting and support that the ops person would have to provide.

Kill your apps

Reducing app count and complexity also improves IT operations control. With DevOps, quicker functionality development means IT operations teams can pull the plug on legacy apps kept alive for years.

U.S. Citizenship and Immigration Services in the U.S. Department of Homeland Security creates diverse applications for internal and external users. As part of its DevOps transformation, Immigration Services is eliminating legacy apps in two ways: developing a single new application to replace the functionality of multiple existing apps and rewriting monolithic applications into distributed microservices architectures. For example, a new, feature-rich case management application written as a microservice will enable the agency to turn off applications that each fulfilled some part of that functionality in isolation, said Yemi Oshinnaiye, chief of the systems engineering division of Immigration Services' office of information technology.

While paying down that technical debt to create a new unified application is difficult, noted Verizon's Nambiar, it unshackles infrastructure scalability and allows IT operations control to shift from constant administration to long-view strategic engineering.

Operations is more disruptive in nature than development, Morrison said, with a lot of unplanned work that makes team members jump from emergency to emergency. The "duct tape engineering" approach of fixing development problems in the operations sphere built up complexity in production, resulting in high technical debt. "We were in a world of chaos" with outages and a lack of IT operations control, Morrison said.

CSR reorganized into service delivery teams, grouping operations and development and enforcing best practices, such as automated testing. Constant feedback showed developers exactly where the problems were in their code. Incidents over time dropped sharply, leveling off at a 10x quality improvement.

Operations required development principles to reap the benefits of DevOps: automation, version control and peer review. Manual processes in operations lead to infrastructure as a task instead of infrastructure as code, Prugh said. CSG brought in architects and developers to drive automation in tasks from configuration to reporting. Operations submitted configurations and other changes to source control and peer review just as developers did with application code.

Looks the same -- works the same

CSR deploys development, quality assurance and production environments exactly the same way, and this strategy has worked for telecom provider Verizon.

If you give developers infrastructure that's API-driven, they can experiment and deliver into production more easily with code that scales, said Chivas Nambiar, director of DevOps and public cloud platform engineering at Verizon, who also presented at DevOps Enterprise Summit.

Operations had misgivings about changing any production infrastructure, fearing instability. However, standardizing on a private cloud platform that developers already used improved IT operations control and lessened complexity and errors on live systems -- Verizon experienced a 30% reduction in outages year over year. Next, the operations team learned how to write infrastructure as code at scale, building in security and compliance on a public cloud platform.

It's a difficult transition, Nambiar admitted, but the effort not only reduced outages, but also cut infrastructure costs 30% and reduced time to deployment.

Agile software development can pay off for operations teams even where 100% DevOps integration is not possible. A government contractor sometimes only handles development or operations for a project, or may need a distinct handoff between those groups within the same company to satisfy compliance requirements. Nonetheless, applying DevOps principles within those confines demonstrably leads to safer, faster code deployments.

A trial period of Agile in a change-adverse, decades-old Lockheed Martin aeronautics team halved the anticipated software build time and spending, but also produced "a record number of low defects at hardware integration," said Robin Yeman, Agile and DevOps coach for the Bethesda, Md.-based contractor. And while this point won't appear on any IT operations control dashboard or incident analysis report, "these engineers ... [had] never had so much fun" on a project, Yeman said.

Next Steps

Using DevOps for efficient IT platforms

Out of business chaos comes DevOps order

How DevOps adopters lead organizations

Dig Deeper on Application Rollout Planning and Problems