sommai - Fotolia
SAN FRANCISCO -- A DevOps transformation in large organizations can't just rely on mandates from above that IT pros change the way they work; IT leaders must rethink how teams are structured if they want them to break old habits.
Kaiser Permanente, for example, has spent the last 18 months trying to extricate itself from 75 years of organizational cruft through a consumer digital strategy program led by Alice Raia, vice president of digital presence technologies. With the Kaiser Permanente website as its guinea pig, Raia realigned IT teams into a squad framework popularized by digital music startup Spotify, with cross-functional teams of about eight engineers. At the 208,000-employee Kaiser Permanente, that's been subject to some tweaks.
"At our first two-pizza team meeting, we ate 12 pizzas," Raia said in a session at DevOps Enterprise Summit here. Since then, the company has settled on an optimal number of 12 to 15 people per squad.
The Oakland, Calif., company decided on the squads approach when a previous model with front-end teams and systems-of-record teams in separate scrums didn't work, Raia said. Those silos and a focus on individual projects resulted in 60% waste in the application delivery pipeline as of a September 2015 evaluation. The realignment into cross-functional squads has forced Kaiser's website team to focus on long-term investments in products and faster delivery of features to consumers.
IT organizational changes vary by company, but IT managers who have brought about a DevOps transformation in large companies share a theme: Teams can't improve their performance without a new playbook that puts them in a better position to succeed.
Scott Nasellosenior manager of platforms and systems engineering, Columbia Sportswear Co.
At Columbia Sportswear Co. in Portland, Ore., this meant new rotations through various areas of focus for engineers -- from architecture design to infrastructure building to service desk and maintenance duties, said Scott Nasello, senior manager of platforms and systems engineering, in a presentation.
"We had to break the monogamous relationships between engineers and those areas of interest," Nasello said. This resulted in surprising discoveries, such as when two engineers who had sat next to each other for years discovered they'd taken different approaches to server provisioning.
Short-term pain means long-term gain
In the long run, the move to DevOps results in standardized, repeatable and less error-prone application deployments, which reduces the number of IT incidents and improved IT operations overall. But those results require plenty of blood, sweat and tears upfront.
"Prepare to be unpopular," Raia advised other enterprise IT professionals who want to move to DevOps practices. During Kaiser Permanente's transition to squads, Raia had the unpleasant task to inform executive leaders that IT must slow down its consumer-facing work to shore up its engineering practices -- at least at first.
Organizational changes can be overwhelming, Nasello said.
"There were a lot of times engineers were running on empty and wanted to tap the brakes," he said. "You're already working at 100%, and you feel like you're adding 30% more."
IT operations teams ultimately can be crushed between the contradictory pressures of developer velocity on the one hand and a fear of high-profile security breaches and outages on the other, said Damon Edwards, co-founder of Rundeck Inc., a digital business process automation software maker in Menlo Park, Calif.
A DevOps transformation means managers must empower those closest to day-to-day systems operations to address problems without Byzantine systems of escalation, service tickets and handoffs between teams, Edwards said.
Edwards pointed to Rundeck customer Ticketmaster as an example of an organizational shift toward support at the edge. A new ability to resolve incidents in the company's network operations center -- the "EMTs" of IT incident response -- reduced IT support costs by 55% and the mean time to response from 47 minutes to 3.8 minutes on average.
"Silos ruin everything -- they're proven to have a huge economic impact," Edwards said.
And while DevOps transformations pose uncomfortable challenges to the status quo, some IT ops pros at big companies hunger for a more efficient way to work.
"We'd like a more standardized way to deploy and more investment in the full lifecycle of the app," said Jason Dehn, systems analyst for a large U.S. retailer he asked not to be named. But some lines of business at the company are happy with a status quo, where they aren't entangled in day-to-day application maintenance.
"Business buy-in can be the challenge," Dehn said.
Break your applications with chaos engineering to pre-empt problems
DevOps needs AI in its future, but admins worry about their place
Every organization should encourage their early adopters