SAN DIEGO -- DevOps training amid a skills shortage can be an arduous task, but some companies with clout have found a way to ease the process.
Representatives from Starbucks, Etsy and CSG International explained during last week's PuppetConf how to transform an existing organization so it embraces DevOps, what to look for when acquiring new talent from outside, and measuring DevOps success in a world of rapid change.
Ultimately, DevOps is not one team, or a job title for one person, but "as an organization we should all be doing DevOps," said Mike Dilworth, a technical director with Paris-based consulting firm Capgemini, who previously led a DevOps transformation at Sainsbury's, a large retailer in the United Kingdom. "We should all be thinking in a systems perspective and all trying to communicate better."
DevOps training begins from within
So where does an IT pro begin to create sweeping change, especially for a large and established organization?
For panelists at PuppetConf, creating a DevOps team began within the existing organization before it expanded to bring in new talent -- and it starts at the top, with new relationships between IT leadership and employees.
At Starbucks, for example, managers were encouraged to become more of an embedded part of their IT teams, while encouraging them to make more decisions, said Courtney Kissler-Hawkins, vice president of retail technology at Seattle's Starbucks coffee shop chain.
Erica Morrisondirector of software development, CSG International
The traditional IT leader sets direction and directs work, as opposed to being a collaborative part of a team, Kissler-Hawkins said of her experience.
"When you start talking about things like, 'You should know what your team's doing' so you can help, you get, 'Oh, you're saying I should micromanage my team,'" Kissler-Hawkins said. "I didn't say go in and tattle, I said go in and see. Go and observe. Ask questions. Be someone who can remove constraints involved in that."
Creating that environment is difficult when managers don't acknowledge that they are no longer the sole director of what work should be done and how it should be done, Kissler-Hawkins said.
Panelists also agreed there's no substitute for on-the-job DevOps training, and that DevOps shouldn't be a permanently separate team within a company -- at least, not for very long.
With a temporary DevOps team, CSG International gained experience in setting up an environment with configuration as code, and then that proof-of-concept team "becomes the evangelists to the rest of the organization," said Erica Morrison, director of software development at CSG International, a business services provider based in Colorado. "We're getting a lot of education simply by doing."
While some companies choose to begin DevOps training, others have it thrust upon them.
"Merging two companies together was a huge endeavor and we doubled the number of systems, and we said, 'How are we going to do this without losing our lunch?'" recalled an executive in the panel session audience from a major U.S. transportation company who asked not to be named. "I don't know that we called it DevOps back then -- it was more about, 'How are we going to connect and collaborate and create value?'"
These organizational changes then force technology transitions organically, the executive said.
"If you speed up your infrastructure and you speed up your software pipeline, you run smack into manual testing and that's where the pain point is," he said. "That's where we're at."
Blue chip DevOps tips on what to look for in new hires
DevOps skills are still rare and DevOps training and certification processes are immature, which means companies must look for raw behavioral qualities in people they hire into DevOps roles rather than a long list of prior experience.
"People who are adaptable, people who like change, that's quite a good thing," Dilworth said. "Someone who'll see new things and, without any prompting, go off and research them."
Drive and initiative go together with the learning and improvement process that's crucial to creating an effective DevOps team, Morrison added.
"Maybe you don't have a DevOps skill set today, but I'm going to give you a problem, and if you don't know how to solve it you're going to go figure it out and be better for it," Morrison said.
Discipline is also a critical personality trait in new hires for Kissler-Hawkins, "because there has to be a minimal set of guardrails to make this successful -- just the right amount of structure," she said.
Hiring for DevOps means finding people who can live with that, and look to strategically solve problems rather than relying on tactical heroics, Kissler-Hawkins said.
Etsy's Morgue: How to measure DevOps success -- and failure
When panelists at PuppetConf were asked by the audience how they know they're on track with creating a DevOps team, answers varied.
Release velocity is a key measure of DevOps success, Dilworth said -- but a more interesting measure is whether the marketing department uses the IT department to solve technical problems, or if they're still trying to make a "shadow IT" end run around it using the cloud and a credit card.
"If the marketing department is going to IT, that's a huge success," Dilworth said. "Because normally in a large enterprise they'll do anything they can to avoid going through IT."
At Starbucks, DevOps teams are asked how much of their time is spent working on breakthroughs versus operational maintenance, Kissler-Hawkins said.
"If your percentage of breakthroughs is going up, then you can typically tie that to delivering more value for the company, and it also resonates with the team," she said. "Then they see, they're not really automating their way out of a job but getting freed up to do more breakthrough work, or more value-add [activities] ... it also ties to employee engagement."
The ability to safely make a change told CSG's Morrison that the DevOps process was headed in the right direction.
"It's a very fuzzy metric, but also the extent to which you fear change or fear deployments is a good measure of how your DevOps initiatives are working," she said.
How an organization responds to failure is just as important as how it measures success, said Bethany Macri, a software engineer with online crafts marketplace Etsy, speaking in a separate panel at PuppetConf.
Macri helped create a chat platform Etsy calls "Morgue," which provides a place for DevOps teams to do a postmortem on failures and outages.
There are no 'shoulds' allowed in Morgue, and the use of the word 'why' is also discouraged, all as part of an "obsessive focus on blamelessness at Etsy during postmortems," Macri said.
"Contrary to what it might seem like, blamelessness does not mean a lack of responsibility," she added. "And it absolutely does not mean that everyone agrees, [but] when you're not afraid of being blamed, the kind of candid disagreements that can happen ultimately make better software."
DevOps can mean it's time for an IT overhaul
How much do you know about DevOps?
The dark side of DevOps