kantver - Fotolia

Enterprise DevOps, open source hit Target's bull's-eye

Pick up a cartful of DevOps advice from retailer Target, which moved to the DevOps culture in its data centers with a balance of stability and innovation.

BOSTON -- No matter how many times we say DevOps is a culture and a mindset, it's hard to deny that it is also a fairly sizeable chest of tools.

Target operates two main data centers to support its retail locations as well as distribution centers. The retail giant moved to an enterprise DevOps model to empower the backend IT team to provide technology services in an entrepreneurial way.

"We're very interested in the public cloud and what that's going to offer in the future, but we do have data centers and mainframes, and that's not going away in the future either," said Jeff Einhorn, group manager of rAPId Infrastructure Services at Target, during the Red Hat Summit here this week.

Target's IT team identified the problems with its legacy method of operation: Developers were frustrated by the provisioning time for new infrastructure; operations staffers were frustrated by the quality of code pushed to production; projects would be completed but no one was left behind to manage and own the project after initial completion; the team had a lot of platform owners, but no stack owners.

"Cloud, mobile and big data with convergence of [infrastructure as a service] and [platform as a service] blurs the lines, bringing operations into development," said Paul Cormier, president of products and technologies at Red Hat.

By collaborating, standardizing and automating, Target could free up time to work on its core infrastructure.

"Data centers aren't going away, so if you can use OpenStack to make someone in the bare-metal world's life easier, that's the way to go," Einhorn said.

Target is developing its DevOps toolset and methodology on Red Hat OpenShift platform as a service (PaaS) atop an internal RHEL OpenStack private cloud, with automation from Chef, Jenkins and other tools. OpenShift was born out of the Red Hat Enterprise Linux and JBoss middleware product lines at Red Hat, so it's a natural combination of operations and development, Cormier said.

"You have a platform with all the tools and libraries in it to go from app development to deployment," he said.

The DevOps approach appeals to IT pros in diverse industries. One enterprise infrastructure manager at Red Hat Summit said he's investigating cloud for certain uses to try it out, and PaaS has some appeal. As Einhorn said, you can't "agilely" rack and stack a server, but you can apply the agile mentality to everything, even mainframe updates. This attendee finds operating data centers more cost effective than a cloud deployment, but still wants to benefit from the agility of DevOps.

Practical tips for DevOps adoption

Target approaches DevOps with a mix of culture, tech practices and measurement to keep improving.

Get support from executives for this restructuring of IT, Einhorn said. If you can't, wait for changes in leadership and prove pilot projects before pushing for the major overhaul again. When you get support, request literal space for the DevOps teams to come together.

Einhorn recommends a "bottoms up" approach to involve separate IT groups with each other.

"Get all the people needed for a service into a room, and do it all in one day," he said.

DevOps makes lean cool again

"You think it's hard to adopt DevOps in your org? Look at what Toyota was in post war Japan. There's hope for cultural change in your IT organization," said William Henry, Red Hat's DevOps strategy lead.

The Toyota Way, also referred to as lean manufacturing, is a precursor to DevOps born in the era of industrialized but inefficient car manufacturing. Lean stresses knowing your process, modular work, automating everything possible for repeatability and scalability, and continually integrating and improving.

Approach standardization with a critical eye as well. Your CI/CD pipes may look very different, but the workflows will be similar, for example, said Henry.

Then do a 30-day challenge with weekly demos. Have internal DevOps Days and hackathons -- the point is to get the right people together to ask the right questions and solve problems.

"Engineers will line up [for group projects] outside of their traditional silos, like a [business intelligence] team member working with an Ops team member on OpenShift," Einhorn said.

Move to a service ownership mentality so projects aren't considered "done" when they're completed. There are still updates and patches and optimizations to do. Target's "Project Argus" helped the team build better management dashboards incorporating system alerts, for visibility over the "entire supply chain," Einhorn said.

"Automation is very good for quality," added Gordon Haff, a Red Hat cloud strategy manager, who also presented here. "It takes some effort to set up that automation and get it right, but then you're done."

Take the time to stand up automation tooling in your various data center environments to maintain consistency and speed of delivery from development.

Consistency in the world of rapid iterations is still a work in progress, Einhorn said. Once a continuous integration and continuous delivery (CI/CD) pipeline is established, it gives innovation a solid foundation.

The DevOps flow.
A well-founded enterprise DevOps process incorporates CI and CD.

His group is considering a centralized repository of Chef cookbooks to prevent variations and duplications.

"If we write a cookbook for Websphere, for example, we want to be able to use it across the whole enterprise," Einhorn said.

Customization is a last resort. Target's goal is to have every node managed with Chef.

"Sometimes you do have competing [cookbooks] and one will be maintained more" to the point where it wins out as the adopted methodology, he added. 

While Target is still getting the infrastructure in place to deploy apps in containers with Red Hat OpenShift PaaS, Einhorn reports the container methodology reduces steps from development to deployment.

"Containers feel like a great way to do hybrid cloud. To deploy containers that work on OpenStack and public cloud providers would be really cool," he said.

Target's IT security staff also like that developers can mock up a new application or service in OpenStack for them to investigate and test for weaknesses, Einhorn said.

CI/CD pipeline for DevOps.
A CI/CD pipeline should work across the dev and production environments.

Target also takes a DevOps approach to mistakes. One bad configuration mistake can have a broad data center impact, Einhorn said, and no one is immune to it happening. However, getting different IT silos communicating regularly means less finger-pointing when things do go wrong. "Once you see a mistake, you write a test [to catch] it, and then that [mistake] never happens again."

One attendee worried about creating the "haves" and the "have-nots" in the data center when some applications and services move to this collaborative, automated environment and others cannot.

"You do have to reorganize," Einhorn said, "but you don't have to do it all at once." He also recommended rotating people into different projects for exposure to the tools and culture of DevOps.

Meredith Courtemanche is the senior site editor for SearchDataCenter. Follow @DataCenterTT for news and tips on data center IT and facilities.

Next Steps

Should DevOps be your goal?

Keys to success with the methodology

Dig Deeper on DevOps Team Organization