One IT organization runs 50 data centers, while another started natively on the cloud and never looked down. Unsurprisingly,...
they have different expectations of container management software.
Every company and team has different goals and requirements to deploy containers. Technological differentiation is not the only -- or even the biggest -- factor when they select a container management tool.
Expertise on staff, tool cost, implementation decisions and the existing ecosystem and underlying infrastructure play a large role in the vendor, tools and technology that's the right fit to scale containers.
"Some like to stick with the Docker product vertical due to lifecycle UX and a focus on simplicity and security," said Bret Fisher, an independent DevOps and Docker consultant, trainer and speaker involved in open source communities, at the O'Reilly Velocity Conference 2017 in New York. "Some choose Kubernetes because it seems the current winner of orchestrators, and others choose Mesos and [Mesosphere] DC/OS due to flexibility and maturity."
The management tool marketplace reflects the maturing nature of containers. "We're just now standardizing what it means to be a container runtime and a container image," Fisher said. The difference between container management tools such as Kubernetes and Docker Enterprise Edition (EE) represents an ecosystem war reminiscent of the iPhone vs. Droid phone wars, he said. Orchestrators and schedulers have 75% the same features, so it often comes down to which one people know and feel comfortable on.
Dealer Tire, a Cleveland, Ohio-based automotive industry distributor, modernized from physical machines to virtual ones a few years ago, and now its web platform operations team is six months into container adoption on private servers in two data centers with VMware virtualization as the host layer. The container management tool evaluation covered Docker, Kubernetes, Mesos and Rancher.
Mesos and Kubernetes seemed complicated, and the team didn't want to manage native Docker via the command-line interface, said Andrew Maurer, IT manager of web platform ops.
"Rancher seemed to make sense. It was low level of barrier to entry; to get it up and running was extremely simple," he said.
It's not the containers, it's the kernel
One reason teams struggle with containers is that they deploy to older kernels such as Linux kernel 3.10, said Bret Fisher, independent DevOps and Docker consultant, trainer and speaker. Containers and orchestrators use modern features of the kernel, so when users try to do container orchestration on aging Linux distributions, they face struggles that wouldn't exist with the 4.x kernel series.
Container orchestration and deployment are so core to the future of IT -- and cram so many workloads onto one OS -- that they now influence the direction of Linux kernel development, as well as Windows, Fisher said.
"As we consolidate more containers onto fewer OSes, we're pushing the limits, which mean you generally want the latest kernel you can get," he said. The OS has to work harder and in a different manner to manage, for example, 100 Apache processes in 100 containers on one VM that has one OS, rather than one Apache process per VM in 100 VMs, which each run a full OS.
Dealer Tire also wanted guidance through not just container adoption but also a shift to treat servers not as pets but as cattle, Maurer said.
Other companies' IT teams have started to branch out from native Docker tooling.
Cox Automotive's inventory solutions group is evaluating Kubernetes and Mesos technologies for container management as its Docker deployment grows, said Jason Riggins, the group's director of production engineering, who discussed his company's DevOps and cloud adoption at Delivery of Things World USA in San Diego.
The primary requirement of a container management tool-- and any other tool they select -- is production stability. "We know how to move stuff really fast," Riggins said, "[but that's not a good thing if] even bad stuff moves fast." And his group also wants a more dynamic tool than the native Docker options, with a particular focus on the container registry. The tiebreaker for container management tool selection will be how much effort goes into maintenance and upkeep.
Container management tool choices often fall along data center vs. cloud lines. "People going with Google Cloud [Platform] tend to prefer Kubernetes. People with complex private data centers tend to consider Mesos, though that's changing as data center venders have started to support Kubernetes and Docker EE," Fisher said. Cox Automotive is consolidating data centers and adopting public cloud, so a container management product must work with on-premises infrastructure and public cloud deployments.
Part of Cox's evaluation of Kubernetes and Mesos is to examine the "scar tissue" from difficult previous container deployment attempts, Riggins said. Peers who have already implemented each technology are also valuable information sources, he said.
When to orchestrate a change
Most companies stick with their container management tool from pilot to large-scale production, and only change course when they hit a limitation. One popular goal for container orchestration is more flexible integration between components, but the market isn't that mature yet, Fisher said.
Social Tables, a cloud-native 100% Amazon Web Services customer, bucked the comfort zone trend when it chucked its initial choice of AWS Elastic Compute Cloud Container Service (ECS).
Bret Fisherindependent DevOps and Docker consultant
"We switched from ECS to Rancher because we wanted to move away from ELB [Elastic Load Balancing] and run our own global load balancing service for better control over our traffic," said Michael Dumont, lead systems engineer in DevOps at the Washington, D.C., firm which provides social event planning and management SaaS. The company also required persistent storage for a Cassandra cluster, an Elasticsearch cluster, Redis, and Prometheus, and with Rancher it also gets DNS-based service discovery, Docker-Compose support, and GitHub OAuth integration for authentication and authorization.
While companies are unlikely to switch container orchestrators, sometimes they don't have a choice. In this emergent space, container orchestrators, schedulers and related tools for storage and network management change constantly. For example, Rancher Labs brought in Kubernetes for Rancher 2.0. Both Maurer and Dumont hope Rancher will keep Kubernetes under the hood to preserve the familiar interface while enriching its management capabilities.
Support matters in emerging technologies
In the rapid modernization climate for an IT organization, any new tool has to do more than provide necessary technology -- it has to be supported.
Cox Automotive will select a supported version of Kubernetes or Mesos, not pure upstream open source, because it encountered difficulties getting container deployments up and running at enterprise scale, Riggins said, adding that they're familiar with taking the unsupported open source route, but not right in this case.
During Dealer Tire's container management tool evaluation period, Rancher's support engineers worked through a problem. "This was before we spent a dime with them," Maurer said. Today, his group relies on enterprise support, and he believes commercial versions of open source technologies are the best option for IT organizations that want to safely move into new areas, and avoid the time and money to get a platform running only to find out support falls flat.
"My biggest challenge with purchased software is it's really hard to [simulate real use] when you're limited to a two-week trial," he said. "It's nice to be able to deploy something, configure something significant and then decide, 'I've invested quite a bit of my business into the software -- I need to buy support to make sure my business continues to succeed.'"
Work with what you have
Social Tables' cloud-native, startup pedigree is the tailor-made case for containerization, but enterprise IT pros can suit up their traditional apps with containers, too.
At Dealer Tire, Maurer's team started with a simple app that was not customer-facing as the lowest-risk entry point to containers. The team communicates with application owners about which apps are a good fit on containers, and which are not. A 100% move to containers is not going to happen at Dealer Tire, but Maurer expects to convert all the web apps. At the same time, the company puts new software development in containers -- a natural fit, in his estimation.
Dealer Tire also decided to stay on premises during its ramp up of containers. It was too much change at once, and changing responsibilities, to go to a cloud model, and some of the company's diverse supported apps are not conducive to cloud ops, Maurer said. However, a future phase of cloud migration would be easier with these workloads encapsulated in Docker containers, he said.
"There's a learning curve, and because the system's new you have to set new expectations on every facet," he said. "What directory are you using? How do you log things? ... How do you communicate your errors and metrics?" Whereas before everything lived on the server, now systems are volatile and ephemeral. "It's not just moving to containers -- you're changing everything about your environment," he said.
The Docker vs. Kubernetes container orchestration war is over
New machine learning monitoring tools better manage infrastructure
Determine how to acquire the right DevOps toolchain