CAMBRIDGE, Mass. -- Google's Kubernetes has made it to prime time in some forward-thinking IT shops.
Kubernetes is a system for automating deployment, operations and scaling of containerized applications, which groups containers into logical units, called pods, for management and discovery. Companies that have put it into production said it can be used to create a flexible infrastructure that can easily be updated to accommodate changes, as apps are continuously developed and deployed.
Expense management software as a service provider Concur Technologies Inc., which was acquired by SAP in late 2014, uses Kubernetes for its receipts management service because it needed something that could run on-premises, as well as in the existing Amazon Web Services (AWS) public cloud environment, according to Dale Ragan, senior software engineer for the company, based in Bellevue, Wash. Kubernetes is also very pluggable. For example, Concur swapped out Google's cadvisor performance monitoring tool for Prometheus and Grafana, which Ragan said is more performant and sends less traffic across the network.
Though Concur hasn't swapped out any other components of the Kubernetes stack in favor of self-developed or third-party options, it's nice to know they can if, for example, they want to try out a new scheduler, Ragan explained at ContainerDays Boston here this week.
One year later, Kubernetes is still going strong in production at Concur. Dale Ragan gives an update on the company's progress with open source and proprietary flavors of Kubernetes, as well as the multi-infrastructure roadmap Kubernetes will be part of at the company.
Kubernetes in production on AWS
Deploying Kubernetes in production on AWS has taken some coding work for Concur, as Kubernetes doesn't currently offer Availability Zone support for the open source, distributed key-value store etcd used as a hub for cluster coordination and state management. Concur has had to create this itself, but plans to contribute the work back to the open source community.
Barkly Protects Inc., an endpoint security startup based in Boston, turned to a third-party utility created by engineers at Monsanto Company to improve automated failover in etcd. Natively, it can be difficult to determine the number of nodes needed for failover, for example.
The Monsanto open source project, meanwhile, references AWS Auto Scaling groups for discovery, and "the autoscaled etcd cluster fails over perfectly every time," said Mike Splain, lead DevOps engineer at Barkly.
Better native support for external DNS, which Barkly has instantiated using Terraform to make changes in Amazon's Route 53 that point to nodes in the Kubernetes pods, is coming in Kubernetes version 1.3, Splain said. DNS internal to clusters and pods, he added, works as advertised and "is pretty sweet."
Barkly also uses an AWS back end, and Splain said registering Kubernetes nodes with Amazon's Elastic Load Balancers takes a bit more time than he'd like.
AWS was already in use when Splain and Ragan started at their respective companies. Amazon's EC2 Container Service was still in beta when Splain started investigating container management services. Ragan has stayed away from Amazon's native container service due to the need to run common environments on premises and in the cloud, as well as a desire to avoid vendor lock-in with Amazon.
Immutable, malleable infrastructure
Kubernetes works very well for the rollout and rollback of infrastructure and application changes, Ragan and Splain both agreed.
Concur has made its Kubernetes infrastructure immutable, which means if changes to the infrastructure are made, instead of modifying the existing cluster, Concur's engineering team spins up a whole new one and migrates over to it. This means less risk of downtime, according to Ragan.
"In my experience in the past, when you start modifying existing things, that's when you can run into downtime," Ragan said. With immutable Kubernetes clusters, "you can feel confident when you migrate over that it's just like deploying a new app -- we version our whole infrastructure."
Kubernetes makes applications easier to debug using the kubectl and kubectl exec commands, according to Splain, who calls the easy-to-use commands "my favorite thing about Kubernetes."
Help wanted in support, documentation
Overall, Kubernetes is still really young, Barkly said. Ironically, it's owned by Google, but "not really Google-able" to discover solutions to common problems, he pointed out. Support is available on GitHub and a Kubernetes Slack channel, but Splain and Ragan both want better documentation.
"As it gets more and more standardized, the documentation will get better, too," Ragan said. Writing better documentation is "an easy way to contribute back to the community."
While Kubernetes is in production at both companies, it doesn't encompass their entire operating environments. Concur has yet to roll out Kubernetes outside its receipts service, and the Kubernetes cluster at Barkly is not yet getting full production traffic.
"Our company is still in beta, so we haven't sent a huge amount of data through Kubernetes," Splain said. "However, we have load-tested it pretty heavily to compare it to running containers on CoreOS alone, and the performance was almost identical."
IT pros navigate tangled container web
Automating Kubernetes clusters to manage containers