carloscastilla - Fotolia

EBay to bottle its special sauce for Kubernetes management

Kubernetes is notoriously difficult to deploy in private data centers. EBay's Suneet Nandwani explains the company's answer to Kubernetes management: TessMaster.

One of the world's largest enterprises intends to contribute its formula for OpenStack and Kubernetes management...

to the open source community next quarter.

E-commerce titan eBay Inc. has one of the world's largest OpenStack private clouds. Since 2015, it has rolled out Kubernetes container orchestration and management to a farm of 22,000 CPU cores in six availability zones to support 178 applications, including Elasticsearch analytics engine, Apache Kafka distributed streaming and an artificial intelligence (AI) platform.

To accomplish its aggressive on-ramp, eBay engineers created software that automates the deployment and management of Kubernetes clusters on its OpenStack private cloud. The tool, dubbed TessMaster, was first discussed publicly by eBay in late 2016, but at this month's OpenStack Summit the company formally introduced the project and said it will make the source code public in the next quarter. caught up with Suneet Nandwani, senior director of cloud infrastructure and platforms at eBay, who gave that OpenStack Summit presentation, to dig into the details of TessMaster for Kubernetes management.

How exactly is TessMaster positioned against OpenStack and Kubernetes themselves?

eBay's Suneet NandwaniSuneet Nandwani

Suneet Nandwani: We developed it to manage the Kubernetes lifecycle on top of OpenStack. Using TessMaster, we can deploy Kubernetes, add capacity, decrease capacity, remediate Kubernetes itself in case there is a fault or Kubernetes nodes that go down, and we can do Kubernetes networking. We've designed it [for Kubernetes management] not only on OpenStack but also on top of some other cloud provider like Google Cloud [Platform], AWS [Amazon Web Services] or [Microsoft] Azure.

Although most of what we've talked about has been about managing Kubernetes, TessMaster is designed to be much more. We intend to manage a big portion of our data centers using TessMaster, because it is extensible. So we can [use it to] manage things like configuring networking, onboarding racks, [and] setting up load balancers. Since we started using TessMaster, things have become so much more elegant and easy for us [that] we want to use it to manage other infrastructure, some of the lower-level things that are under our cloud.

Is all of that missing from Kubernetes today?

Nandwani: Kubernetes doesn't come with capabilities to integrate with a cloud provider. They sort of leave it up to whoever's implementing it. Kubernetes doesn't have any support for deploying Kubernetes on OpenStack, or many other public clouds. In the [Kubernetes] community, things are a little fragmented, but people have come up with their own [ways of solving] this. For example, Google gives you a fully managed instance of Kubernetes so you don't have to deal with that problem. But we don't want to use the vanilla Kubernetes stack Google would provide on its cloud. We want to run our own instance of Kubernetes because we're making some additions for eBay, and we want to run the same Kubernetes in both private and public clouds.

Other open source tools have popped up, [such as] Kolla and kops, and those are also playing in the same space. TessMaster was designed to be a little different, because it's a Kubernetes application -- it uses Kubernetes to manage Kubernetes. It's self-bootstrapping. Kubernetes has a lot of good principles about being declarative, being drift-proof, doing auto healing and remediation, and we are able to leverage that for Kubernetes itself.

Why make that a separate project? Why not contribute that IP to Kubernetes or OpenStack?

Nandwani: We do have a plan to contribute this back. [TessMaster] came up as an internal need. We will have conversations with the [Cloud Native Computing Foundation], as well as the OpenStack Foundation, but it will likely be a separate project wherever we decide to put it. It's not core Kubernetes.

Don't public clouds such as Azure have their own support for Kubernetes already?

Nandwani: When you use Azure's Kubernetes you're obligated to use whatever distribution Azure provides you. Especially for companies that run hybrid cloud, you don't want to have a different flavor of Kubernetes on the public and private cloud. The Kubernetes we get on Azure doesn't work for us. [TessMaster] allows us the flexibility to use our own customized distribution of Kubernetes, which has some value-adds for eBay.

Our [change management database] integration is important because any application that runs on our Kubernetes cluster automatically gets populated into our CMDB. We also integrate our own logging and monitoring platform, and we want to run the same logging and monitoring on the public cloud, as well as the internal cloud. One size fits all doesn't work for everybody unless you're a small-to-midsize company, [and] for somebody at our scale it may not work.

How do you make TessMaster accessible to other companies? EBay has scale beyond most environments.

Nandwani: If you're running Kubernetes on an internal cloud you still need to solve the problem even at [a] smaller scale. You have options, you can use kops or Kolla or some of the other tools [for Kubernetes management], but I think TessMaster has some inherent advantages over those tools. In fact, we looked at a lot of these tools before we decided that none of them would meet our needs. One of the basic things we looked at was being drift-proof. That means we have a declared size of [a] Kubernetes [cluster] and then we have the real size of [the cluster] in the environment and you need to reconcile them automatically. That is something that TessMaster gives us.

Did you look at Red Hat OpenShift?

Nandwani: We have. OpenShift is for the lifecycle management of applications running on Kubernetes, but we are managing Kubernetes itself.

Will eBay continue to use OpenStack or will TessMaster eventually subsume it?

Nandwani: We are running Kubernetes on top of OpenStack, just like Google runs it on their cloud infrastructure or Azure runs it on their cloud. Our Kubernetes is layered on top of OpenStack, and we see the value of programmable infrastructure and don't plan to move away from that.

Beth Pariseau is senior news writer for TechTarget's Data Center and Virtualization Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.

Next Steps

For modern IT, turn to immutable infrastructure

Set up a DevOps tools home lab today

Track real Kubernetes on OpenStack adoption numbers

Dig Deeper on Scripting, Scheduling and IT Orchestration