GP - Fotolia

IT pros contend with Linux container complexity

Deploying a handful of containers is simple, but deploying across thousands of machines means dealing with some serious complexity.

CAMBRIDGE, Mass. -- Linux containers aren't new to the market anymore, but they're just beginning to open new frontiers in the deployment of systems at scale.

Broadly speaking, clustered Linux container systems change how IT pros imagine apps and design development pipelines.

"You have to think about your applications in a very ephemeral sense -- they're running on OS instances that can quickly die, so you have to architect for failure," according to Santosh Iyer, research engineer for CommerceHub, an e-commerce service provider based in Albany, N.Y.

From app to app, the effect of containers might differ, "and overall, there's a whole new ecosystem that comes into play" of resource management back ends and a scheduling framework to manage them, he said.

Still, with the upfront investment to change the app architecture, containers can improve efficiency and make engineers' jobs easier to develop apps and maintain them, Iyer noted.

Containers also can provide a new way to segment and secure workloads, according to Jason Toy, founder and CEO of, a machine learning startup based in Cambridge.

However, Toy said, Docker doesn't mix very well with the high-performance GPUs his company uses for computation: "That Docker value proposition of 'write once and deploy everywhere'-- we don't have that luxury with GPUs."

Cluster management nitty-gritty

Taking containers "from hello world to production-grade" traverses layers of problems to worry about, especially when container deployments get large enough to require clusters of systems managed by abstraction layers, such as Google Kubernetes and Apache Mesos, according to Shannon Williams, co-founder of Docker container orchestration software vendor Rancher Labs, based in Cupertino, Calif., who presented at ContainerDays Boston here this week.

At the very least, more than a few hosts means containers must be monitored using log tools, such as Prometheus and Logstash, and "they themselves are complex enough to need their own compose file," Williams pointed out.

Beyond that, "you immediately grow in complexity as you start to dive into the technology," he said. Suddenly, IT pros must think about configuration injection, metadata services, persistent storage, access control, secrets management and audit trails.

"People are passing these hurdles, but you don't want to take it on without building up your knowledge a bit," Williams said.

Orchestration is still on the horizon for's Toy.

"When our machine learning models have to start talking to each other, I see a lot more orchestration being required," he said. But so far, "even the beginner tools still seem kind of like overkill -- it's all about simplifying it and finding the right level of abstraction."

Each type of clustered container scheduler comes with its own problems, according to another presentation by MIT researcher Malte Schwarzkopf.

For example, monolithic cluster schedulers, like the one currently offered as Docker Swarm, can lead to inflexible policies and code growth, Schwarzkopf said. Two-level schedulers, such as Mesos, can lead to resource hoarding, where resources are locked by the scheduler, even if it's not going to claim them. The shared-state model, based on a Google design and commercialized by CoreOS, can lead to scheduling conflicts when two schedulers try to claim the same resource at the same time.

There are newer cluster scheduler models, but they are strictly academic today, including a system developed by Schwarzkopf and a colleague at the University of Cambridge, called Firmament. With Firmament, a rudimentary integration exists with Kubernetes, and integrations with Docker and Mesos are still in discussion, Schwarzkopf said.

Further complicating matters is that choosing between cluster managers isn't an exclusive decision, according to CommerceHub's Iyer. For example, the company recently decided to use Amazon's EC2 Container Service (ECS) after evaluating Kubernetes and Mesos. "ECS fits with the other cloud-based apps we're currently using," Iyer said.

But he didn't rule out using Mesos or Kubernetes at some point in the future as applications dictate: "It's just a matter of the appropriate use case coming up."

Security, compliance further container complications

Experts at ContainerDays Boston also shared some best practices for security and compliance in container environments, such as not putting passwords and keys in the service tree or in Docker files. Users also should consider whether logs contain protected information, and whether they need to make logging and audit trails temper-resistant, according to a presentation by Elliot Murphy, CEO of Kindly Ops LLC, a managed DevOps service based in Portland, Maine, which specializes in compliance.

"It's time for the security team to learn some new tricks [with containers]," Murphy said. Time-tested utilities that IT security pros are used to working with, such as ossec and auditd, don't fully support container environments and are hard to configure.

Traditional antivirus software "constantly gets eye rolls and derision" from IT pros, "but rather than dismiss them, you should try to think of a better choice -- there are better tools out there," Murphy said. He recommended a tool from, which uses a domain name system to detect when malware is trying to phone home.

Containers do make one aspect of security easier: running centralized scans against code written in multiple languages and for multiple environments, according to Murphy. "You can take one thing you build, and put it in a bunch of different environments and run checks against it," he said.

Beth Pariseau is senior news writer for TechTarget's Data Center and Virtualization Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.

Next Steps

Why DevSecOps brings dev and ops closer together

Container and configuration management differences

The container effect on server architectures

Dig Deeper on Managing Virtual Containers