This content is part of the Essential Guide: Tools for IT shops seeking full-stack control and unified operations

Programmable infrastructure fends off configuration drift

A new way to automate the provisioning and management of infrastructure is appearing on the horizon, spearheaded by startups.

Programmable infrastructure designs should be on enterprise IT's radar, even if widespread implementation is still on the horizon.

Programmable infrastructure is a natural extension of infrastructure as code and automated configuration management tools, taking things a step further by adding a platform as a service (PaaS) abstraction layer that automatically generates a full software-defined infrastructure based on application changes.

A subset of programmable infrastructure is known as immutable infrastructure, in which changes to the application prompt the provisioning of an entirely new infrastructure, rather than making changes to the existing environment.

San Francisco-based startup Zenefits, which offers HR management software as a service, acquired a software developer last December who had designed a programmable infrastructure system, Project Duplo.

Duplo is heavily influenced by PaaS systems, particularly Microsoft Azure, where Zenefits principal engineer Venkat Thiruvengadam once worked. However, unlike PaaS offerings from service providers that abstract infrastructure completely away from the user organization, Duplo allows Zenefits' infrastructure administrators to set policies for underlying resources, including the orchestration of monitoring tools.

Thiruvengadam says he finds programmable infrastructure a happy medium between automated configuration tools, which he feels don't have a broad enough scope, and full-fledged PaaS, which he sees as too prescriptive.

Programmable infrastructure "is a middle ground," Thiruvengadam said. It can set up the infrastructure by implicitly reading the application needs and providing a declarative interface to application teams ... [and] does not mandate that the application be written in a certain way."

Homegrown programmable infrastructure

At Zenefits, Duplo is used to create programmable infrastructure on Amazon Web Services.

"Usually what happens is there is an infrastructure team, and they have a conversation with the application team that is typically offline, in emails and ticketing systems," Thiruvengadam said. "Then every time an application changes the same process has to be repeated."

With this system, "it can be very slow" to get applications deployed, taking anywhere from days to weeks, he said. App developers also often forget to tell infrastructure teams when resources should be deleted.

With Duplo, by contrast, once an infrastructure admin sets base policies for the infrastructure, the software learns the requirements of the application. From there it derives the infrastructure, and generates and applies a configuration to a pool of AWS resources. This infrastructure can include Docker containers in Duplo's current design -- and so it can take the place of orchestration software, such as Google Kubernetes or Mesos, as well as configuration management tools, such as Chef or Puppet by interfacing with Amazon application programming interfaces.

"Traditionally, when an infrastructure administrator gets configuration requirements, he goes and calls automation tools like Terraform, Chef or Puppet to make it happen," Thiruvengadam said. "There is still automation, but the config generation is sort of manual."

Duplo immutable infrastructures are used for tests of the infrastructure-generating system at Zenefits, but can also be used for production infrastructure, as it is at SAP's Concur and other forward-thinking companies.

Prepackaged programmable infrastructure: Fugue

Most companies can't create software such as Project Duplo for themselves, but another startup launched this week at AWS Summit in New York looks to offer programmable infrastructure to the masses.

Like Duplo, Fugue offers a declarative model using infrastructure as code to generate automated deployments on AWS as part of the continuous integration and delivery process. It also enforces policies and ensures resources are terminated when they are not in use. Fugue is offered as a software product deployed in a user's own AWS account rather than as software as a service, to appeal to large security-conscious enterprises.

Fugue, founded by a former AWS principal solutions architect, looks to offer a better version of the cloud giant's CloudFormation utility, which is "hard to use and doesn't do policy enforcement," according to Fugue CEO Josh Stella.

Similarly, AWS supports security standards and regulatory requirements such as the Health Insurance Portability and Accountability Act, but it's a very do-it-yourself environment in which "it's easy to do the wrong thing and break the rules," Stella said.

Fugue is generally available and is priced based on the number of 'conductors' (provisioning agents, essentially) that are deployed; generally one conductor can manage one AWS account. Specific pricing numbers were not disclosed. Fugue also did not have public customer references at press time.

Programmable infrastructure's audience small but growing

So who will buy Fugue or look to recreate Duplo? Not too many people yet, according to Donnie Berkholz, analyst with 451 Research, as widespread deployment of programmable infrastructure remains a future proposition.

Today, around 40% of IT decision-makers surveyed by 451 Research are using some kind of configuration management tool to automate infrastructure, but haven't yet taken the next step of setting up programmable or immutable infrastructure.

Environments that most commonly lend themselves to immutable and programmable infrastructure are containerized, since containers can easily be regenerated unlike VMs. However, just 15% of 451's surveyed IT decision-makers have deployed at least one containerized app in production so far, and programmable and immutable infrastructures probably comprise a slice of that 15%.

Berkholz expects programmable and immutable infrastructure techniques to grow in popularity over time, "but it is a shift in how code and infrastructure are built and deployed."

Beth Pariseau is senior news writer for TechTarget's Data Center and Virtualization Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.

Next Steps

Automate, or be consumed by microservices

Chef aims for app-level automation

Ansible pushes automation beyond DevOps

Dig Deeper on Scripting, Scheduling and IT Orchestration