Database DevOps contradictions snarl enterprise IT automation

Databases are part of the application modernization process but can also present an intractable problem for DevOps teams.

For many companies, the words database DevOps present a contradiction in terms.

Database DevOps initiatives occur in two areas of a company's IT automation pipeline -- the preproduction test bed and the production deployment process. And both environments raise their own set of tricky technical questions.

In DevOps for database test environments, organizations must quickly spin up system clones to offer data that accurately reflects the production environment, while protecting sensitive corporate information. On the other hand, database changes in production represent the widest gulf between Agile application development techniques and the ability to deploy them onto real-world IT infrastructures.

Database DevOps stares down testing issues

Typically, developers are the first to uncover the technical difficulties of working with databases in automated test environments.

"They don't have the right data," said Brandon Cipes, managing director of DevOps at cPrime, an Agile consulting firm in Foster City, Calif. "How do they make the test environment illustrative of production, and how do they keep that environment updated and secure?"

Test data management tools are available from vendors such as DBmaestro and CA Technologies, but this is where people and process challenges begin to emerge as well, said Nirmal Mehta, senior lead technologist at Booz Allen Hamilton, a consulting firm based in McLean, Va.

"Siphoning production data off a queue and cloning production traffic for realistic inputs can incur infrastructure costs a lot of organizations don't want to pay for," Mehta said. "It comes down to not wanting to spend the money for automation -- DevOps is trying to force the issue."

Containers are all the rage in automated application tests because developers can quickly spin them up and then destroy them, but stateful database applications don't yet lend themselves well to containerization.

"To some extent, there's a fundamental incongruity between databases and containers," said E.T. Cook, chief advocate at Dallas-based consulting firm,, which helps companies deploy containers.

There are tools for persistent storage in containerized applications, but the most tried-and-true approaches don't solve the issue of true data portability, Cook said. Those require pinning a container to a host so it remains attached to a data volume. Products are available that offer truly distributed container persistent storage, but enterprises still must learn how to manage and integrate these systems.

New tools also tackle containerizing databases so they'll spin up quickly in test scenarios. Some examples are Ocean9's containerization offering for SAP HANA and WinDocks' containerization product for Microsoft SQL Server. Microsoft also supports containerization of SQL Server in its most recent version. For the most part, containerized databases for automated test environments come closest to aligning databases and DevOps.

Database DevOps tools evolve, but gaps remain

New tools continue to emerge for database DevOps in production environments. Security tools, such as HashiCorp's Vault, Docker Secrets and Chef Vault, can automate the distribution of security secrets, such as passwords and access keys, in dynamically changing environments. But there's still room for these tools to improve on how they dole out granular permissions.

"Generally, there's a market gap in the tool set where subsets of permissions aren't easily identified within application profiles," said Nirmal Mehta, senior lead technologist at Booz Allen Hamilton, a consulting firm based in McLean, Va. Still, there are many more dynamic database security tools available than organizations are using right now.

Enterprise clients have recently shown increased interest in rapid database provisioning tools, such as those marketed by Datical and Redgate Software, as traditional applications are brought into the DevOps fold, said Robert Stroud, an analyst at Forrester Research.

Despite the availability of these tools, many enterprises still develop their own abstraction layers to manage multiple database types. These abstraction layers act as buffers between applications and databases so that changes to either of them don't disrupt the other, said cPrime's Cipes. Approaches to such abstraction tools can vary widely depending on an individual enterprise's data and the types of databases used.

"I expect there'll be more [vendor] players in the next year or two," Cipes predicted. "Customers are increasingly asking questions about data."

Production databases present further DevOps challenges

Even when database DevOps problems are solved in test, the gulf is still wide between the test environment and the production infrastructure where apps will actually run. Production databases can be cumbersome to update, require tribal knowledge of complex data schemas to access and operate and are slow to scale.

To tackle these issues, some organizations have moved production applications to NoSQL databases, which don't require schemas. However, many have backtracked after NoSQL systems presented management problems in production, Cook said.

"A lot of teams realized that, when people said MongoDB was simpler, they meant for setup," Cook said. "But for a production app, people end up having to build a schema and data sanitization system on top of it anyway, and if you're not a DBA [database administrator], NoSQL turns into a mess really easily."

At the same time, databases such as PostgreSQL have advanced in the last 18 months to incorporate nonschema objects, which allow unstructured data to be part of queries against indexes.

"All of a sudden, Postgres became the best of both worlds," Cook said.

The result is that many organizations find themselves with a patchwork of different databases that range from in-memory databases, such as ElastiCache and Redis, to traditional central data stores in MySQL and Postgres systems and repositories of unstructured objects in NoSQL databases, such as Mongo and Amazon's DynamoDB. This can burden DBAs with too many systems to manage in the production environment.

Many organizations turn to cloud database services to avoid the need to hire and train armies of DBAs. Cloud database services can also solve common on-premises database issues with rapid scaling and performance tuning in production as applications change. As always, cloud computing comes with caveats around operational costs and the security of systems outside the corporate firewall.

And all of that, of course, assumes an organization's DBAs have agreed to come along for the database DevOps ride.

Find out more about the cultural difficulties of database DevOps in part 2.

Beth Pariseau is senior news writer for TechTarget's Data Center and Virtualization Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.

Next Steps

Manage a distributed database with a sandbox

Tune database performance through some simple steps

More database support options available from Catalogic

Dig Deeper on Managing Virtual Containers