pixel_dreams - Fotolia

How Big Finance and the feds conquered SecOps with code

SecOps is among the most daunting frontiers of DevOps transformation, but banks and the U.S. federal government are forging ahead using Puppet IT automation.

SAN DIEGO -- The only thing DevOps shops have to fear about SecOps is fear itself, say big banks and government consultants who've begun to secure their infrastructures with code.

Puppet IT automation and code from the auditing tool OpenSCAP are in use at large financial institutions, as well as some agencies of the U.S. federal government, according to panelists with experience in those institutions who spoke at a public session at last week's PuppetConf here.

"I've been consulting recently with an insurance company, a bank, state and federal government, and in all those brownfield environments we've found the same problem, which is just fear," said Trevor Vaughan, co-founder of consulting firm Onyx Point Inc., in Hanover, Md.

Banks, feds enforce security as code

Government agencies that have succeeded in moving past that fear incorporate all facets of the organization into a DevOps project that starts small, Vaughan said. The project pulls in security, as well as operations and upper management, from the beginning. In one agency, security IT pros who refused to learn to code security policy for the infrastructure were let go, he recalled.

Everybody in the organization is becoming part of the development workflow, because the development workflow is driving the mission of the business.
Trevor Vaughanco-founder, Onyx Point

"Effectively, everybody in the organization is becoming part of the development workflow, because the development workflow is driving the mission of the business at this point," Vaughan said.

Government security pros Vaughan works with are writing Puppet code, as well as OpenSCAP code, and using these tools to apply security policies programmatically to their environments, so they can do security checks in a standardized manner.

This actually improves security, Vaughan said.

"You would have to actually unharden my systems to scan them, which, of course, we won't do," he said.

To convince auditors the systems are still secure, Vaughan's team wrote an open source plug-in to Puppet that maps the intent behind changes.

"You can actually tell if the Hiera [key/value lookup] data or the [external node classifier] data that was applied to your Puppet system as it compiles is what your policy says it should be," Vaughan said. "Then, it'll give you a report back into PuppetDB, or it can be placed on the server for analysis later."

Meanwhile, at large financial firms, similar SecOps changes are happening within IT organizations.

"We're seeing some engineering ops people come into security more to try and get their insight in, and we're seeing a little bit of Puppet coming in," said Robert Grignon, senior solutions architect for TIAA, based in New York. His company is working with service compliance software that it will manage with Puppet.

TIAA also recently rolled out a 12,000-node Puppet Enterprise deployment in a greenfield environment, in part, to prove the security-as-code concept to the rest of the organization, Grignon said.

Elsewhere in the financial services industry, part of the DevOps initiative is to flatten a very hierarchical organization into "a sensible group of people who know what they're doing and can ensure they've got the right level of security on their application changes," said Seamus Birch, independent consultant with experience working at Deutsche Bank, Barclays Capital and other large financial institutions.

Security must shift left and become involved throughout the development process, Birch said.

"Instead of having these teams of people running around doing compliance and audit work, you can just cut that out by ensuring the changes that were being made were secure, and that means bringing the security team into that process," Birch said.

Fear and loathing lingers in SecOps

SecOps is still an uphill battle at many companies, audience members said in Q&A portions of the sessions.

One executive from a Midwestern community bank said even if he trusts security people to harden systems through code, auditors might not.

"It's trust, but also verify," he said. "How do I know that something that's being done with configuration management that's beyond my control isn't planting a rootkit?"

DevOps panelists urged the bank executive to exchange information with peers in the financial industry, such as Capital One.

Another audience member at the later security and compliance panel asked for advice on what one thing she should say to security professionals at her company to have a conversation about DevOps.

"I would say, 'It's all about locking everybody out of the system. You don't give anyone the keys; no one's got access; everyone's locked out,'" Birch replied. "There's not a security person in the world [who] doesn't want to have that conversation."

Organizations can achieve consistency using Puppet IT automation for remediation, and offer visibility through a change history in version-control platform Git, said Sean Millichamp, enterprise architect for managed hosting provider Secure-24 LLC, based in Southfield, Mich. "A lot of times, those sorts of things were hard to detect or lurking in the shadows, and to be able to see it that concisely is very valuable -- that's where I'd start that conversation."

In the end, there's also no substitute for experience to answer such questions, Vaughan added.

"Cultural fear is high with brownfields," he said, recounting a conversation with one client about how developers were afraid about how patching would work in a Puppetized system.

"It was, 'How is my application going to run? My developers are afraid. I can't patch it, I can't update, I can't do anything,'" Vaughan said. "I'm sitting there going, 'You can build a box in 10 minutes and just try it. Just try it. If it doesn't work, say, we can't do it, fine -- but nine times out of 10, it'll work.'"

Beth Pariseau is senior news writer for TechTarget's Data Center and Virtualization Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.

Next Steps

Advice to launch DevOps training at your shop

Beware these backward DevOps habits

What does rapid, iterative security look like?

Dig Deeper on Application Rollout Planning and Problems