Application development teams are under pressure to deliver releases as fast as possible -- provide new features, fix bugs and beat the competition. Increasingly they turn to continuous integration (CI) and continuous delivery (CD) to build, test and add small incremental updates quickly. Many or most of the steps in a CI/CD are automated.
But the need for speed often omits or minimizes meaningful security testing. Vulnerabilities discovered in the production version of an application can lead to compromised systems and data, which cost significant time and resources to fix and ultimately will delay future releases.
To reduce the chances of vulnerabilities going undetected during the software development lifecycle, organizations must add continuous security validation to the CI/CD pipeline. They also must treat potential security issues as equally important as any other bug or quality issue within the development process.
Security built throughout the CI/CD pipeline maintains agile development by giving developers early warnings of vulnerable or flawed code. This makes developers overall more productive, which reduces time to market despite the added layer of security checks. And more secure applications over time will gain consumer trust, versus other applications that are shown to put users and their data at risk.
Should you secure the CI/CD pipeline itself?
It's important to integrate continuous security testing into the CI/CD pipeline, but there is another angle of CI/CD security to consider: securing and protecting the CI/CD pipeline itself. This involves security-specific tasks such as tracking logins to development machines, or authenticating users pushing changes to the CI/CD pipeline.
As organizations embrace agile development and DevOps, CI/CD pipelines become critical business assets. As the SolarWinds security breach and Codecov attack show, these assets are increasingly high-value targets, and organizations must carefully consider how to shield these assets.
Infusing security throughout a CI/CD pipeline requires a major change in daily activities. An organization's security tools may not fit into a CI/CD pipeline, which means upfront investment in different tools and training. The addition of new practices and security checks will initially slow down development times. Unlike unit and integration tests, security tests may generate false positives which can break a build. Security tests must recognize when human intervention is required -- automated tools won't eliminate the need for some manual testing.
Here are some tips to help incorporate security into your CI/CD pipeline, from preplanning through the coding and build phases and through your deployment method.
1. Conduct threat modeling
First, conduct a threat modeling exercise to map threats to the application, so everyone understands what needs protecting and how to do it. The outcomes of this exercise will guide how, when and where you add security to the CI/CD pipeline, and helps everyone understand the threat environment in which the application will operate. This avoids time and resources spent on tools and processes that are not ideal or effective against an application's particular attack surface.
2. Know the CI/CD pipeline and components
It's also important to have visibility into the CI/CD pipeline -- its assets and boundaries, the steps, tools and code repositories -- to establish an asset registry and a full understanding of the application architecture and software development lifecycle. The aim is to collect any data that will show changes to assets and actions. List all the sources of possible telemetry the pipeline produces -- system logs are obvious sources, but also look into repository, build and deployment logs. Ensure cloud-based resources output to relevant logs, and that logs capture software build and code repository activity.
With these foundations in place, you can start to add security into the development and deployment processes. The choice of tools may depend on the frameworks, languages and operating systems used in a project, but many vendors offer versions that cover most scenarios. To avoid unnecessary disruption or frustration in the DevOps process, try to add new tools one at a time, so they can be tuned and understood by everyone in the team.
3. Set up checks and safeguards to commit code
In the development phase where code has not yet been checked into version control, the most important security tool is an IDE security plugin. This analyzes code as it is written in real time, warns developers if they will potentially introduce a vulnerability, and provides remediation advice. Examples include Microsoft DevSkim, SonarLint and Puma Scan.
Code written by less experienced developers should be peer-reviewed using a security checklist such as OWASP's Cheat Sheet Series. Delivering code in small units makes it easier for these manual code reviews to detect any errors.
Establish rules and controls for committing code into the central repository, such as pre-commit hooks to inspect code, and ensure the code meets baseline standards for quality and security. One essential safeguard is to avoid copying sensitive data such as API keys and authentication tokens to Git repositories or production environments. Tools such as CyberArk Conjur, git-secrets and detect-secrets all block secrets from entering the code base.
4. Analyze committed code quickly
Developers should receive prompt feedback once they've committed their code. Static code analysis tools are the best for the job because they do not require the application to be running, and many provide remediation advice as well.
Send code scan reports to security or development teams so they can prioritize any follow-up actions. Add any warning or alert generated during these tests (and any other tests) to a bug tracker, such as Jira, to ensure the vulnerability is assigned to someone and fixed, not forgotten.
5. Test for security just like other functions
Run security unit tests based on the application's threat model just as you would run regular unit tests against components and functions. Analyze misuse and abuse cases, as well as edge cases. Tests on authentication, session management and other security requirements must validate that both allowed and disallowed actions are handled correctly. Run these scans against the repository, not the build pipeline, otherwise failed security checks may stop a build and frustrate everyone. Brakeman and phan are commonly used analysers, while tools such as SonarSource and Veracode are available as SaaS or for local installation.
6. Check open source vulnerabilities
Another important check at this stage is to analyze imported open source libraries and components for known vulnerabilities. These third-party packages play a vital role in modern software development but new vulnerabilities can crop up at any time. These vulnerabilities could impact the security of an application even if its code hasn't changed.
Software composition analysis (SCA) tools analyze open source code, third-party components, and binaries to provide real-time security alerts, and also flag compliance and licensing issues. Two popular SCA tools are WhiteSource Bolt and snyk.
Once code quality is verified and a CI build has succeeded, the application can be deployed to a testing or QA environment to verify that there are no security vulnerabilities as the application runs.
7. Accept and validate with more security tests
Security acceptance testing, performed along with other acceptance tests, helps to ensure risks highlighted in the threat model are fully mitigated. Institute passive scans for code pushed to the pipeline's test environment to identify obvious vulnerabilities. Also, run more detailed active scans as a nightly job to simulate common hacker techniques and uncover misuse cases that trigger hidden or unknown vulnerabilities and failures. A popular, free penetration testing tool is OWASP ZAP, which can integrate into most deployment processes.
In addition to the application, validate the infrastructure to check for any vulnerabilities. Cloud providers such as Microsoft Azure and AWS provide tools to check configuration and the status of the infrastructure on which the application runs, to flag insecure infrastructure settings or assets.
8. Automate with infrastructure as code
Continuous deployment presents a big security advantage with the ability to implement infrastructure as code (IaC). With IaC, machine-readable definition files automatically provision secure infrastructure, repeatedly and at scale, instead of manual interactive processes to set up complex networks and devices. Automated tools analyze these files to spot errors and configuration drift post deployment. Only the CD pipeline should be allowed to create and deploy the infrastructure, which means no manual changes or direct access to the infrastructure via the console. Code deployment to production should only occur after the acceptance stage is completed and approved.
9. Continuously monitor after deployment
Once you've introduced an application and its supporting infrastructure you must continuously scan and monitor everything to detect, prevent and remediate new vulnerabilities or weaknesses because the threat landscape constantly evolves. To help visualize application state and performance, tools such as Kibana and Grafana can bring diverse data sets into dashboard-style reports and generate alerts if thresholds are broken. Netflix and NASA use the open source StackStorm which uses an "If This Then That" engine to fix known problems and when necessary, escalate them to humans. Another tool, Detectify, continuously scans development, staging and production environments and provides notifications of vulnerabilities and remediation advice.
Successfully implementing security throughout the CI/CD pipeline requires changes to tools and processes and continuous collaboration across development, security and operations teams. Conduct regular security awareness training to keep everyone abreast of the latest threats, and schedule postmortems to learn from a project's mistakes and successes. These steps will improve the security of your organization's CI/CD pipeline.