7 best practices to ensure your CI/CD pipeline's security
CI/CD security means fortifying everything that flows through your software pipeline with securing elements -- but the pipeline itself can be a target. Lock it down with these CI/CD protective steps.
Many enterprises step up from DevOps to DevSecOps by adding continuous security validation into their CI/CD pipelines. A shift-left model builds security into applications from the beginning to greatly reduce the chances that production releases contain exploitable flaws and vulnerabilities. This compares with a process flow in which security testing occurs at the end of the development lifecycle.
However, the CI/CD pipeline itself is a target of compromise and abuse. Its access to proprietary code, databases, credentials, secrets, and development and production environments make it a rich prize for hackers. To many DevOps-driven organizations, this is a growing concern. This is especially true in the wake of the SolarWinds attacks and the Codecov breach, which compromised the software supply chains for thousands of organizations.
Take preventative steps with these tips to keep your CI/CD pipeline secure.
1. Map threats and secure connections
First, you must understand what potential security threats exist and which vulnerable points within the entire build and deployment process need additional protection. Conduct a threat modeling exercise to map threats to the pipeline.
Every connection to the CI/CD pipeline is a potential point of compromise. Regularly patch and scan all devices that connect to the pipeline, and block the devices if they fail to meet security policy requirements. All connections should be made over Transport Layer Security. Configurations for repositories and build servers should be locked down and any scripted builds scanned for vulnerabilities.
2. Tighten access control
Establish access control lists and rules to control all access to the CI/CD pipeline -- it should be easily and instantly clear who has access to what, when and how. Log, monitor and manage access to every component and resource of the pipeline, whether it's role-based, time-based or task-based. Conduct regular audits to ensure redundant machine or service accounts and ex-employees are closed or have permissions revoked. Require strong authentication for all users, and regularly rotate passwords. Machine identity and authentication are vital to secure nonhuman access in containers.
3. Separate duties and enforce permissions
Development environments tend to be permissive when it comes to implementing least privilege because permissions can slow down or interfere with testing. Nevertheless, it's important to establish enforceable permissions for everyone who accesses the pipeline to control who can commit code changes to the repositories, create containers and deploy code to different environments. Least privilege also applies to applications, systems and connected devices within the pipeline that require privileges or permissions to perform essential tasks.
Clear and enforced separation of duties should ensure no one person or account can access more than it needs in the continuous software delivery pipeline. Improperly credentialed users, or bad actors, can't establish control over the entire pipeline and mess with the code or processes. Moreover, they won't easily be able to penetrate deeper into other corporate systems and data.
4. Keep secrets safe
secrets are authentication credentials, such as usernames and passwords, API tokens, SSH keys and encryption keys, that allow access to applications and services. They are literally the keys to a project's data and resources. If these credentials are improperly secured and used, they can open the doors to a serious data breach or intellectual property theft.
Control where you locate secrets and who has access to these credentials with a dedicated key management service. This encrypts, stores and injects secrets at runtime only when they are required. So, they are not exposed during the application's build or deployment, and they never appear in the actual source code.
Even if you use a key management tool, it's a good idea to monitor and audit code repositories to identify and remove secrets that have been committed to the code base. Better still, use tools to stop them being pushed or passed on in pull requests. Developers should never write code that prints a secret to the console log, even during testing -- some CI tools can mask secrets if they are printed or output in any way, such as to the console or debug logs.
5. Lock up your code repository
Businesses that self-host a Git-based code repository roll the dice against the high risk that a misconfiguration could create a vulnerability and expose the pipeline to attack -- Nissan's North America division recently learned this the hard way. However, even a hosted version control service must ensure secure access to the repository. Implement two-factor authentication and signed commits to prove the identity of the author, and preferably two people, to perform a commit. Define access roles on a per-repository basis to ensure only developers with valid access credentials can interact with individual repositories.
Train developers how to use Git and particularly how to use .gitignore correctly so they understand which files will and won't be ignored when they commit code. The best strategy is to use inclusion, not exclusion, to avoid constant updates to .gitignore as they add new files to a project.
Code repositories, such as Git, are of high value to hackers because they contain proprietary source code and intellectual property -- there have been ransom attacks where repositories have been wiped. Sign up for repository alerts and advisories. And always maintain a secure local backup, which is typically part of an overall backup policy rather than part of the CI/CD process.
6. Diligently monitor and clean up
A continuous software delivery pipeline, by definition, is a flow of constantly moving parts and processes for builds and deployments, but don't let the cadence distract you from proper security maintenance chores. Always monitor the CI/CD environment as it runs, and terminate any temporary resources such as containers and VMs after you complete tasks. To reduce the attack surface of containers and VMs, remove any unnecessary tools and utilities -- netcat is one example -- and launch containers in read-only mode if possible.
7. Stay informed and have a plan
You can compromise third-party resources and services in a pipeline at any time. IT teams must closely monitor the security feeds and notices of vendors whose products and services they use to immediately discover and act on any breaking news. Establish an incident response plan to handle such an event and help minimize any impact on the pipeline. For example, in response to the Codecov breach, both Netflix and the Vim text editor quickly rotated their credentials as a precaution, while HashiCorp rotated one of its GNU Privacy Guard keys used for release signing and verification. Whenever you add new tools or services to the CI/CD pipeline, update and include them in the incidence response plan.
CI/CD pipelines are core to many organizations' products and services, so treat them like other operations that are critical to the business. Secure them accordingly to prevent supply chain attacks or other failures that will impact the build and delivery processes. Segment them from the rest of the enterprise network, and monitor them constantly for signs of suspicious or inappropriate activity. The abundance of interconnecting services and components, many of which are supplied or controlled by third parties, means there can be no set-and-forget approach when it comes to pipeline security.
