Increasing security by hardening the CI/CD build infrastructure

CI/CD pipelines are made up of various different components such as application code, source code repositories, image repositories, build servers, containers, container orchestration frameworks and the hosts running these. All of these components work with each other and they depend on trust relationships with each other. A security failure in one component can lead to an attack on the entire CI/CD infrastructure.

This article provides some insights into how to increase security by hardening the CI/CD build infrastructure.

Learn Container Security

Build your skills around Docker and Kubernetes security, including key technologies, creating and running a secure cluster, and more.

Start Learning

Locking systems 

All the components in a CI/CD pipeline must be well hardened and appropriate access controls should be implemented. Only authorized users should be allowed to access systems in a pipeline. This includes granular controls at user, group, and project levels.   

Build systems should be locked down to only allow internal access from corporate networks or through corporate VPN.

Host repositories 

Source code repositories is where an organization’s proprietary information resides. A compromised code repository can become a source of a backdoored build running in production. At a minimum, unauthorized access to the code can lead to intellectual property theft. 

Secret data often gets committed to the code repositories accidentally. Pre-commit checks must be performed to prevent such accidents and already pushed secrets must be scrubbed from the repositories’ history.

Code should be committed to a repo from a developer machine with appropriate controls. Great restrictions on code commits should be applied. For instance, pre-commit security checks should be performed on code being committed and developers should sign the commit using a hardware token so it is trusted and verifiable. Developers should be able to commit to only their projects and authentication controls should be implemented to access the source code repositories. When the code is pulled from the repo, the same rules apply. The user/process/system should be appropriately authenticated and authorized. 

Configuration managers  

There are several configuration management tools commonly used in CI/CD pipelines. Some of these configuration management tools consist of instructions on how the infrastructure should be set up. This is helpful in building robust, stable and consistent infrastructure. Considering docker containers as an example, Docker uses a configuration file named Dockerfiles. These files contain a list of commands (build instructions) based on which a docker image is built.

Controls should be in place to lockdown changes to these Dockerfiles as any unauthorized modifications on these files can lead to a malicious image leading to backdoored or vulnerable containers running in production.

Build servers  

Hardening the build infrastructure is crucial to prevent software supply chain attacks.  Access to the build servers should be provided only to authorized users. A malicious user gaining access to the build server can do massive damage to the build infrastructure as well as the users/systems, which are using the builds produced on the infrastructure.

Traditional hardening such as updating the operating system, installing security patches, limiting network level exposure of the services should be done. Imagine a Jenkins server capable of spinning up containers is compromised by an attacker. Chances are high that the attacker can also elevate the privileges to root due to the privileges CI servers usually contain. 

Build servers should go through security audits, looking for deviations from hardening best practices in the form of corporate hardening policies and alerting the administrators allowing them to appropriately harden the infrastructure.

The build server should not contain any secrets that may give access to other systems. Secrets should always be retrieved at runtime from a secret vault. In addition to it, user access and code used for the build process should be regularly audited. Another commonly seen issue on build systems is running build tools with root privileges. This should be avoided and least privileges should be used wherever possible.

In addition to the preventive controls in place, tasks performed on the build server should be verifiable and immutable logs should be maintained to be able to perform regular audits on the entire build process.

Learn Container Security

Build your skills around Docker and Kubernetes security, including key technologies, creating and running a secure cluster, and more.

Start Learning

Conclusion

Since CI/CD pipelines are made up of various different components such as application code, source code repositories, image repositories, build servers, containers, container orchestration frameworks and their underlying hosts; every component should be appropriately hardened and their exposure should be limited to increase the security of a CI/CD pipeline.

Access to these systems should be locked down with appropriate access controls. 

Sources

https://reproducible-builds.org/

https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html

https://www.bankinfosecurity.com/improving-supply-chains-verified-reproducible-builds-a-15816