Managing Secrets in CI/CD Pipelines
Managing secrets in CI/CD pipelines is crucial to improve cloud security. Learn how Harness utilizes secrets management in CI/CD.
The most common source of data breaches and hacked supply chains? Exposed data and secrets. From Uber to SolarWinds, every year, companies and customers are finding the importance of security, data, and governance. But how can engineering and product teams balance the demands of innovation and speed with the confidence of delivery and change? This blog explains secrets and how to secure them through automation, tooling, and CI/CD pipelines.
What Are Secrets?
If you’ve ever posted a private key to your code repository, then you’ve shared a secret. A secret is any piece of information that you want to control access to, such as passwords, API keys, configuration files, OAuth tokens, and ssh keys.
Secrets are only for a few people to know. In software delivery, secrets live in environments, 3rd party APIs, production infrastructure, and tools.
Why are Secrets Relevant with CI/CD?
So why do secrets and CI/CD pipelines matter?
Secrets exist throughout the delivery lifecycle as application secrets, credentials, and other sensitive information should be secured, obfuscated, and governed.
- Code repositories like GitHub, GitLab, and BitBucket.
- Continuous Integration servers and Continuous Delivery platforms – Jenkins, Bamboo, JFrog, TeamCity, Drone, Harness, GitLab CI/CD, CircleCI, and so on.
- Cloud Providers and Container Platforms – AWS, GCP, Azure, Kubernetes (as keys, configuration files, and other information).
- Infrastructure – (SSH keys, load balancer credentials).
- Collaboration providers – Slack, SMTP, Jira, ServiceNow.
- Verification providers – AppDynamics, New Relic, Splunk.
- and in the software service or application itself.
These days, organizations have multiple application services deployed in different environments with different tools and levels of access managing the process. CI/CD pipelines carry our code changes through various phases of the development lifecycle quickly, safely, and repeatedly delivering any changes to users. CI/CD, therefore, provides a foundation for secrets management throughout a delivery lifecycle if we can integrate secrets management practices into the process.
In the next half of this blog, we’ll share how to solve major secrets management challenges.
The Challenges of Managing Secrets in CI/CD
Over the past six months, we’ve listened to hundreds of customers share their pain and challenges for managing secrets while delivering with CI/CD pipelines. Below are the most common features requested of software delivery platforms:
- A central place to see all the secrets that are used within CD tools. .
- Fine-grained access management for each secret.
- Visibility into secrets usage across deployment workflows and pipelines, e.g., ssh key being used in workflow X.
- Changelog capabilities to track when was the secret added, by who, and what changed.
- Notifications when a secret is used at runtime (e.g., Jenkins password being used to download artifacts or SSH key being used to deploy the artifact to a server).
- Deprecate and remove outdated secrets - implement secrets rotation.
- Migrate existing secrets to a new service, e.g., moving secrets to a different Vault setup or a different Amazon KMS.
- Deactivate the secret at KMS or Vault level in case there is a threat of compromise.
Let’s discuss in more details the kinds of solutions that exist today for addressing these challenges.
Secrets Storage and Management Tools
Secrets management systems offer solutions for storing and accessing secrets. Popular solutions include HashiCorp’s Vault, Docker Secrets, KeyWhiz, and AWS Secrets Manager. It’s fairly common now that secrets management tools offer not only secret storage, but also key management/encryption and identity management (this is where we hear about services like AWS KMS and PKI as a service, respectively). Let’s discuss HashiCorp Vault and Amazon KMS in more detail.
HashiCorp’s Vault is a commonly used open-source solution for storing and managing secrets. The stored data is encrypted, and access to that data is shared with tokens. These tokens can be created using specific privileges for specific lease times giving access to specific secrets. The tokens can be revoked at any time, making the secrets inaccessible.
It’s a popular way of centralizing sensitive data. Every component of your delivery architecture that uses secrets accesses them at runtime via access tokens and discards them immediately, making sure that secrets access is never available for longer than needed.
If you are evaluating secrets management tools, this New Stack blog provides some considerations when evaluating HashiCorp Vault’s features.
Amazon KMS is a popular solution for encrypting and decrypting secrets. Using Amazon KMS, users create an encryption key assigned an Amazon IAM role, which is configured to share the level of access for services, environments, machines, and any other component in our architecture, including CI and CD systems. This key reference is used for encryption and decryption.
To gain a better understanding of how key management systems work, here are the steps involved with encryption/decryption.
- During data encryption, a new encryption key is asked from KMS, which returns the key and the key’s encrypted value.
- Using the plain text key, the secret is encrypted.
- The encrypted key and encrypted secret then are stored.
- For decryption, the encrypted key is sent to KMS, which in turn returns the plaintext key.
- Using the plain text key, the secret is then decrypted and used.
This ensures that sensitive information is never stored in a system as plaintext data (key or value), and that for each secret that gets stored, the encryption key is unique. The capability for encryption/decryption of keys can be controlled through the AWS console, and if there’s ever a breach, the master key can be disabled, ensuring no one gets further access using the exposed secret.
Improve Your Secrets Management Process in CI/CD
So, how can DevOps, DevSecOps, and Engineering teams better automate and leverage these solutions as part of Continuous Delivery? Software delivery platforms for today need to address certain components of the secrets management process in order to scale and govern the supply chain.
At Harness, the management of secrets is also core to our architecture and solution.
Harness includes a built-in Secrets Management feature that enables you to store encrypted secrets, such as access keys, and use them in your Harness applications. Below is a diagram representing how secrets are used with Harness.
The secrets are always accessed/decrypted at the time when they are needed, and at no time are they stored unencrypted. Furthermore, the Harness manager does not have access to your key management system. The delegate sits in the private network within your setup, so with Harness, you never have to make your secret managers accessible publicly. This adds an important layer of security.
Example Secrets Management CI/CD Workflow
When using CI/CD platform solutions, it’s common to first add a secrets manager tool. In Harness, you can do this by selecting Security > Secrets Management. This dashboard will allow you to click Configure Secrets Managers.
From here, click “Add Secrets Manager.”
The Configure Secrets Manager dialog box adds a secrets manager. Our documentation provides an overview for adding any of the following secrets managers.
- Add a Google Cloud KMS Secrets Manager
- Add an AWS Secrets Manager
- Add an AWS KMS Secrets Manager
- Add a HashiCorp Vault Secrets Manager
- Add a CyberArk Secrets Manager
- Add an Azure Key Vault Secrets Manager
In this example, we are adding an AWS KMS secrets manager instance.
Provide the account access information needed.
If a user does not create any manager, by default the Harness platform uses its KMS default implementation to encrypt/decrypt your secrets:
Your secrets manager(s) shows how many secrets it has encrypted so far.
Harness also supports the functionality to migrate your secrets to another secret manager if there is a threat of compromise on any secret manager.
It’s important to note that secrets can also come in the form of files. Users can upload text or file-based secrets. Within the Harness CI/CD platform, you can leverage these secrets in one place across multiple parts of your organization and throughout the delivery process.
As you can see in the screenshot above, each of these secrets has information on where they are being referenced, who has uploaded or changed them, and where at runtime (e.g., workflow) they have been used so far. This gives you a centralized way to manage your secret texts and secret files and enables you to see which workflow in which environment is using your secret, providing you with complete visibility.
Managing Secrets with Harness CI/CD
This blog post shared multiple ways CI/CD is solving for security and secrets management challenges. We covered what is a secret, what tools are used to manage sensitive information, and how CI/CD platforms are helping to prevent major hacks and data breaches. If you’d like to try any of the workflows shared, try Harness for free today.