The Basics of GitOps Secrets Management

When Linus Torvalds created Git in 2005, the software industry shifted to a more distributed and declarative working model. Unlike traditional central version control systems, Git was fully distributed and could be hosted locally wherever a developer desired. This allowed for techniques such as branching, merging, and pull requests. All of these terms are common knowledge for developers because Git became the world’s standard, and Git repositories became the single source of truth for developer projects. 

But the new world standard presented a new challenge: in a world where Git repositories acted as a project single source of truth, how do you manage Git secrets? 

Secrets like API keys, OAuth tokens, certificates, and passwords are extremely sensitive. Conventional wisdom would say to not host them in a Git repository, but secrets still find their way into dangerously exposed repositories because it’s convenient, they're hardcoded into textfiles, or a developer thinks it’s safe because the repository is private, secrets aren’t always kept safe. The reality is Git repositories can be cloned, and once cloned, everything stored goes along with it. Private repositories don’t offer enough protection to ensure someone won’t get their hands on sensitive secrets. Developers need to keep close tabs on their secret management processes to reduce the risk of a breach. 

GitOps adds additional complexity to GitOps secret management. GitOps is an intuitive way for developers to deploy software efficiently and relies heavily on Git repos as the single source of truth for projects. In order to successfully complete a GitOps-based deployment, developers will need access to the secrets. An organization won’t want to impede developer velocity by coming up with a convoluted secrets management process, but at the same time, improper secrets management could post a huge risk to the business. 

GitOps Basics and Best Practices

GitOps uses a system’s desired configuration stored in a revision control system, such as Git, to manage software deployments. Instead of making changes via a UI or CLI, a developer makes changes to the configuration files that represent the desired state. When compared, a difference between the desired state stored in Git and the system’s actual state indicates that not all changes have been deployed. These changes can be reviewed and approved through standard revision control processes such as pull requests, code reviews, and merges to master. When changes have been approved and merged to the main branch, an operator software process is responsible for changing the system’s current state to the desired state based on the configuration stored in Git.

GitOps doesn’t require a particular set of tools, but the tools must offer this standard functionality:

• Operate on the desired state of the system that is stored in Git.
• Detect the differences between the desired state and the actual state.
• Perform the required operations on the infrastructure to synchronize the actual state with the desired state.

In an ideal implementation of GitOps, manual changes to the system are not permitted and all changes to the configuration must be made to files stored in Git. In extreme cases, permission to change the system is granted only to the operator software process. The infrastructure and operations engineers’ roles in a GitOps model then shift from performing infrastructure changes and application deployments to maintaining GitOps automation and helping teams review and approve changes.

How Kubernetes GitOps Secrets Work

At the most basic level, a "secret" in Kubernetes is a method to safely store sensitive data and information, such as private keys, passwords, or authentication tokens. The information is easily accessible and still protected from scenarios that could jeopardize security. By default, containerized applications running in Kubernetes often require credentials to operate correctly and interact with other infrastructure or applications.

Kubernetes provides Secrets as objects to store confidential information without being visible in the application code. This mechanism makes it possible to use sensitive data with less risk of accidental exposure when managing the operations of Kubernetes itself. From a basic level, the process to create Kubernetes secrets is fairly straightforward:

• A trusted individual or source defines a Secret by creating a specific name for it and entering the confidential data. Kubernetes operators never see the actual data itself, only the name.
• Once a workload needs to access and use the confidential data, it will refer to that previously created specific name to retrieve it

Handling Kubernetes Secrets in GitOps

Git relies on Git workflows and automation to keep configurations in repositories in sync with production environments as updates are made. But keeping sensitive data in Git in any capacity introduces a security risk, even if the repository is private with strict access controls. If a secret gets committed to a Git repository in plain text form, it must be revoked and no longer used. However, there are still ways to handle secrets in GitOps that mitigate security risks.

For those who want to take a GitOps approach but also provision secrets to their applications without putting their sensitive information at risk, let's go over two of the popular methodologies to manage secrets in GitOps:

1. Store encrypted secrets in a Git repository and leverage automation to decrypt and render them as Kubernetes Secrets
2. Store references to the secrets in a Git repository Instead of storing encrypted secrets directly in Git. Then leverage automation to fetch the actual secrets based on their references before rendering them as Kubernetes Secrets.

Method 1: Encrypted Secrets

Bitnami's Sealed Secrets and Mozilla's SOPS (Secrets OPerationS) are the two most popular open-source options for storing encrypted secrets in Git repositories and share a similar approach. 

Both tools do have their own advantages, disadvantages, and limitations which we'll cover.

Bitnami Sealed Secrets

Known for ease of use, Bitnami Sealed Secrets takes advantage of public-key cryptography to encrypt secrets with two core steps: using source control to store secrets and using a decryption key within a Kubernetes cluster to decrypt secrets. Encryption occurs on your computer while the decryption occurs on the Kubernetes cluster by a controller. A CLI tool, Kubeseal, makes encrypting secrets simple because it has access to the cluster to retrieve encryption keys.

There is also a custom resource, or an extension of the Kubernetes API, called SealedSecret that holds encrypted information so that the controller can decrypt it and create a Kubernetes Secret. This sealed secret controller acts as a "watch guard, "looking for any sealed secrets in a cluster across every namespace. This means that each time new sealed secrets get created, the controller will automatically detect them.

The main advantages of Bitnami Sealed Secrets are usability and eliminating the need to use a separate Secrets Manager, like Hashicorp Vault. That being said, you will still need to manually encrypt each secret. The main limitations to call out are that the solution exclusively works with Kubernetes and becomes harder to manage at scale with a large number of clusters.

Mozilla SOPs

SOPS (Secrets OPerationS) is a more flexible CLI tool for encryption and decryption not limited to use cases with Kubernetes. This tool has been around for a while and gained popularity during a time when keystores were the standard. Given that it is a CLI tool, using it effectively in native Kubernetes environments can be less effective and requires additional tools. These tools include the use of plugins for tools like Helm, extra work to build custom container images with SOPS, and extensions to support using it, such as Argo CD.

SOPS supports multiple input formats like YAML, ENV, INI, BINARY, and JSON. It supports integrations with Key Management Systems (KMS) like Hashicorp Vault, Azure Key Vault, GCP KMS, and AWS KMS to provide encryption keys for securing secrets rather than storing the secrets directly within the KMS. When no KMS is available, the option exists to use a Pretty Good Privacy (PGP) key pair from the command line instead. This is a unique approach that provides flexibility across different development environments. For example, teams could use a PGP key pair in sandbox environments and then use a KMS in production environments without the need to change the layer of tooling underneath.

Similar to how SealedSecrets work, secrets are manually encrypted by developers. It uses the SealedSecrets workflow but uses SOPS for encryption. It reads decryption keys from a key store and then decrypts secrets using a SOPS binary that a tool, such as Argo CD, can run with the SOPS plugin.

Disadvantages for Encrypted Secrets

At a high level, a developer needs to manually take secrets in plain text form, encrypt them, and store them in Git. However, this creates an opportunity for the identity of the person committing the code to become compromised because the information left behind in Git logs can be easily traced back to the committer. The other downfall of this approach is that if any encrypted keys are compromised, it's difficult to retroactively find and revoke all of them.

It's worth noting that SOPS and SealedSecrets both have problems scaling due to the manual encryption process and the level of human intervention needed. Using a KMS when possible makes managing keys much easier and opens the door to additional options for managing secrets.

Method 2: GitOps Secrets Reference

The secrets reference method uses an external secrets management system to manage your Kubernetes secrets and requires a GitOps operator to deploy secrets to the cluster. A manifest is stored in Git to represent a reference to a secret stored in the aforementioned secret management system. The GitOps operator then deploys the manifest to the Kubernetes cluster. From there the secrets can be fetched from the secrets manager and applied to the cluster as a Kubernetes secret.

This approach sounds effective –  but how do you retrieve the secrets from the manager and get them into your cluster? You'll need an operator installed on your cluster to interact with the secrets manager. The two most popular tools for this in the market are ExternalSecrets and Kubernetes Secrets Store CSI Driver.

ExternalSecrets

The ExternalSecrets project was created by GoDaddy to safely connect secrets stored in an external secrets management solution to a Kubernetes cluster. The ExternalSecrets community continues to add support for additional secret managers making to tool extremely versatile for different toolsets.

In practice, a developer would commit an ExternalSecret custom resource containing a reference to a secret to their Git repository. The GitOps operator will deploy the ExternalSecret to the cluster where the ExternalSecret Operator uses the reference to retrieve the secret from the external key management system. The operator ingests the secret and creates a Kubernetes secret. The ExternalSecret resource is safe to store in a Git repo because it doesn't actually contain confidential information.

Kubernetes Secrets Store CSI Driver

Similar to ExternalSecrets, the Secret Store CSI Driver is a solution to take secrets from an external secrets management tool and bring them into a Kubernetes cluster. The tool natively supports several secret managers but has less support than ExternalSecrets. To bridge this gap, Secrets Store CSI Driver allows third parties to develop their own connections to secret stores. This approach allows for more flexibility but can also require manual effort if the secret management tool used by your company isn't supported natively.

CSI Driver is more complex than  ExternalSecrets. Instead of retrieving external secrets and creating secret resources, this solution uses a separate volume attached to a pod to store secrets. A developer commits a SecretProviderClass with a reference to a secret. The GitOps operator deploys the change and the CSI Secret Store Operator Plugin then retrieves the secret from the secret management system. The operator plugin then creates a volume with the secret that is attached to a specified pod.

Alternative Tools for Secrets Management

Here are some additional solutions to manage secrets with a GitOps approach that are worth noting:

• Hashicorp Vault
• Banzai Cloud Bank-Vaults
• Helm Secrets
• Kustomize secret generator plugins
• aws-secret-operator
• KSOPS
• argocd-vault-plugin
• argocd-vault-replacer

GitOps Secrets Management Made Easier with Harness

Secrets management strategies need to be thought through before deciding to make the switch to a GitOps deployment model. Once a strategy is decided on, setting up safe GitOps practices can be a challenge. Harness GitOps-as-a-Service provides enterprise controls like advanced RBAC, audit trails, and governance rules to keep deployments secure and compliant. 

Harness GitOps GitOps-as-a-Service is simple to implement and use and requires virtually no engineering effort to maintain and troubleshoot. If you're interested in trying Harness GitOps, request a full demo.