This post will be the first in a series on Verified GitOps. We will start with an overview of GitOps, the benefits, and the drawbacks. Next, we will take a deeper dive into what GitOps actually looks like in practice. Lastly, we will have a longer post which goes over what Verified GitOps is and how it looks in practice.

GitOps: Overview

Containerization! Cloud Native! CI/CD! Other marketing buzzwords!

It’s amazing how most technological concepts start and grow, starting with engineers, eventually becoming a best practice, leading to marketing and sales buzzwords, and then management enforces the best practices on their teams, which then leads to other technological concepts to solve issues or gaps that exist. A vicious cycle that defines how OOP became popular, functional programming, cloud computing, virtualization, and many others.

One technological concept which was practiced before it was officially termed is GitOps. The idea of keeping a codified version of something in a source code management solution using git has been a part of Application Development for a very long time. Eventually, users began leveraging declarative languages to represent the infrastructure in code in order to make provisioning infrastructure more scalable, repeatable, and reliable. What this has lead to, which is where the terminology came into the market, is when WeaveWorks created a tool that allows you to leverage the declarative application code for Kubernetes (Kubernetes Specs, Helm, Kustomize, KSonnet) and ensure that what is in the code is accurately represented in the cluster. This leads to some major benefits for the SDLC process, specifically allowing the user to quickly spin up a new cluster in a Disaster Recovery scenario.

GitOps: Breakdown

Let’s quickly breakdown what GitOps looks like currently:

The whole process starts with the Engineer that makes a change to their application or infrastructure specification code, which then syncs to their git repo through a commit (Pull Request), then, depending on the method of GitOps Sync, either a webhook pushes the changes, or an agent/operator pulls the changes to the application or infrastructure. This allows for the git repo to be the source of truth, and the application/infrastructure is a reflection of the git repo.

GitOps: Benefits

There are some significant benefits to GitOps, especially over what came before:

  1. Leverage Developer Native Tooling: Because developers are already using Git for their application code, and maybe even Infrastructure code, allowing them to handle deployment specifications in their git repo makes it easy to train and use.
  2. Automated Recovery: Because the infrastructure and application specification code is stored in a git repo, if a cluster goes down, everything can be spun up very quickly because it already exists in code.
  3. Credential Management: Since engineers are already required to authenticate into their git repo, that authentication and authorization can be leveraged for deployment as well.
  4. Git Blame and Versioning: If the declarative code for application configurations and infrastructure is stored in git, then you have native versioning of the code, which allows for reverting as needed, and also the ability to find out who made the changes.
  5. Easier Knowledge Transfer: Since everything exists in a directory structure for git, it is easy to see how things relate to other applications or files.

GitOps: Issues

As great as this process sounds, there are some inherent flaws:

  1. Auto-deploying to the infrastructure or application the moment that the git repo is updated is not a good process for any environment above Development.
    • A solution for this is making an air-gap between each repo and manually pushing the change up the repo chain when it is approved
  2. By defaulting to Helm/Kustomize/Kubernetes Specs and letting those files dictate what is deployed and how, it is near-impossible to add things like pauses, approvals, testing, etc. in the process since there is no way of representing those requirements natively
    • A solution for this is to set up the air-gap model mentioned before which allows you to move from one environment deployment to the other by implementing a manual pause.
  3. When you get to the point of setting up the git repo to best integrate with a GitOps process, the use of a tool for GitOps is almost pointless, other than getting a visual understanding of what is being deployed and its current state.
    • Additional tooling might represent a vertical definition of GitOps.
  4. Advanced technological understanding of the Application, Infrastructure, and Architecture is required to make sure that the git repo is ready for GitOps processes, which alienates anyone who is not well versed in the underlying technology
    • The only solution for this is to bring in more people with advanced knowledge and build out tribal knowledge internally
  5. Current GitOps oriented tooling only supports Kubernetes
    • If the company is using anything else (serverless, other container orchestration tools, traditional applications, etc.) then they need to set up multiple tools and processes.
  6. Automated Rollbacks result in Configuration Drift
    • If the solution being used offers automated rollbacks, then the configuration in the infrastructure or application will be the previous version, compared to the git repo. The potential workaround is to not have automated rollbacks, but rather that a notification is sent when there is an issue, requiring an end-user to revert the changes in the git repo, which will then sync the changes down.

GitOps: Summary

Being able to codify the infrastructure and application specifications can be very useful. The ability to have repeatable configurations without the need for tribal knowledge is not just helpful in GitOps, it is one of the main reasons why technologies like Kubernetes are so popular and heavily used!

Onboarding new people also becomes easy, since the configuration code is declarative and easily readable, allowing those with access to the Git repo to easily see what the environment is intended to look like.

Yet, even with these benefits, there are significant drawbacks to GitOps which make it hard to implement properly at an enterprise scale. In the next post, we will talk about what GitOps looks like in practice. Stay Tuned!

Keep Reading

  • What is CI/CD? Everything You Need To Know

    What is CI/CD? Everything You Need To Know

    This blog post will share everything you need to know about CI/CD, from code changes to continuous integration, continuous delivery, deployments, and CI/CD pipelines.
  • Modernizing Jenkins CI/CD Pipelines

    Modernizing Jenkins CI/CD Pipelines

    As paradigms start to shift and with the rise of Continuous Delivery, leveraging Jenkins for a complete CI/CD pipeline can be seen as a “DevOps Duct Tape”. In a recent webinar with, a very knowledgeable pair of architects weigh in on where Jenkins fits in their respective ecosystems today.
  • Intro to Deployment Strategies: Blue-Green, Canary, and More

    Intro to Deployment Strategies: Blue-Green, Canary, and More

    Deployment Strategies For Continuous Delivery. Which Deployment Strategy Should You Use: Basic, Multi-Service, Rolling, Blue/Green, Canary, A/B Testing?