The Git Sync Experience in Harness

The advent of DevOps has blurred the line between development and deployments. Engineering teams are now expected to create, maintain, and use the configurations and Continuous Integration and Continuous Delivery (CI/CD) pipelines. The traditional way of creating a configuration in a wizard, and then clicking next, is a big hassle for developers. We prefer to write our configurations the same way that we write our code. The main objective of the Git Sync Experience is to make the CI/CD experience of our users as easy as writing a code.

Using Git Sync, we wanted to create a great developer experience that enables our users to:

1. Store and retrieve their Harness configurations to/from Git.
2. Change the Harness configuration just by changing the YAML files in Git.
3. Easily create the CI/CD pipelines for a newly created microservice by simply adding the pipelines YAML in Git.
4. Maintain their configurations like they maintain code.

Challenges and Solutions

Git Sync

The most important part of the Git experience is allowing bi-directional sync (changes on the UI side should be pushed to Git, and changes on the Git side should be reflected in Harness).

In situations where the same data is stored in two different places, we must decide what place would be the source of truth. We followed the GitOps philosophy here, which keeps Git as the source of truth. This decision will guarantee our customers that whatever they are seeing in Git is the true state, and that they don’t need to visit the UI to confirm it. Whenever a user makes changes in a Git Sync-enabled project, we push the changes to Git first, then we apply the changes in Harness.

The next challenge that we solved was how to push Harness changes to Git. The naive solution which comes to mind is as follows: 

1. Whenever a change is made to Harness, clone the Git repo if it isn’t present.
2. Add the change to the Git repo.
3. Commit and Push the changes.

This approach of cloning the complete Git repo and maintaining it locally wasn’t scalable.

The next solution was to use the APIs provided by the various Git providers. We created an open-source scm service, which is a wrapper over all of the common Git providers. To push a change to Git, we send a request to the SCM service, which pushes the changes to the respective Git provider. Once the changes are pushed to the Git side, we apply those changes on the Harness side.

The next problem is: How do we apply the user’s changes in their Git files to Harness? 

We use the webhooks feature from the Git providers to solve this issue. Whenever a user commits a change to the configuration repository, we receive the webhook request on the Harness side. The request contains details about the changes, which we use to apply the changes to Harness entities.

Multiple Repo Support

There are various patterns that developers follow to store their configuration files. Common patterns include: 

1. Configuration files are stored along with the code repository itself.
2. Configuration files are stored in a repository separate from the code.
3. Store the prod configurations in one repo, and the non-prod ones in another repo, so that only the selected developers can access prod configs.
4. Store the configuration files of different environments in different branches.
5. Store the pipelines in one repository, and the other configuration files in another.

We made sure that our design is flexible enough to support all of the above use cases. 

The user can use multiple Git repositories in the same project. Before pushing each configuration, we ask the user to select the repository where they want to push the changes. This provides the flexibility to the user to maintain their Git repositories in the way that they want.

Harness Folders

All of the Harness entities are synced to the .harness folder in the user’s repository. We process all of the files within the .harness folder and ignore the other files. Inside of the .harness folder, the users can store their YAML files in any folder structure that they want. 

They can store all of the Connectors together in a folder, and the Pipelines in another folder, or they can group the YAML files by project/team. This flexibility lets users store Harness configs either along with their code, or separately in a different repository. The user can select a default folder among all of the folders.

Summary

Git Sync promises a lot of potential for modern DevOps solutions. It provides an easy way to make config changes and helps developers reduce the overall time to set up and maintain CI/CD pipelines.

CI/CD processes in most organizations are tedious and manual. The Ops team ends up maintaining, updating, and running the pipeline. Moreover, a communication mismatch from the Dev team to the Ops team can result in a prod issue. This is where Git Sync can make your life easy: you can create and maintain new pipelines by simply creating/modifying YAML files in Git. 

Keep your eyes peeled on Monday where we'll release a sister article on Git Branching. In the meantime, why not read up on our new GitOps announcement?

Edit: Article on Git Branching is live now!

This article was written in collaboration with Deepak Patankar, Abhinav Singh, Akash Nagarajan, and Rama Tummala.