Harness CI Enterprise for UI Builds
This post walks you through the process of creating a Harness CI Enterprise (CIE) Pipeline, as well as how we use it to create our own pipeline for UI Builds. We will also cover the caching strategy used, which ensures faster job execution.
What Is CI?
Before starting with Harness CIE, let's understand what CI is.
Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project.The integration process can be automated via some specific tools that use scripts and processes as frequently as needed. This means that the compatibility of the new code is being tested automatically. Moreover, this supports the integration errors being found earlier. In addition, unit tests or system level tests can be done automatically without any developer intervention. For further information, please review our What Is Continuous Integration? post.
Why Harness CIE?
When it comes to CI, numerous tools are available, but the main drawback is that these tools involve quite a lot of manual scripting, and minimal integration exists between CI and CD. One popular example of these tools in the CI space is Jenkins.
Harness provides some great features, such as Test Intelligence, which reduces the test time significantly by running only the tests required to confirm the quality of the code changes that triggered the build.
At Harness, we use Kubernetes as a Build farm to run our CI Build jobs. In Kubernetes, pipeline steps are run in containers, which are lightweight abstractions of the host operating system that can package code and dependencies independently of the steps. Furthermore, we don’t need to worry about dependencies because all of the steps run in containers, and the plugins have their own containers.
Harness CIE also provides a graphical representation of the Pipeline with nested steps. Build directly from the UI, or use the YAML editor if you prefer. The YAML editor functions similarly to an IDE, and you can easily configure it as code.
How We Leverage Harness CIE to Create Our Pipeline for UI Builds
Four simple steps create a CIE Pipeline and execute it:
- Create a new Pipeline and clone the codebase while creating a Pipeline stage
- Define the build farm infrastructure
- Add execution steps
- Run the Pipeline
‘Create a Pipeline’ in the CI module takes the user to the Pipeline studio, where we then add stages. We can create multiple Stages by combining various CI Steps. Each Stage includes Steps for building, pushing, and testing your code.
Therefore, either we can create multiple stages in parallel within a Pipeline for the PR checks, or we can create individual pipelines for each PR check.
The main prerequisite for codebase cloning is to install a Harness Delegate. The Delegate is in charge of all of the Harness CIE operations, and it connects the Harness Manager to all of your code repositories, artifacts, infrastructure, and cloud providers.
The main action items for cloning a codebase are:
- Creating the GitHub Connector
- Connecting Harness to the GitHub repo with the codebase
For defining build farm infrastructure, we must select a Kubernetes Cluster (or create a Kubernetes Connector). This Connector connects Harness to the cluster to use as the build farm.
In Namespace, we must enter the Kubernetes namespace to use.
A Pipeline Execution Step is defined as a series of commands used to carry out an action. For example, the Run and Run Tests step executes one or more commands on a container image. The commands are run within your git repository's root directory. All of the steps in your pipeline stage share your Git repository root, also known as the Workspace. In Harness CI, we can add a Step or a combination of Steps to create a Pipeline tailored to your specific requirements.
Caching Strategy
We use the caching mechanism in Harness while configuring the Pipeline for UI Build PR checks. Caching ensures faster job execution by reusing the expensive fetch operation data from previous jobs.
In a Harness CIE Pipeline, we can save the cache to an Amazon Simple Storage Service (S3) bucket in one stage using the Save Cache to S3 step, and restore it in the same or another stage using the Restore Cache from S3 step.
Harness also provides steps to Save Cache to GCS and Restore Cache from GCS.
The execution steps we use in configuring our PR check Pipeline are executed in the following order:
1- Get Hash: For getting hash, we use the Run execution step, where we can add custom scripts. For this example, I am using the node:16 image from my docker-hub registry to run the custom script for the step. We get the yarn lock hash using `openssl sha1 yarn.lock | cut -c 22-` and expose the hash using the output variable. This lets us store the cache.
2- Restore Cache: The next step is to restore the cache. This step is added before saving the cache or installing the node module step to check whether the cache already exists, and to restore it from there. For this, we just need an Amazon S3 or GCS bucket to store the cache. We save the cache against a key. The key we are using in the example to store the cache is: nodeModules-${outputVariable}, where outputVariable is the variable we used in the previous step to expose YARN_LOCK_HASH.
3- Install Node Modules: The next step is to install node modules. We will add scripts to connect Harness CIE to the npm registry. Then, add the yarn script to install the node modules.
4- Save Cache: After installing node modules, we must save the files/folders in the cache. This means that it can be restored from the cache in the next execution. While configuring the Save Cache step, we should make sure that the Amazon S3/GCS bucket, region, and key to save cache against should be the same as that of the Restore Cache step. The source path, as present in the attached image, is the list of the files/folders to cache. In our case, it is node_modules.
5- Run Step to Execute the Build: The final execution step in configuring our UI build for PR Check is to add the Run step. Here, we can add custom scripts to execute a particular PR Check, such as ES Lint, Prettier, or Jest. We have again added scripts to connect Harness CIE to the npm registry, followed by the script to run the ESLint job using yarn lint.
Note that, while configuring steps, even if Restore from Cache steps fails, the pipeline should not fail, which is the Harness Pipeline’s default behavior.
To achieve this, Always execute this step must be checked in the Install node modules step (in the Advanced tab), and the <+execution.steps.Install_Node_Modules.status> == "SUCCEEDED" condition should be added in the steps following Install node modules (as depicted in the below image).
View Your Pipeline Execution in Action
After we are done with the configuration part, the only step remaining is to Save and Run the Pipeline. We can view the result in the execution stage details or the console view.
Performance Matters
In the previous sections, we discussed that caching improves the performance of the pipeline execution. Let's try to understand the same thing using the following examples.
Scenario 1
In this scenario, we don’t restore the file from the cache, and instead install the entire node modules. This takes 5m 2s to complete the pipeline execution.
Scenario 2
In this scenario, the cache was present in the bucket and we restored the node modules from the cache. This execution took only 1m 25s.
This is just an example of a very simple pipeline. However, for a pipeline with lots of stages and steps, the difference between the caching and non-caching strategy would be clearly noticeable.
The application of this caching strategy to the jobs running for Harness private repository saved us approximately ten minutes for our UI builds. In my opinion, this is a great success metric.
Using the caching strategy also lets us save network bandwidth. We run hundreds of CIE jobs on a daily basis at Harness. Therefore, if we have to download compressed node modules of an average size of 1 GB for each job over the network, then imagine the amount of network bandwidth consumed for this task. In addition, usually our node modules don’t get updated. In turn, the caching strategy saves additional network bandwidth.
Conclusion
Thank you for taking the time to read this post and understand how Harness Pipelines are created for the UI builds. I am ending this article with the high hope that you can now set up your own Harness CIE Pipeline.
If you want to learn more about CI, please read how we migrated from Jenkins to Harness CIE and try it yourself. To learn more about Harness CIE, see the Harness CIE docs!