Optimizing Facebook’s Proxygen Project Build Time with Harness CI: A Case Study

Dhrubajyoti Chakraborty

Updated

Introduction

Open-source projects are a crucial aspect of software development, but their increasing size and complexity can result in longer build times. This can make it difficult to meet software release deadlines and carry out effective testing.

In this article, we will build Facebook’s Proxygen project on Harness CI module and draw a comparison of the build time with Github Actions (Proxygen’s Default CI Tool)

Proxygen: Facebook's C++ HTTP Libraries

Facebook’s Proxygen project is a collection of C++ HTTP libraries including an easy-to-use HTTP server. In addition to HTTP/1.1, Proxygen (rhymes with "oxygen") supports SPDY/3 and SPDY/3.1. Proxygen is not designed to replace Apache or nginx — those projects focus on building extremely flexible HTTP servers written in C that offer good performance but almost overwhelming amounts of configurability. Instead, the focus is on building a high performance C++ HTTP framework with sensible defaults that includes both server and client code and that's easy to integrate into existing applications.

Setting up Proxygen on Harness CI

To set up the Proxygen project on Harness CI, fork the facebook/proxygen repository into your GitHub account and sign up for Harness CI if you haven't already. Once you're in the Continuous Integration module, follow these steps to create your pipeline:

Click on "Create Pipeline" and give your pipeline a name. You can also provide a description and tags if desired.
Choose the "Remote" option for setting up your pipeline and provide your Git connector.
To create a new connector, go to "Connectors" and click on "+ New Connector."
Select "GitHub" as the connector type and name it "proxygen" For the URL type, select "Repository" and provide the GitHub repository URL for your forked repository.
Enter your GitHub username and a personal access token (PAT) as the secret value.
Click "Save" to allow the repository to be fetched, then click on it and select "Apply Selected."
Turn on API access and ensure the secret token is created.
Click on "Connect through Harness Platform" and select your repository (proxygen) and the Git branch will be automatically fetched.
Provide your YAML path as ".harness/{PIPELINE_NAME}.yml", where {PIPELINE_NAME} is the name of your pipeline.
Finally, select "Start" to initiate the pipeline.

It is important to note that the root folder ".harness" is required for the YAML path. For more information on connectors, refer to the Harness CI documentation.

Here’s the YAML to build the proxygen project:

pipeline:
 name: test-proxygen
 identifier: testproxygen
 projectIdentifier: default-project
 orgIdentifier: default
 tags: {}
 properties:
   ci:
     codebase:
       connectorRef: GITHUB_CONNECTOR
       repoName: proxygen
       build: <+input>
 stages:
   - stage:
       name: Build
       identifier: BuildKernel
       type: CI
       spec:
         cloneCodebase: true
         execution:
           steps:
             - step:
                 type: Run
                 name: Fetch Deps
                 identifier: Fetch_Deps
                 spec:
                   shell: Sh
                   command: python3 build/fbcode_builder/getdeps.py fetch --no-tests <+matrix.deps>
                 failureStrategies: []
                 strategy:
                   matrix:
                     deps:
                       - ninja
                       - cmake
                       - zlib
                       - zstd
                       - boost
                       - double-conversion
                       - fmt
                       - gflags
                       - glog
                       - googletest
                       - libevent
                       - lz4
                       - snappy
                       - autoconf
                       - automake
                       - libtool
                       - gperf
                       - libsodium
                       - xz
                       - folly
                       - fizz
                       - mvfst
                       - wangle
                     maxConcurrency: 5
             - step:
                 type: Run
                 name: Build Deps
                 identifier: Build_Deps
                 spec:
                   shell: Sh
                   command: python3 build/fbcode_builder/getdeps.py build --no-tests <+matrix.deps>
                 failureStrategies: []
                 strategy:
                   matrix:
                     deps:
                       - ninja
                       - cmake
                       - zlib
                       - zstd
                       - boost
                       - double-conversion
                       - fmt
                       - gflags
                       - glog
                       - googletest
                       - libevent
                       - lz4
                       - snappy
                       - autoconf
                       - automake
                       - libtool
                       - gperf
                       - libsodium
                       - xz
                       - folly
                       - fizz
                       - mvfst
                       - wangle
                     maxConcurrency: 1
             - step:
                 type: Run
                 name: Build Proxygen
                 identifier: Build_Proxygen
                 spec:
                   shell: Sh
                   command: python3 build/fbcode_builder/getdeps.py build --src-dir=. proxygen  --project-install-prefix proxygen:/usr/local
             - step:
                 type: Run
                 name: Copy artifacts
                 identifier: Copy_artifacts
                 spec:
                   shell: Sh
                   command: python3 build/fbcode_builder/getdeps.py fixup-dyn-deps --strip --src-dir=. proxygen _artifacts/linux  --project-install-prefix proxygen:/usr/local --final-install-prefix /usr/local
             - step:
                 type: Run
                 name: Test Proxygen
                 identifier: Test_Proxygen
                 spec:
                   shell: Sh
                   command: python3 build/fbcode_builder/getdeps.py test --src-dir=. proxygen  --project-install-prefix proxygen:/usr/local
                   reports:
                     type: JUnit
                     spec:
                       paths:
                         - "**/*.xml"
         platform:
           os: Linux
           arch: Amd64
         runtime:
           type: Cloud
           spec: {}

Please modify the connectors in the pipeline YAML to use the ones you created, go to the Triggers tab of the newly created pipeline to confirm the creation of two triggers, including a Pull Request trigger and a Push trigger. In this case, the Pull Request trigger should be enabled.

To compare the build time on GitHub Actions, enable the GitHub Actions workflow from the Actions tab on GitHub (the cloned repository already has the actions workflow file).

Finally, create a Pull Request with a few source or test file changes. This will trigger the Harness CI pipeline, as well as the GitHub Actions workflow if enabled.

Testing & Validating the Pipeline & Build Time

Here are the execution times for a same Pull Request triggered build on GitHub Actions and Harness CI.

GitHub Actions build (avg of 10 runs): 2 hours 10 minutes
Harness CI build (avg of 10 runs): 17 minutes (7x faster)

Understanding the Pipeline Configuration file

The pipeline can consist of several stages, each of which is composed of one or more steps. In this case we have a single stage pipeline with the stage name as "Build" and has 5 steps: "Fetch Deps", "Build Deps", "Build Proxygen", “Copy Artifacts” and “Test Proxygen”.

The "Fetch Deps" step fetches dependencies needed for building the project using the "getdeps.py" script. The "Build Deps" step builds the dependencies, and the "Build Proxygen" step builds the project itself.

The next step i.e "Copy artifacts" copies the built artifacts to a final installation location. The final step is called "Test Proxygen" and runs the tests for the project.

The pipeline is defined to run on a Linux operating system and uses the AMD64 architecture on Harness Cloud infrastructure

This pipeline uses a matrix strategy, which allows for parallel execution of steps with different dependencies. In this pipeline, the matrix strategy is used to define a set of dependencies required for building the "proxygen" project, and to execute the "Fetch Deps" and "Build Deps" steps in parallel for each dependency.

The matrix strategy is implemented over the dependency list, which in this case includes dependencies such as "ninja", "cmake", "zlib", etc.. For each dependency in the list, a parallel execution of the "Fetch Deps" and "Build Deps" steps is initiated. This means that if there are 10 dependencies in the list, then there will be 20 parallel executions of the "Fetch Deps" and "Build Deps" steps, 10 for each step.

By using the matrix strategy the pipeline can make efficient use of resources by executing multiple steps in parallel which can significantly reduce the overall time required to complete the pipeline. This is particularly useful in cases where there are multiple configurations or dependencies that need to be built, as it allows for a more streamlined and efficient execution of the pipeline.

Conclusion

Harness CI is built for speed and offers various parallelism strategies as well as high speed Cloud infrastructure to dramatically improve the build times. Harness also offers several other build and test optimizations to shorten build times (and cost) while also improving developer productivity.