Harness Extends Continuous Verification To Datadog

Authors:

No items found.

Table of Contents

These days, it's not uncommon for customers to use Cloud or infrastructure monitoring software across their dev, QA and production environments.

Back in February, we announced support for Dynatrace to accompany our AppDynamics and New Relic APM support. Today, our Datadog support is now GA.

Why Harness + Datadog?

Harness helps customers master Continuous Delivery (CD) so they can deploy new applications and services into production with speed and confidence.

Datadog helps customers monitor the performance of their cloud applications and infrastructure.

Customers typically have one or more deployment pipelines for each application/service. A typical pipeline has several stages that reflect the environments (e.g. dev, QA, staging, production) which are used to test/validate code before production deployment occurs.

Integrating Harness with Datadog now allows customers to automatically verify cloud application and infrastructure performance across their deployment pipelines.

A Simple Deployment Pipeline Example

Let's imagine we have a Docker microservice called 'My Microservice' and our Continuous Integration (CI) tool Jenkins creates a new build (#101) which results in a new artifact version for us.

With Harness, we could pick up that new version and immediately trigger a 4-stage deployment pipeline like this:

Example deployment pipeline

Each pipeline stage deploys our new microservice to a different environment, runs a few tests, verifies everything is good before proceeding to the next stage.

You can see from the above screenshot that stages 1 thru 3 (dev/QA) succeeded and turned green. This is generally good, it means the artifact passed all tests and is ready for production and customers.

The bad news it that stage 4: production turned red and our pipeline status is 'failed'.

Automating Deployment Health Checks With Harness

At Harness, we take CD one step further with something called Continuous Verification.

One of our early customers Build.com used to verify production deployments with 5-6 team leads manually analyzing monitoring data and log files. This process took each team lead 60 minutes and occurred 3 times a week. That's 1,080 minutes or 18 hours of team lead time spent on verification. With Harness, Build.com reduced verification time to just 15 minutes, and also enabled automatic rollback to occur in production.

With Datadog, Harness is able to deploy and verify the performance of cloud application deployments instantly in every environment. The second a new application or service artifact is deployed, Harness will automatically connect to Datadog and start analyzing the application/service/infrastructure performance data to understand the real business impact of each deployment.

Harness applies unsupervised machine learning (Hidden Markov models & Symbolic Aggregate Representation) to understand whether performance deviated for key services and flags performance regressions accordingly.

Let's take a look at what this verification looks like with Datadog.

Continuous Verification With Datadog

The below deployment workflow relates to the above-failed pipeline, and specifically stage 4 in production.
You can see that phase 1 of the deployment failed. The deployment in terms of deploying the service succeeded, as did the verifications and tests relating to Jenkins. However, the Datadog verification failed which means new performance regressions have been identified. The resulting action, in this case, was an automatic rollback (this is the safety net that Harness provides).

Clicking on the Datadog failed step (red) shows us why the verification failed:

We can see that one key web transaction “get_/todolist/inside/load” has a request duration regression after our new microservice version was deployed. Mousing over the red dot we can get more details – response time actually increased 58% from 840ms to 1330ms post-deployment:

The default time period for this verification was 15-minutes but this number can be custom. Some deployment pipelines and workflows can take hours to complete because not everything is automated (manual checks/approvals etc.).

You can also configure/filter any Datadog metrics by clicking the Metrics Filter:

Configuring Datadog Continuous Verification

Our integration was pretty simple to build using the standard Datadog API.

The first thing you need to do is Setup > Connectors > 'Add New Verification Provider':

Then enter your Datadog URL, API Key, and Application Key:

You can get your API Keys from your Datadog Integrations APIs screen:

Adding Datadog Verification To Deployment Workflows

First, add or edit an existing workflow in Harness.

Simply click 'Add Verification' and select Datadog from the menu.

Next, select your Datadog server, enter the service name you want to verify, select the metrics you wish to verify, and enter a verification time period (default is 15 minutes).