Elevating Pipeline Visibility: Harness Pipeline Notifications for Datadog

Introduction

In the world of software delivery DevOps, every second counts. Delayed notifications about pipeline failures can cost your team valuable time and resources. Imagine having real-time visibility into your CI/CD pipeline directly in the monitoring platform of your choice. That's exactly what Harness Pipeline Notifications for Datadog delivers.

In today's fast-paced software development landscape, maintaining visibility into your CI/CD pipelines is crucial for ensuring smooth operations and quick issue resolution. At Harness, we understand the challenges DevOps teams face in monitoring complex pipeline processes. While we have long offered pipeline event notifications through popular communication and incident management platforms like Email, Microsoft Teams, Slack, PagerDuty, and Webhook, we recognized the need for even deeper integration with observability tools.

That's why we are thrilled to introduce our latest feature: Harness Pipeline Notifications via Datadog. This powerful integration bridges the gap between your CI/CD processes and your monitoring infrastructure, empowering teams to respond faster, collaborate more effectively, and maintain a clear view of their software delivery process.

Need for enhanced Pipeline visibility

As CI/CD pipelines grow in complexity, the need for real-time monitoring and alerts becomes increasingly critical. DevOps teams often struggle with:

• Delayed awareness of pipeline failures
• Difficulty in pinpointing the exact stage or step where issues occur
• Lack of integration between pipeline events and existing monitoring tools

These challenges can lead to increased downtime, slower issue resolution, and frustrated development teams.

Harness Pipeline Notifications for Datadog

Our new Harness Pipeline Notifications for Datadog feature addresses these pain points by seamlessly integrating pipeline events into your existing Datadog workflows. This integration offers:

• Real-time visibility into pipeline events
• Configurable events to get notified about such as specific pipeline stages and steps
• Custom headers for enhanced event categorization
• Native integration with Datadog with enhanced events search and analytics capabilities

Harness Notifications integration in Datadog

The Harness Notifications Datadog integration enables you to tag the Pipeline notifications ingested to Datadog with source:harness_notifications. If the integration is not installed, every notification is tagged with source:my_apps. Learn more about this in Datadog official documentation. Additionally, through the integration, you get access to an in-built dashboard that gives you a time-series view of the types of events and their occurrence frequency. Please note that installing this integration is not required to send Pipeline notifications from Harness to Datadog. However, it provides you with some additional capabilities as explained above.

Set Up Pipeline Notifications for Datadog

Prerequisites

1. Create a pipeline in Harness. Refer how to create a sample CD pipeline. 
2. Ensure the feature flag PIPE_DATADOG_NOTIFICATIONS is enabled. Reach out to Harness Support to enable this feature.
3. Create a API key or use an existing one from Datadog. This will be required to post Pipeline events to Datadog using the Events API (v1).
4. Datadog instance URL for your account (e.g., https://app.datadoghq.com or your region-specific endpoint).

Add a new notification rule with Datadog channel

1. Create a notification rule
   
   1. In Harness Pipeline Studio, click on Notify in the right sidebar.
   2. To create a new notification rule, click on +Notifications button. The Notification Rule settings appear.
   3. Enter a name for the Notification rule. Example: Datadog notifications
   4. Click Continue.
2. Select Pipeline events: You can either select All events or configure the events that need to be sent. Supported pipeline events include: Pipeline Start, Pipeline End, Pipeline Success, Pipeline Failed, Stage Start, Stage Success, Stage Failed, and Step Failed. 
   
   1. Select events for the pipeline or stages. If you select the stage events, you can select which stages to use.
   2. Click Continue.
3. Choose Datadog (/v1/events API) as the Notification Method
   
   1. Under Channel Type, select Datadog (/v1/events API).
   2. Provide the Datadog URL followed by /api/v1/events eg. https://app.datadoghq.com/api/v1/events/
   3. Enter the Datadog API key.
   4. Add headers: You can add custom headers to events for enhanced filtering and categorization in Datadog.
4. Click on Finish.

The notification rule is saved and is by default enabled. Once the Pipeline is run, the configured Pipeline events get ingested into Datadog. 

Install Harness Notifications integration in Datadog (optional)

Users with Administrator permissions in Datadog can configure integration settings for the entire instance or for specific projects or groups.

1. In your Datadog account, click on Integrations in the left sidebar.
2. On the Integrations page, search for Harness Notifications.
3. Hover over the Harness Notifications integration card and click on Install.

Now the Pipeline events notifications will be tagged with source:harness_notifications.

Use Cases and Benefits

1. Leverage Datadog's dashboards to visualize pipeline performance
2. Use custom headers to categorize events for easier filtering in Datadog
3. Create alerts in Datadog for unhealthy trends in Harness pipelines based on ingested notifications. For example, an engineer can set up alerts for failed stages in critical pipelines, enabling rapid response to issues before they impact production.

Example: A sample pipeline

To demonstrate the power of Harness Pipeline Notifications for Datadog, let's walk through a real-world scenario using a sample pipeline. This example will showcase how the integration provides critical visibility and enables rapid response to issues.

Our Sample Pipeline

We'll use a straightforward yet realistic pipeline consisting of two key stages:

1. Approval Stage: A manual gate requiring sign-off before deployment.
2. Deploy Stage: Automated deployment to a production environment.

‍

‍

Setting up Notifications

For this pipeline, we've configured a notification rule to send all pipeline events to Datadog. This includes:

• Pipeline start/end
• Stage start/end
• Step success/failure
• Overall pipeline success/failure

‍

Scenario 1: Successful Deployment

When everything goes smoothly, here's what you'll see in Datadog:

1. Pipeline Start Event: Marks the beginning of the process.
2. Approval Stage Start: Notifies relevant team members that approval is needed.
3. Approval Stage End: Confirms the deployment has been approved.
4. Deploy Stage Start: Indicates the deployment process has begun.
5. Deploy Stage End: Signals successful completion of deployment.
6. Pipeline End Event: Confirms the entire process has completed successfully.

‍

These events provide a clear timeline of the deployment process, allowing teams to track progress and confirm successful execution.

Scenario 2: Deployment Failure

Now, let's consider what happens when something goes wrong during the Deploy stage:

1. Pipeline Start Event: Process begins normally.
2. Approval Stage Events: Proceed as expected.
3. Deploy Stage Start: Deployment begins.
4. Rollout Deployment Step Failure:
   
   • Datadog receives an immediate notification of the failure.
   • The event includes details such as the specific step, error message, and relevant metadata.
5. Deploy Stage Failure: Triggered as a result of the step failure.
6. Pipeline Failure Event: Indicates the overall pipeline has failed.

In Datadog, these events would appear like this:

Once the events are sent to Datadog, you can then create a case or declare an incident which can then be assigned to a user or on-call engineer based on your configured workflow.

Leveraging Datadog's Capabilities

With these events in Datadog, your team can:

1. Create Targeted Alerts: Set up notifications for critical failures, ensuring the right people are informed immediately.
2. Build Informative Dashboards: Visualize pipeline health for a holistic view.
3. Perform Detailed Analysis: Use Datadog's analytics to identify patterns, such as frequent failures in specific stages or steps.
4. Streamline Incident Response: Quickly create incidents from pipeline failure events, automatically assigning them to the appropriate team or on-call engineer.

Conclusion

Harness Pipeline Notifications for Datadog bridges the gap between your CI/CD processes and your monitoring infrastructure. By providing real-time pipeline event notifications directly in Datadog, we're empowering teams to respond faster, collaborate more effectively, and maintain a clear view of their software delivery process.

Ready to take your pipeline visibility to the next level? Enable Harness Pipeline Notifications for Datadog today and experience the difference. 

Get started now:

1. Log into your Harness account
2. Enable the PIPE_DATADOG_NOTIFICATIONS feature flag
3. Set up your first Datadog notification rule in your pipeline

Questions or feedback? We'd love to hear from you! Contact our support team or visit the Harness Developer Hub for more information.