Setting Up PagerDuty Notifications for Incident Management Using Harness

Setting Up PagerDuty Notifications for Incident Management Using Harness

Every organization today has a team of engineers working hard to keep their systems up and running, and they must create processes for handling incidents that inevitably arise. Even if you have the best developers and site reliability engineers (SREs), managing and addressing these incidents will take time and effort. However, as with everything, incident response is much easier to manage if you have the right tools and strategy. A mix of best-of-the-breed continuous integration and continuous delivery (CI/CD) platforms integrated with the best incident management tool is what every organization needs.

If you are an IT pro who feels like your team is constantly struggling to catch up with alerts and incidents, you may feel like incident management is the primary friction point for your team. This article will demonstrate how to integrate PagerDuty, an incident management tool, with Harness to get a clear view of your CI/CD pipeline events. 

What is Incident Management? 

Incident management is a process of identifying, investigating, and resolving incidents affecting a service's availability. The incident management process begins with an outage or event that triggers an alert from monitoring tools or from manual detection by operations staff. The incident manager then assesses the impact on customers and the business before deciding how to respond based on the severity of the problem, who is impacted, and how long it will take to resolve it. Incident management is a key component of DevOps. It encompasses the steps that are taken to respond to incidents and reduce their impact on business operations.

The CI/CD pipeline is a series of steps that are taken to manage the software development process. It includes automated testing, CI, and deployment. The pipeline is meant to shorten the time it takes for code to be developed and deployed into production. Having incident management configured somewhere in the CI/CD pipeline completes your pipeline as it helps you monitor incidents and get notified immediately. 

Why Incident Management?

Incident management is important because it helps organizations maintain the availability, reliability, and security of their systems and services. By quickly identifying and resolving incidents, organizations can proactively minimize downtime from incidents and ensure that they continue to meet the needs and expectations of their customers. Incident management is an integral part of a successful DevOps strategy as it helps to ensure that any issues or incidents are resolved in a timely, efficient, and cost-effective manner. 

Incident management helps not only the developers but also the security team to work with enhanced security and compliance to fill any security gaps. Furthermore, it monitors applications and services for any errors or shutdowns. The incident management tool will alert the responsible team members if something goes down. Then, with the set of policies decided by the team, the application or services in question will be restored immediately. Therefore, an incident management tool is essential in your CI/CD pipeline. 

Incident Management Strategies

A successful incident management strategy should involve automation, collaboration, and communication tools. In addition, incident management consists of responding to, resolving, and documenting any service disruption or failure as quickly and efficiently as possible. Incidents can range from minor software glitches to major system outages, and a well thought out strategy is necessary to ensure the stability and security of the system. 

Implementing the right strategy for incident management can help DevOps teams stay on top of issues and ensure service continuity. DevOps teams can quickly and efficiently respond to any incident, ensuring the system remains stable and secure. With incident management in place, the engineering organization can save a lot of money and avoid some stressful situations.

To give an example of what an incident management strategy may look like, here are four high-level steps:

1. Monitor the CI/CD pipeline to identify any issues that may arise. When an issue is detected, an alert should be generated, so the relevant team members are informed. 
2. Once an alert is received, triage the issue to determine its severity and impact. If the issue is deemed severe, it should be escalated to the appropriate team for further investigation and resolution. 
3. Resolve the issue as quickly as possible and document the successful implementation step by step. 
4. A follow-up meeting is needed on this incident to ensure everything is resolved and the systems are restored to perform as expected. 

This is a simplified example of what an incident management strategy may look like. In reality, you’ll want to tailor your approach based on your service level agreements (SLAs), service level objectives (SLOs), service level indicators (SLIs), and error budgets.

What is PagerDuty?

PagerDuty is a cloud-based IT incident management tool that helps notify and coordinate responses to incidents across an organization. Here are some of the features and advantages of PagerDuty:

• Helps determine the urgency of an incident and assigns it to the engineers accordingly, so they can take appropriate action
• Provides live notifications when there is an update on an incident
• Allows users to assign tasks and track their progress
• Enables notifications for specific incidents

Tutorial: How to Send PagerDuty Alerts Using Harness

In this tutorial, we’ll demonstrate how to send PagerDuty alerts in case incidents are detected during a pipeline execution. 

When a developer commits a code to a source control management (SCM) system, like GitHub, the CI/CD tool will build and test the code for any misconfigurations, bugs, or vulnerabilities. The code then gets deployed to a target environment, such as Kubernetes. Here, we will configure the alert notification so that when an unsuccessful deployment happens, the pipeline will notify PagerDuty to alert the developers and the team responsible for taking further actions. 

Harness is the heart of this whole setup, from cloning the code to continuous integration to deploying it to the target environment and alerting the responsible people. 

Tutorial Prerequisites:

• Harness free account
• PagerDuty free account
• A sample application to connect with the Harness platform (you can fork my repo and use the same)
• A Kubernetes cluster from any cloud provider to deploy our application (you can use Minikube or Kind to create a single node cluster)

‍Note: Built-in notifications are a capability of the Harness platform, and as such, they can be applied to any stage types.

Harness Setup

If you are new to Harness, you need to have a free Harness account. 

Next, sign in to your Harness account and select the CD module. Set up a simple pipeline to deploy your application. We will use a Kubernetes cluster to deploy our sample application. 

‍

Add the required connectors for your project, such as your GitHub repository (we use a sample application; you can fork and use it) and a Delegate. The Harness Delegate is a service you run in your local network or VPC to connect all of your artifact, infrastructure, collaboration, verification, and other providers with Harness.

Once you set up your project and the pipeline to deploy your application, it is time to set up notifications. On the right-hand side of the dashboard, you will see a notify icon. Click the icon and set up your incident notification strategy. 

As we discussed above, I’ll show you how you can integrate the PagerDuty notifications.

Add notifications.

The setup will ask you what event you would like to receive notifications about. I am simply setting up notifications for any deployment, i.e., for any successful deployment of the application/service, notifications will be generated.

Next, we will pick PagerDuty as the notification method.

Once you pick PagerDuty as your notifications tool, you are required to provide the PagerDuty Key in the next step.

To get the PagerDuty key, you need to access your PagerDuty account.

Setting Up PagerDuty

Create an account on PagerDuty and provide a service name to continue.

In the next step, you’ll set up integrations. Go to the “All” tab and search for Harness, and select Harness as shown below.

The next step will ask you how you would like to get notified. I put my email address. See the example email below sent by PagerDuty.

Next, go to Service and click Service Directory to see your services configured. You should see the added service by you. 

Click on Manage and then on View Integration to see your key.

The integration key is what we need. Copy it and save it.

Adding PagerDuty to Harness

Back in the Harness app, add the key to your PagerDuty notification tab.

Test if your connection is successful.

Save the changes and apply them. Then, go to your Harness dashboard, and run the pipeline. 

On the successful deployment of our application, we should now see the event notification on PagerDuty and receive an email response.

‍

Similarly, you can customize your workflow and events, assign people to look into the incidents, and more. The PagerDuty integration can help your support and engineering team keep track of what is happening to resolve issues as quickly as possible. 

Incident Management is Critical for CI/CD

Once you deploy your service, you need to know what is happening and report different events to your team. Having an incident management tool is an important addition to any CI/CD platform to ensure everything is going as expected. PagerDuty is an excellent example of one of the many notification tools that can be integrated with Harness, taking your incident management to the next level.

For more Harness product tutorials, check out the Harness Developer Hub, learn more about Harness CD features, or try it for free today.