SlackOps for PagerDuty Part 3

View earlier blogs in this series here.

For our third installment of our series to help you master your incident management, we will help you automate creation of Slack channels for your PagerDuty incidents. This is a key element of PagerDuty Slack integration. Following this practice of having a dedicated channel for each incident is a great idea, though some customers like to use a “War room” or “service channel” approach. There are two key things to consider when choosing your approach.

First, When you have more than one issue going on, you will have to fight cross talk if you do not use a dedicated channel. This can lead to distractions for key resources, missing important messages, and general confusion for those who are jumping in late trying to get up to speed. One way to mitigate this is to use a Slack thread for the discussion around the incident.

Second, having more than the required people engaged in an incident in the channel can lead to “unhelpful” commentary, distract non-essential people from the work they should be doing, and potentially lead to “unmanaged” updates getting out. A dedicated channel can ensure this is avoided, while a thread approach might not.

Creating a dedicated slack channel and bringing in the right people at the right time mitigates these risks and offers a the best change for a complete and clear record of the incident activity and who worked engaged in resolving it. This is of course an essential part of a good postmortem. However, you should ultimately choose an approach that works best for your business. So lets jump in and show you how to make this task a simple thing with RigD.

Step 1 It’s easy to open a PagerDuty Slack Channel Manually

We provide two ways to open a dedicated PagerDuty incident channel or thread. The first is through a RigD activity. You can initiaite that by typing

start work

Then choose PagerDuty as the work tool and provide the incident number.

The Slack channel or thread will be created with the tool type and incident number in the name to make it easy to find. To be helpful we also add a note to the incident with a deep link to the Slack channel or thread. Additionally, we add any incident assignees or responders to the channel so they can start to engage right away.

The second way to create the incident channel or thread is from an incident notification if you set up a Service Feed through RigD. Here you simply click the start work button and choose channel or thread.

Step 2 Use RigD Automation to Open the PagerDuty Incident Slack Channel

Using those manual options for opening the incident channel or thread are not difficult at all and they can work for most. However, there may some circumstances where you or the on call person need to get the team engaged and working the incident before you can get into Slack. To set up this automation we are going to first establish a RigD feed which gets incident events from PagerDuty and delivers them in Slack. Second we are going to use another flow to run our start work activity, which creates the channel. Then we are going to create a RigD trigger that will start the flow based on seeing a specific element in the incident event. If it sounds a little complicated don’t worry, there is a setup guide that will do all the heavy lifting. Start from the PagerDuty help by typing

help with pagerduty

Then choose the Open An Incident Slack Channel button

Now you have thee pieces of information to provide to complete the setup. The first is the name of the PagerDuty service you want to open channels for.

Next provide the type of event to use to create the channel or thread. This guide supports just a simple scenario of opened and acknowledged, you may have other more complex cases you want to explore down the road, such as acknowledged and with a P1 priority.

Now we need a channel for posting the notification that the incident channel has been created. Typically this is the same channel where you have your regular incident notifications sent to.

Finally, the big decision. To use a channel or a thread. What ever you decide we have got you covered.

Congrats you’ve just made your incident management process significantly better! Your Feed is set up and new incident events will create a Slack Channel or thread. Give it a try.

Once the incident is opened and then acknowledged the channel is created. The notifications for the incident from the Feed will now provide a button to jump right to that channel. The notification channel will also have a message with the channel details.

When you are all done working on the incident you can easily archive the channel by typing the following in the channel. This also let’s RigD know that the incident is done and to generate a digest of the incident activity.

stop work

There were many benefits mentioned earlier in regards to opening a Slack channel or thread per incident. So assuming we want to do that let’s look at the benefit of using RigD vs doing it manually, as we did with our other series parts. Opening the channel through RigD takes just 4 seconds as compared to about 20 seconds when creating the channel manually. As with our previous two examples it’s not a massive amount of time on it’s own, but it adds up. We will use the figures from the Rand Group report ,$5,600 cost per minute of downtime for an enterprise, and 20,483 incidents and 14 outages from the PagerDuty ROI study. This results in $20,907 in outage avoidance and 91 hours of time back.

Keep an eye out for the next part of our series which covers automating the incident update process. Also please do share your comments or questions so we can keep making incident response easier and faster.

Learn more about RigD here, and give our Slack App a try.

Next Up in this Series: Part 4 – PagerDuty Incident Status in Slack