Dec
Automate PagerDuty Incident Postmortems to Drive Improvement
SlackOps for PagerDuty Part 6
View earlier blogs in this series here.
Our final installment of our series on SlackOPs for PagerDuty will cover arguably one of the most important parts, the postmortem. Why is it the most important? Being a hero and solving a single incident is admirable. However, if you’re on vacation when the problem happens again the team won’t be in a good position to resolve incident efficiently. Reviewing the incident and taking action to prevent the problem from occurring again or ensuring that next time it can quickly and reliably be resolved is what will yield transformative results. So the goal is simple, run a postmortem for every incident. However, execution is not. Time and resources are scarce, so it’s easy for postmortems to find their was through the cracks. RigD helps ensure this doesn’t happen to your PagerDuty incident postmortems by enabling you to assign the postmortem from Slack, as well as create and assign tasks related to the postmortem. You can even automate the assignment for greater convenience and reliability.
Step 1 Let’s Assign the PagerDuty Incident Postmortems
Similar to creating a Slack working channel, as discussed in Part 3 you can assign the PagerDuty incident postmortem from several places; after you resolve the incident using RigD, from an incident resolution Feed notification, or using a simple command as illustrated below.
assign incident postmortem
Then choose PagerDuty for the tool and the resolved incident number. Next, assign the postmortem to a specific user or the the incident assignee. Then you will want to indicate when the incident is due. This is crucial to ensure the postmortem gets done! Finally you can have the postmortem reminders sent to a channel as well.
Once the postmortem is assigned you can create a task related to it.
Similar to the postmortem tasks can be assigned to yourself or to someone else and you can specify when it is due. Completing the task is done with a click of the button.
When you indicate when your postmortem is due there will be a reminder halfway to the due time, the it is due, and then periodically once it is past due. As with a task just click the complete postmortem button when done.
There is also a handy command to list postmortems. You can list those for a specific person, or the whole team. All you need to do is type list incident postmortems to initiate.
assign incident postmortem
Step 2 Automate PagerDuty Incident Postmortems Assignment
Why do manually what you can automate? If doing a postmortem is crucial for taking your operations to the next level, then you need a RigD flow to automate the assignment. We’ve set up a guide to help configure that assignment automation in just a few seconds. Start by getting the PagerDuty help.
help with pagerduty
Then choose the Assign Incident Postmortems button
There are just a few options to provide. The first of which is the PagerDuty service you want to auto assign postmortems for.
Next you will incidate that all important due date. Finally you can optionally add an additional channel to deliver the reminders messages.
That’s it, you are now set up to drive a better incident response process each and every time. You can give it a try just resolve an incident in the selected PagerDuty service.
In our prior segments we have computed the time and monetary savings of driving the PagerDuty incident response process through Slack with RigD. It’s hard to quantify the savings from running PagerDuty incident postmortems. Those savings are generally realized in future incidents or in the avoidance of them. One thing we can do is compare the time to make this assignment in RigD vs the PagerDuty UI. With RigD the process is fully automated and completes in just about 3 seconds after an incident is resolves. Doing this manually using PagerDuty takes about 22 seconds. In PagerDuty’s recent ROI study the enterprises averaged 20,483 incidents annually. With that many incidents you could be saving 108 hours over the year. That’s before considering any savings from performing the postmortems.
No Comments