Howdy folks! I have some alerts setup against a prometheus datasource in Grafana 9.1.7, and a notification policy that is directing a particular matched label to a PagerDuty contact point using the Events API V2, but I’m not seeing the expected behavior when an alert triggers and an incident is created in Pagerduty.
What I want: I want to be able to Acknowledge the incident in Pagerduty and if subsequent instances of the Grafana Alert fire I want Pagerduty to dedup those into the same incident and NOT notify again. Basically, at this point in my project, I want any and all alerts to just dedup to any active incident until it is resolved.
What actually happens: if I ack the incident, the next grafana alert event triggers a new Pagerduty incident. Eg if I click the grafana contact point “Test” button, PD gets it and creates an incident, I ack it. If I click it again immediately, PD creates a new incident. I tried doing this with a custom label “dedup_key=testing” on the test event and it didn’t make a difference. They are sent with different “incident_key” values which I assume controls this but I don’t see a way to customize that. Documentation on the web seems conflicted about which keys are used by PD.
I don’t see anything in the alert configuration or the contact point configuration that results in a dedup_key or incident_key that is in my control. Is there some way to do this?