I used the PagerDuty integration Key for alerting. I run grafana in a pod installed via helm on k8s on GKE.
I used the send notification test, and the test went through successfully to PagerDuty Alerts page one time.
After that, I tried redoing it again about an hour later.
I would press “Send Test” and the test would say sent.
The logs in the grafana container of the grafana pod produced this output associated, one each with both the successful test that showed up in pagerduty alerts, and the non successful one:
“t=2018-10-10T22:28:17+0000 lvl=info msg=“Notifying Pagerduty” logger=alerting.notifier.pagerduty event_type=trigger”
Tthere is also a grafana-dashboard, grafana-datasources containers and didnt seem to provide any relevant log info.
So I cannot get any indication what would be different here, or why it would not propogate to PagerDuty succesfully.
Where do I troubleshoot from here? I would think maybe I would want to make sure this integration key/call makes it out successfully to pagerduty to the outside world.
So taking some stab in the dark questions, what would be a similiar networking request I could make from within side the grafana pod to test to see if a POST can make it to pagerduty?
I don’t know how the integration key is different from the API key, I just know using the API key with grafana doesnt work, and I should and am using the right key, the integration key.
I dont want to be trying to simulate networking POSt requests that are not related to how the integration key works.