Grafana Webhook Notification Errors (400 for Slack, 500 for Parny)

Grafana Alerting
, ,
1

Hi,

I am facing an issue with webhook notifications in Grafana. I have two separate notification channels configured:

  1. Slack Notification Channel: Returns a 400 Bad Request error in Grafana logs.
  2. Parny Call Application Notification Channel: Returns a 500 Internal Server Error in Grafana logs.

Here are the details:

  • Manual Tests (curl commands): When I send POST requests to both webhook endpoints directly from the Grafana server using curl, the requests are successful, and I receive valid responses for both Slack and Parny endpoints.
  • Grafana Logs:
    • Slack: logger=alerting.notifier.webhook t=... lvl=eror msg="Failed to send webhook" error="webhook response status 400 BAD REQUEST" webhook=slack
    • Parny: logger=alerting.notifier.webhook t=... lvl=eror msg="Failed to send webhook" error="webhook response status 500 INTERNAL SERVER ERROR" webhook=parny

What I have tried:

  1. Verified both endpoints using curl from the Grafana server—both are working as expected.
  2. Checked for payload differences between Grafana and curl requests.
  3. Validated that the webhook URLs are correctly configured in Grafana.

Questions:

  1. Why does Grafana fail to send notifications while curl works without any issues?
  2. Are there specific payload structures or headers that Grafana requires for these notification channels (Slack and Parny)?
  3. How can I debug this further to identify why Slack returns a 400 and Parny returns a 500?

Any insights or suggestions to resolve this issue would be greatly appreciated.

Thank you!

2

image
image2032×1316 167 KB

I use v8.5.13