Notification -> script.exec

1

Hi,
New here… Got Grafana in my knee, need to integrate alerts with our Control Centre. Easiest way for me is if Grafana alert engine could execute a custom script and parse variables into it.

Possible? Howto?

2

There is an alert notification webhook for custom integrations:

http://docs.grafana.org/alerting/notifications/

3

This is the worst reply ever. We want to start some script in case of an alert. That makes no sense for us to build our own backend server that listens to http request for starting a simple script or other program.

1 Like
4

I strongly agree, This is a missing link in automation.
My server runs out of memory, Grafana picks it up, and kicks off a Jenkins Jobs/Script that does the restart for me.

5

It is the more generic approach (using webhooks). It implies more coding yes but imagine to support most of scripting languages and consider related issues like security or maintainability. IMHO it is not affordable.

6

My Apologies, After looking at the weebhook I managed to easily create a Jenkins pipeline job with generic webhook plugin.

Works like a charm.

Have some examples if someone needs it.

1 Like
7

Can you please post some examples?

8

Hi there @oandewaal,

On Grafana I configured a notification channel called JenkinsCall.
Type: webhook
Url: http://jenkins_url/generic-webhook-trigger/invoke?token=abc123
Http Method: POST
Reminder: 5m

On Jenkins I installed the plugin “Generic Webhook Trigger Plugin”
This allows you to set a token in the Jenkins job.

Assigned the JenkinsCall to an alert.

I on the other hand did not leave it right there, I used a jenkinsfile to control the workflow.

Example:

def jobToken
def jenkins_url
def args
def job
def description
node {
 properties([
  pipelineTriggers([
   [$class: 'GenericTrigger',
    genericVariables: [
     [ key: 'title', value: '$.title' ],
     [ key: 'ruleId', value: '$.ruleId' ],
     [ key: 'ruleName', value: '$.ruleName'],
     [ key: 'ruleUrl', value: '$.ruleUrl'],
     [ key: 'state', value: '$.state' ],
     [ key: 'imageUrl', value: '$.imageUrl' ],
     [ key: 'message', value: '$.message' ],
     [ key: 'evalMatches_metric', value: '$.evalMatches[0].metric', expressionType: 'JSONPath', regexpFilter: '', defaultValue: '' ],
     [ key: 'evalMatches_tags', value: '$.evalMatches[0].tags', expressionType: 'JSONPath', regexpFilter: '', defaultValue: ''  ],
     [ key: 'evalMatches_value', value: '$.evalMatches[0].value', expressionType: 'JSONPath', regexpFilter: '', defaultValue: ''  ]
    ],
     
    causeString: 'Grafana alert from $title',
    
    token: 'abc123',
    
    printContributedVariables: true,
    printPostContent: true,
   ]
  ])

 ])

 stage("Gather Facts") {
  deleteDir()
  sh '''
  echo "Title: $title"
  echo "RuleId: $ruleId"
  echo "RuleName: $ruleName"
  echo "RuleUrl: $ruleUrl"
  echo "State: $state"
  echo "ImageUrl: $imageUrl"
  echo "Message: $message"
  echo "EvalMatches Metric: $evalMatches_metric"
  echo "EvalMatches Tags: $evalMatches_tags"
  echo "EvalMatches Value: $evalMatches_value"
  sleep 1

  if [ "$state" = "ok" ]
    then
        echo "Alert returned Status: $state"
        exit 0
  fi

  if [ "$state" = "no_data" ]
    then
        echo "Alert returned Status: $state"
        exit 0
  fi

  '''
 }

 stage("Do something with the data") {
 
 }

}
1 Like
9

Great thanks! Will have a look

10

Thanks Malanvaneck!
I’m not a developer but it turns out its quite easy to do with the help of your hints and example code.
Just install jenkins and the rest was quite straightforward.