Invalid alert data. Cannot save dashboard

Hello,

I’ve created a dashboard. This dashboard has just one graph panel. Then I’ve configured one basic alert:

After that I’ve tried to save this dashboard, but got this error:

Grafana backend output: t=2018-03-12T13:17:45+0000 lvl=eror msg="Invalid alert data. Cannot save dashboard" logger=context userId=1 orgId=1 uname=admin error="Invalid alert data. Cannot save dashboard"

I’ve also tried to use Cmd+S, same result.

After deleting the alert the save operation went good.

What Grafana version are you using?

5.0.1

What datasource are you using?

Prometheus

What OS are you running grafana on?

Docker image from Docker

What did you do?

Created a dashboard with one panel including an alarm and tried to save this dashboard

What was the expected result?

Message = “Dashboard saved”

What happened instead?

Message = “Invalid alert data. Cannot save dashboard”

Hi,

Could you please share your dashboard json (setting -> view json)? Thanks

Marcus

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "5.0.1"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": "5.0.0"
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "5.0.0"
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "alert": {
        "conditions": [
          {
            "evaluator": {
              "params": [80],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": ["A", "5m", "now"]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "frequency": "60s",
        "handler": 1,
        "name": "Panel Title alert",
        "noDataState": "no_data",
        "notifications": []
      },
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "node_cpu_usage",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "A"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "fill": true,
          "line": true,
          "op": "gt",
          "value": 80
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "Panel Title",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ]
    }
  ],
  "schemaVersion": 16,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d"],
    "time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d"]
  },
  "timezone": "",
  "title": "Basic_test",
  "uid": "XmBtXTRik",
  "version": 4
}

Hi,

This is not from setting -> view json. This is from share -> export as json. Please include the setting -> view json.

Marcus

We are seeing a similar response when updating existing dashboards with alerts through the API. Confirmed working in 5.0.0

on 5.0.0:

(ins) ~ (0)$ curl -XPOST -H "Content-Type: application/json" -H "Authorization: Bearer abcde1234" https://grafana/api/dashboards/db -d @/tmp/dashboard.json
{"id":20,"slug":"dev00-elasticsearch-health","status":"success","uid":"000000020","url":"/d/000000020/dev00-elasticsearch-health","version":53}

after updating to 5.0.1

(ins) ~ (0)$ curl -XPOST -H "Content-Type: application/json" -H "Authorization: Bearer abcde1234" https://grafana/api/dashboards/db -d @/tmp/dashboard.json
{"message":"Invalid alert data. Cannot save dashboard"}

if i copy the dashboard it looks like it removes all alerts, and recreating alerts on this copied dashboard, extracting json and updating through the API works as expected, but updating with the existing json template or creating a new dashboard after deleting the existing dash fails.

If this turns out to be a different issue I’ll open a separate thread.

Turns out the new ‘view json’ converts to the correct values for the API to recognize, but our existing templates are not compatible with 5.0.1. Below is a simplified version of a template payload that is failing:

{
  "dashboard": {
      "annotations": {
        "list": [
                {
                    "builtIn": 1,
                    "datasource": "-- Grafana --",
                    "enable": true,
                    "hide": true,
                    "iconColor": "rgba(0, 211, 255, 1)",
                    "name": "Annotations & Alerts",
                    "type": "dashboard"
                }
            ]
        },
        "description": "Health for the AWS hosted elasticsearch clusters",
        "editable": true,
        "gnetId": null,
        "graphTooltip": 1,
        "hideControls": false,
        "id": null,
        "links": [],
        "rows": [
            {
                "collapse": false,
                "height": 237,
                "panels": [
                    {
                        "alert": {
                            "conditions": [
                                {
                                    "evaluator": {
                                        "params": [
                                            0
                                        ],
                                        "type": "gt"
                                    },
                                    "operator": {
                                        "type": "and"
                                    },
                                    "query": {
                                        "params": [
                                            "B",
                                            "15m",
                                            "now"
                                        ]
                                    },
                                    "reducer": {
                                        "params": [],
                                        "type": "avg"
                                    },
                                    "type": "query"
                                },
                                {
                                    "evaluator": {
                                        "params": [
                                            0
                                        ],
                                        "type": "gt"
                                    },
                                    "operator": {
                                        "type": "or"
                                    },
                                    "query": {
                                        "params": [
                                            "C",
                                            "15m",
                                            "now"
                                        ]
                                    },
                                    "reducer": {
                                        "params": [],
                                        "type": "avg"
                                    },
                                    "type": "query"
                                }
                            ],
                            "executionErrorState": "alerting",
                            "frequency": "60s",
                            "handler": 1,
                            "name": "elasticsearch - cluster status alert",
                            "noDataState": "no_data",
                            "notifications": []
                        },
                        "aliasColors": {},
                        "bars": false,
                        "dashLength": 10,
                        "dashes": false,
                        "datasource": "aws-cloudwatch",
                        "fill": 1,
                        "id": 1,
                        "legend": {
                            "avg": false,
                            "current": false,
                            "max": false,
                            "min": false,
                            "show": true,
                            "total": false,
                            "values": false
                        },
                        "lines": true,
                        "linewidth": 3,
                        "links": [],
                        "nullPointMode": "null",
                        "percentage": false,
                        "pointradius": 5,
                        "points": false,
                        "renderer": "flot",
                        "seriesOverrides": [
                            {
                                "alias": "/.*ClusterStatus.green$/",
                                "color": "#508642"
                            },
                            {
                                "alias": "/.*ClusterStatus.yellow$/",
                                "color": "#eab839",
                                "legend": false
                            },
                            {
                                "alias": "/.*ClusterStatus.red$/",
                                "color": "#bf1b00",
                                "legend": false
                            }
                        ],
                        "spaceLength": 10,
                        "span": 3,
                        "stack": false,
                        "steppedLine": false,
                        "targets": [
                            {
                                "alias": "{{DomainName}}_{{metric}}",
                                "dimensions": {
                                    "ClientId": "1234567890",
                                    "DomainName": "elasticsearch-dev00"
                                },
                                "dsType": "influxdb",
                                "groupBy": [
                                    {
                                        "params": [
                                            "5m"
                                        ],
                                        "type": "time"
                                    },
                                    {
                                        "params": [
                                            "domain_name"
                                        ],
                                        "type": "tag"
                                    },
                                    {
                                        "params": [
                                            "null"
                                        ],
                                        "type": "fill"
                                    }
                                ],
                                "measurement": "cloudwatch_aws_es",
                                "metricName": "ClusterStatus.green",
                                "namespace": "AWS/ES",
                                "orderByTime": "ASC",
                                "period": "",
                                "policy": "default",
                                "refId": "A",
                                "region": "us-east-1",
                                "resultFormat": "time_series",
                                "select": [
                                    [
                                        {
                                            "params": [
                                                "cluster_status.green_maximum"
                                            ],
                                            "type": "field"
                                        },
                                        {
                                            "params": [],
                                            "type": "mean"
                                        }
                                    ]
                                ],
                                "statistics": [
                                    "Average"
                                ],
                                "tags": [
                                    {
                                        "key": "environment",
                                        "operator": "=",
                                        "value": "$environment"
                                    }
                                ]
                            },
                            {
                                "alias": "{{DomainName}}_{{metric}}",
                                "dimensions": {
                                    "ClientId": "1234567890",
                                    "DomainName": "elasticsearch-dev00"
                                },
                                "dsType": "influxdb",
                                "groupBy": [
                                    {
                                        "params": [
                                            "5m"
                                        ],
                                        "type": "time"
                                    },
                                    {
                                        "params": [
                                            "domain_name"
                                        ],
                                        "type": "tag"
                                    },
                                    {
                                        "params": [
                                            "null"
                                        ],
                                        "type": "fill"
                                    }
                                ],
                                "measurement": "cloudwatch_aws_es",
                                "metricName": "ClusterStatus.yellow",
                                "namespace": "AWS/ES",
                                "orderByTime": "ASC",
                                "period": "",
                                "policy": "default",
                                "refId": "B",
                                "region": "us-east-1",
                                "resultFormat": "time_series",
                                "select": [
                                    [
                                        {
                                            "params": [
                                                "cluster_status.green_maximum"
                                            ],
                                            "type": "field"
                                        },
                                        {
                                            "params": [],
                                            "type": "mean"
                                        }
                                    ]
                                ],
                                "statistics": [
                                    "Average"
                                ],
                                "tags": [
                                    {
                                        "key": "environment",
                                        "operator": "=",
                                        "value": "$environment"
                                    }
                                ]
                            },
                            {
                                "alias": "{{DomainName}}_{{metric}}",
                                "dimensions": {
                                    "ClientId": "1234567890",
                                    "DomainName": "elasticsearch-dev00"
                                },
                                "dsType": "influxdb",
                                "groupBy": [
                                    {
                                        "params": [
                                            "5m"
                                        ],
                                        "type": "time"
                                    },
                                    {
                                        "params": [
                                            "domain_name"
                                        ],
                                        "type": "tag"
                                    },
                                    {
                                        "params": [
                                            "null"
                                        ],
                                        "type": "fill"
                                    }
                                ],
                                "measurement": "cloudwatch_aws_es",
                                "metricName": "ClusterStatus.red",
                                "namespace": "AWS/ES",
                                "orderByTime": "ASC",
                                "period": "",
                                "policy": "default",
                                "refId": "C",
                                "region": "us-east-1",
                                "resultFormat": "time_series",
                                "select": [
                                    [
                                        {
                                            "params": [
                                                "cluster_status.green_maximum"
                                            ],
                                            "type": "field"
                                        },
                                        {
                                            "params": [],
                                            "type": "mean"
                                        }
                                    ]
                                ],
                                "statistics": [
                                    "Average"
                                ],
                                "tags": [
                                    {
                                        "key": "environment",
                                        "operator": "=",
                                        "value": "$environment"
                                    }
                                ]
                            }
                        ],
                        "thresholds": [
                            {
                                "colorMode": "critical",
                                "fill": true,
                                "line": true,
                                "op": "gt",
                                "value": 0
                            }
                        ],
                        "timeFrom": null,
                        "timeShift": null,
                        "title": "$aws_stack_id - elasticsearch - cluster status",
                        "tooltip": {
                            "shared": true,
                            "sort": 0,
                            "value_type": "individual"
                        },
                        "transparent": false,
                        "type": "graph",
                        "xaxis": {
                            "buckets": null,
                            "mode": "time",
                            "name": null,
                            "show": true,
                            "values": []
                        },
                        "yaxes": [
                            {
                                "format": "short",
                                "label": null,
                                "logBase": 1,
                                "max": null,
                                "min": null,
                                "show": true
                            },
                            {
                                "format": "short",
                                "label": null,
                                "logBase": 1,
                                "max": null,
                                "min": null,
                                "show": true
                            }
                        ]
                    }
                ],
                "repeat": null,
                "repeatIteration": null,
                "repeatRowId": null,
                "showTitle": false,
                "title": "Status",
                "titleSize": "h6"
            }
        ],
        "schemaVersion": 14,
        "style": "dark",
        "tags": [
            "aws",
            "elasticsearch"
        ],
        "templating": {
            "list": [
                {
                    "current": {
                        "text": "dev00",
                        "value": "dev00"
                    },
                    "hide": 0,
                    "label": null,
                    "name": "aws_stack_id",
                    "options": [
                        {
                            "selected": true,
                            "text": "dev00",
                            "value": "dev00"
                        }
                    ],
                    "query": "dev00",
                    "type": "constant"
                },
                {
                    "current": {
                        "text": "us-east-1",
                        "value": "us-east-1"
                    },
                    "hide": 0,
                    "label": "aws stack region",
                    "name": "aws_stack_region",
                    "options": [
                        {
                            "selected": true,
                            "text": "us-east-1",
                            "value": "us-east-1"
                        }
                    ],
                    "query": "us-east-1",
                    "type": "constant"
                },
                {
                    "current": {
                        "text": "1234567890",
                        "value": "1234567890"
                    },
                    "hide": 0,
                    "label": "aws account id",
                    "name": "aws_stack_account_number",
                    "options": [
                        {
                            "selected": true,
                            "text": "1234567890",
                            "value": "1234567890"
                        }
                    ],
                    "query": "1234567890",
                    "type": "constant"
                }
            ]
        },
        "time": {
            "from": "now-3h",
            "to": "now"
        },
        "timepicker": {
            "refresh_intervals": [
                "5s",
                "10s",
                "30s",
                "1m",
                "5m",
                "15m",
                "30m",
                "1h",
                "2h",
                "1d"
            ],
            "time_options": [
                "5m",
                "15m",
                "1h",
                "6h",
                "12h",
                "24h",
                "2d",
                "7d",
                "30d"
            ]
        },
        "timezone": "browser",
        "title": "dev00 - elasticsearch - health",
        "version": 12
  },
  "overwrite": true
}

I can confirm that everything works fine with version 5.0.0. The problem with importing/saving dashboards (including alerts) only occurs in version 5.0.1

1 Like

Hi,
I found a solution to this issue because I just experienced it right now too. You have to create the dashboard and save it first then you can go on and create alerts in the dashboard. That worked for me, hope it helps you.

Hi manu92,

this works, of course, but when you further export this dashboard and try to import it in other grafana environments this also fails.

True. The export and import of dashboards is not working with this new version 5.0.1

Had the same issue. If you are updating existing dashboards, include it’s id in JSON instead of null, and it will work.

“Invalid alert data. Cannot save dashboard” should be fixed in Grafana v5.0.4

2 Likes

I reproduce the bug in version 5.1.0 :confused:

What Grafana version are you using?

5.1.0

What datasource are you using?

InfluxDB

What OS are you running grafana on?

Windows10

What did you do?

I’m provisioning 3 dashboard from files.
One dashboard contain 2 graph with alerts.

The dashboard : https://cloud.thibautmarechal.be/s/bB52ZJ4E8YAZA8B
(The file is too long for the message)

What was the expected result?

The dashboard should be loaded with alert configured in it.
At least, it should be displayed without alerts

What happened instead?

The dashboard doesn’t load.
log : t=2018-05-02T11:59:35+0200 lvl=eror msg=“failed to save dashboard” logger=provisioning.dashboard type=file name=default error=“Invalid alert data. Cannot save dashboard”

As I wrote above:

You cannot provision dashboard json model that have been exported (dashboard → share → export → Save/View Json). You must use dashboard json either from dashboard → settings → view json or either HTTP GET of dashboard thru http api taking the content of dashboard property.

Marcus

Oh sorry about that. Thanks :slight_smile:

Using “view json” as the source I am able to import existing dashboards with alerts via the API, but only if I explicitly set a dashboard id. If I set the id to null as I would with a dashboard without alerts defined, I get the “Invalid alert data. Cannot save dashboard.” error. If I pass a existing id it will update that existing dashboard as expected, and if I enter a random unused id it will create a new dashboard. I can work around this, but it seems that dashboards w/alerts should behave the same as dashboards w/o alerts by passing null as a dashboard id allowing grafana to determine the id numbers.

For those encountering issues importing dashboards with alert settings from 5.0.4 to 5.1.3 and getting the same error. Try looking for dashboard: null in your dashboard JSON (mine is from Share → Export → View JSON):

“dashboard”: null

And replace it with a value like:

“dashboard”: $datasource

Then import using the UI (paste json), this worked for me .

Experience the same error importing dashboards with ansible module grafana_dashboard that uses API.
Grafana version 5.1.3
The import of the same json file works without any problems though UI.
Is there any workaround for this issue except removing alerts?
Is there a fix for it that is planned for incoming release?

Alerts aren’t created from alert rules when importing thru the UI, but they are when you use the http api. If you try and save the dashboard after import the alerts should be created and you should receive same error (I think). Please check the grafana server log for more details when this happens.

I had this error, when using a $datasource variable (influxDB). As soon as I select the InfluxDB-Datasource (defined in the Configuration/Datasources), I can create, save and export/import the dashboards with alerts without any errors.