There is something wrong with UNIFIED ALERTS

[gorkemerdogan]

sorry for the dramatic title… I’m not sure if there is a bug with alerts or I’m just doing something wrong… I’d appreciate it if somebody can help me figure it out.

Here is the sample notification I’m receiving;

{
    "receiver": "AWS-API",
    "status": "firing",
    "alerts": [
        {
            "status": "resolved",
            "labels": {
                "alertname": "room52 alert"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:10:09.4695512Z",
            "endsAt": "2021-08-08T09:39:09.4695512Z",
            "generatorURL": "......",
            "fingerprint": "01fae9f107e8bd51",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room52} value=2.365125572430543 ]"
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room12 alarm"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:28:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "a098028d1b41e9d3",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room12} value=1.0623855049899877 ]"
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room34 alarm"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:14:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "4b0d66b2b6892db2",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room34} value=-2.150685102297814 ], 
							[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room34} value=-1.4462688306315763 ]"
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room27 alarm"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:34:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "743396bd962fb20e",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room27} value=-1.0793742662029506 ]"
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room11 alert"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:15:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "20a09a687697d95d",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room11} value=-1.1582701187584759 ]"
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room44 alert"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:39:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "b05d3969851e0d8b",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": ""
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "room05 alert"
            },
            "annotations": {},
            "startsAt": "2021-08-08T09:36:09.4695512Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "......",
            "fingerprint": "58da6e7b8acf20db",
            "silenceURL": "......",
            "dashboardURL": "",
            "panelURL": "",
            "valueString": "[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room05} value=1.5318855962592126 ], 
							[ metric='toxic_gas_level' labels={id=0451, city=paris, location=room05} value=1.144295316738316 ]"
        }
    ],
    "groupLabels": {},
    "commonLabels": {},
    "commonAnnotations": {},
    "externalURL": "https://example.com:3000/",
    "version": "1",
    "groupKey": "{}:{}",
    "truncatedAlerts": 0,
    "title": "[FIRING:6]  ",
    "state": "alerting",
    "message": "some message"
}

There are a couple of things going on here…

1. ‘status’ : ‘firing’ looks like it should group alerts list of only FIRING ones. but as you can see there are 6 FIRING and 1 RESOLVED. If this is some kind of grouping mechanism, it should group RESOLVED and FIRING alerts separately. Am I wrong?

2. ‘valueString’ information is the most important information for our use case. Our algorithm won’t work without ‘valueString’. We are receiving firing notifications with empty ‘valueString’. Why this is happening? ‘firing’ notifications must always have a valid ‘valueString’. Ex; room44 alert

3. resolved notifications should NOT have ‘value’ in ‘valueString’ because tecnhically there is no matching value since it has been resolved. Or it should not have ‘valueString’ all together. Ex: room52 alert

Old grafana alerts had no issues at all with these… I’m not sure if unified alarms suppose to work like this or am i missing something here?