Grafana annotations API do not support variables

[nerijusrazvodovskis]

Hello Community!

In a short ill try to describe a problem which I am currently facing.

I made a grafana annotator script which posts an annotation on a service failure. The problem is that it do not recognize variables of dashboard. For example we got a “node” variable which filters instance name (label_values(node_load1, instance)), so in one dashboard we can select a lot of instances and check the node metrics of it.

The problem is if we post annotation, it posts it globally on all instances, and variable is not recognized.

Any solutions how could i solve it?

Regards

[mattabrams]

Hi @nerijusrazvodovskis ,

Can you share a bit more about your dashboard, its variables, and especially your annotator script?

~m

[nerijusrazvodovskis]

Sure, sharing a bit more:

So im using variable “Node” to recognize all instances for the dashboard, it looks like:

The script looks like:

#!/bin/bash

AUTH=“Authorization: Bearer TOKEN==”
GRAFANA_BASE=“https://grafana-dev.io/api/annotations”
CURRENT_TIME=“$(date +%s)”
CURRENT_TSP=“$(date +%s)“000
DASHBOARD_ID=6
PANEL_ID=1
TAG=TestTag
TEXT=“Test annotation by Nerijus”
BODY=$(cat <<EOF
{
“dashboardId”: $DASHBOARD_ID,
“panelId”: $PANEL_ID,
“tags”: [”$TAG”],
“text”: “$TEXT”,
“time”: $CURRENT_TSP
}
EOF
)

curl -H “$AUTH” -H “Content-Type: application/json” -XPOST -d “$BODY” “$GRAFANA_BASE”

So and the problem is, that when i post annotation, it’s not recognized by variable, so that means it puts for all instances in dashboards:

In example above i’ve posted annotation from test server2, and it appeared in all dashboards because it’s not recognized by variable

[nerijusrazvodovskis]

Hello, @mattabrams how are you?

Any suggestions to the issue above?

Regards

[mattabrams]

Hi @nerijusrazvodovskis,

It looks like there is a pretty active discussion on GitHub around some related issues.
Are you thinking of something like this?

Interpolate dashboard variable when used in annotation

[mattabrams]

@nerijusrazvodovskis,

In the meantime, I was able to pass template variables into an annotation’s text using the HTTP API. The annotations still appear across instances, but at least each annotation is specific to one:

curl -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer my-api-token" -X POST -d "{\"dashboardId\": 29, \"panelId\": 2, \"time\": 1620701703864, \"timeEnd\": 1620701703864, \"tags\":[\"localhost:9100\"], \"text\":\"this is a tesCt on $instance\"}" https://grafana.example.dev/api/annotations

Now from the UI, you can create non-global annotations with specific queries that pass in variables. See the query by tag section in the documentation for annotations.

Then the annotations only appear when viewing that variable:

[nerijusrazvodovskis]

Hey, yes, that’s actually what i need, so ill wait for the new feature because from the issues i can see that the tags are not recognized with a variables. We can close this ticket i think.

Thank you!

[mattabrams]

Ok @nerijusrazvodovskis, so I spoke with another engineer and I actually think what you want to do is possible.

Using the above curl command, I was able to reproduce your problem. I created an annotation for a panel that was using template variables to render data for two different instances. But, the annotation would appear across both panel-views.

I changed the Filter by to Tags in the Annotation UI for that dashboard. I then added the template variable as a tag. Then the views rendered properly.

Here is how the panel rendered before the filter change:

And here it is after the filter change:

will this help solve the problem with your script?

[nerijusrazvodovskis]

Hey!

Yes i see you have replicated my issue, i’ve already tried filter by tags before on my end too, but no success (it was adding to both dashboards anyways ). Could you also provide your variable configuration? Maybe im doing something wrong with the variable Also you are on the newest version of grafana?

Thank you for your answers, Regards.

[mattabrams]

Hi @nerijusrazvodovskis,

I made that test on Grafana 7.4.5. Here is my variable config:

I would suggest seeing if you can pass in a variable using curl manually, and then troubleshoot the bash script

[nerijusrazvodovskis]

Hi @mattabrams,

Did you added the annotation directly in the dashboard or you’ve did it via curl?

[mattabrams]

I added the annotation using cURL from my local machine. Grafana is running on a remote Ubuntu 20 server, on v.7.4.5, behind nginx w/ domain & tls. hmmm…
still no luck?

[nerijusrazvodovskis]

Yes somewhy no luck, really weird

[guilhermecarmo]

I’m having the same problem, have you had an update on this topic?