-
What Grafana version and what operating system are you using?
Open-source Grafana 5.1
Prometheus 3.1.0
Running on Kubernetes 1.31
-
What are you trying to achieve?
-
How are you trying to achieve it?
-
What happened?
We are ingesting a metrics job_name with a label whose values contain special characters, specifically double quote in the current situation. The special character are escaped; e.g. job_name=“Batch Server for Context "pooledServer1"”. The label appears correctly in the Prometheus UI (lower-right in the screenshot below). Within Grafana’s Explore panel also seems to handle things fine and even strips the slashes to improve readability (upper right in the screenshot below).
The problem comes when I attempt to create a filter for job_name in a Grafana dashboard. This results in a bad_data error (as shown on left side of screenshot). The Job_Name variable is defined as label_values(:sas_launcher_pod_info:,job_name) in the Settings/Variables panel. Correct values (including those that have double quotes) appear when I validate the variable definition with the Run query button and when I interact with the pull-down box on the dashboard.
However, the error shown occurs when I add a panel to the dashboard and attempt to use :sas_launcher_pod_info:{job_name="$Job_Name"} in the query. The full error message that appears is bad_data: invalid parameter "query":1:61: parse error:unexpected identifier "pooledServer1" in label matching, expected "," or "}". I believe this means the quotes are confusing the parser.
- What did you expect to happen?
I had expected the panel to display my metrics without problems.
I am aware of Issue # 96063 which appears to report similar problems and links to a number of PRs that appear to resolve all of the issues related to special characters. However, I have upgraded to Grafana 5.1 and Prometheus 3.1.0 and still see this issue.
Are we escaping the label value incorrectly? Or, have I found another place within Grafana that has trouble with special characters? Or, am I making some other mistake?
Any/all assistance is appreciated.
Hi, you can do something like:
up{job=${query0:doublequote}}
(notice I didn’t use job=“${…}”) and it should work (I’m not sure about Grafana version dough - is it really 5.1, or 11.5.1?).
I’ll also drop that - I’m not sure Prometheus expects such characters in the labels.
Thank you for responding @dawiddebowski !
Sorry for the confusion about which version of Grafana I am using, I don’t understand how I typed that incorrectly. I am using Grafana 11.5.0.
According to the Exposition Formats section of the Prometheus documentation, the value of a label can contain special characters including double-quotes.
label_value can be any sequence of UTF-8 characters, but the backslash (\), double-quote ("), and line feed (\n) characters have to be escaped as \\, \", and \n, respectively.
I am having trouble translating your suggestion to my use-case. I wasn’t sure what the “query0:doublequote” referred to. I suspect that is a variable name reference, but I was confused by the colon.
In my case, I’ve defined a Query type variable called Job_Name as shown in the screenshot below.
When I attempt to use that variable in a query, I continue to get the same "bad_data: invalid parameter "query": 1:27: parse error: unexpected identifier "Batch" in label matching, expected string" error. I have tried specifying the query a few different ways but always get that error:
sum(sas_job_info{job_name="$Job_Name"}) my original syntaxsum(sas_job_info{job_name=${Job_Name}}) which I think is what you were suggesting
Please correct me if that was not the syntax you were suggesting.
I also tried to explicitly specify the job_name value and discovered the following
sum(sas_job_info{job_name="Batch Server for Context "pooledServer1""}) - this did NOT work.sum(sas_job_info{job_name="Batch Server for Context \"pooledServer1\""}) - this worked!
This makes me wonder if the unescaped value of the Job_Name value is being passed rather than the real value that includes the slashes. I thought it was a nice feature that Grafana removed the escape characters when displaying the values making it more readable. But maybe that also results in the escape characters are “lost” when Grafana goes to substitute to value into the query.
If that is the case, I think that is a bug. Thoughts?
Thanks again for your assistance.
sum(sas_job_info{job_name=${Job_Name}}) which I think is what you were suggesting
not quite 
query0 is the default name of the variable when you create it and I was just too lazy to change it 
In your case you would need to use
sum(sas_job_info{job_name=${Job_Name:doublequote}})
Using ${<variable_name>:doublequote} will escape " signs (more info about options here).
Oh nice. I didn’t realize your modify the variable value with modifiers that way. That seems to solve the issue.
Although, I feel I have to point out that this puts the burden on the dashboard developer to know/anticipate whether a label might at some point contain a special character. It might be nice if Grafana handled all of this magically behind the scenes.
Thanks again for the help!
Well, I think you’re the one providing the data and you know them the best. I guess in some cases you might not want the escaping, so it might be hard to do it out of the box
Alas, it appears I may have claimed victory prematurely. 
Updating my dashboard to use job_name=~${jobName:doublequote} instead of job_name=~"$jobName" did fix things in the test environment where we have a label value containing double quotes. But, when I imported the updated dashboard into another test environment I am getting errors similar to what I saw earlier. The error message is bad_data: invalid parameter "query": 1:24: parse error: unexpected "}" in label matching, expected string.
I have confirmed that both environments are running the same versions of Grafana and Prometheus. However, this 2nd environment is less busy and so there are no data at all for the metric in question (i.e. if I search for the metric in the Metrics Browser, it doesn’t exist).
I think what is happening is that ${jobName:doublequote} in this case is resolving to null instead of “” (blank space). I think this since I can trigger the same error message by changing the query to sas_job_info{job_name=~} but get no error when I change it to sas_job_info{job_name=~""}. I have the Include All option set for the JobName variable and have not provided a custom all value.
Any suggestions for how to resolve this?
The more I think about this, the more it feels like a bug. If Grafana is altering a label’s value to make it more readable to the user (which it is doing), that’s a good thing. But it must then use the unaltered value (with special characters properly escaped) when constructing the query to Prometheus. It isn’t reasonable to expect the dashboard creator to deal with this.
Can you share how the query looks in Query Inspector? (to the right of datasource picking in dashboard editing). It worked for me with empty variable (I got the label values of a label that doesn’t exists on given metric, so it should be null just like in your case) on Grafana 11.3.
As for the handling it on Grafana’s site - you can always create an issue on Github and let Grafana developers decide
Here are two screenshots showing the Query Inspector for the same query. The first shows Query Inspector with the original version of the dashboard and does not currently report any errors.
I can see that the job_name portion of the query is job_name=~"()".
This second screenshot shows the same view of the Query Inspector taken from the updated dashboard. The only difference between the two dashboards is the use of the :doublequote modifier for job_name in the query.
Notice that the job_name portion of the query is now job_name=~ (i.e. there is no value specified on the right side of the operator). And, I believe that is what is triggering the errors.
Both of these dashboards are deployed in the same environment and this environment does not have any job names containing double quotes. When the updated dashboard is deployed in a different environment where there are jobs with names containing double quotes, there are no errors. And, as reminder, when the original dashboard is deployed in that environment, it produces errors. Unfortunately, I cannot know ahead of time whether or not there may be job names containing double-quotes in any given environment at any given point in time.
Here is a screenshot showing how the jobName (Grafana) variable is defined.
I think I will take your advice and open a GitHub issue since I suspect this is a missed use-case related to the PRs intended to fix Grafana Issue #96063 (mentioned above).
I may also investigate defining a Prometheus rule to redefine the label with the quotes removed/replaced.
Thanks again for your assistance.
Regards,
Greg
1 Like
Just to close the loop on this: I opened an issue on the Grafana GitHub site to report this since it appeared to a unaddressed use-case from Issue # 96063.
But it turns out the fix for that issue is disabled by default and only enabled when a Grafana feature toggle, prometheusSpecialCharsInLabelValues, is set to true. Once I enabled that toggle, Grafana handled the double-quotes properly in my original dashboard and there was no need to use the ${Job_Name:doublequote} syntax.
@dawiddebowski Thanks again for the help on this and for sharing the information about the modifiers that can be applied to variable values. While I don’t need them for this problem, they will be helpful in the future.
1 Like
