Issue: Error log spam with "Exemplar query failed". Exemplars are not even enabled

Hello.

We are using grafana
Version 8.3.3 (commit: 30bb7a93ca, branch: HEAD)

From few days ago we start receiving a lot of errors in the logs similar to:

t=2021-12-18T23:59:04+0200 lvl=eror msg=“Exemplar query failed” logger=tsdb.prometheus query=“sum (app_status_value{host=“host”,name=“component”})” err=“client_error: client error: 404”

About 30-40 messages every 5 seconds from active dashboards.
The concern is that we are not using Exemplars in our Prometheus setup at all.

Due to abnormal log increasing the only way to prevent host from stopping was to exclude error output for that logger with the filter
filters = tsdb.prometheus:critical

Can we do something to get rid of this errors without disabling logger ?
Why it even appeared with disabled Exemplars?

upd: found the same issue on the github

It seems I’ve found a problem.
As I understood, exemplars are enabled by default with the basic queries.


Anyway, my queries didn’t have them, but the button was enabled in the query menu.
Disabling it (making eye grey) solved the issue