Template variable service failed Failed to fetch

Hi All,
I use Grafana to display bug metrics pulled from Jira and store them to Influxdb as data source,
Recently I have started noticing that grafana dashboard started throwing below error when I load my dashboard.
Template variable service failed Failed to fetch
Fail to fetch data on the widgets.

There is no clue of whats happening, is it a grafana issue or Influxdb issue?
Is it a data volume issue or any plugin issue?

When I edit the widgets on dashboard individually and refresh the widget, it shows up data.
But when I do browser refresh, all widgets fail to fetch.
This dashboard use to work before and now it started this behavior, not sure if this due to load, on infrastructure side SRE team has looked into load and it is perfectly fine. Out of ideas

Please help
I am using Grafana 8.5.13
Influxdb 2.4.0

Anyone knows what is the root cause for this fail to fetch error on grafana dashboard?

It looks like query used in dashboard variable is failing and then also panel queries are failing. Use browser console and check what’s their responses.
Maybe you have inefficient queries and those are killing your datasource (influx).

Thank you Jangaraj for quick reply.

I wanted to share one thing which might be causing this issue.

This dashboard is designed to display metrics for various projects selected from the dropdown as shown in the screenshot, that means, all queries are same for all the widgets on the dashboard.
This dashboard was working fine for the selected projects until recently. Even now for other projects, dashboard does display metrics without issues.

One of the root causes - may be a primary one is
Recently there was a mass update to jira bugs for two projects where we are seeing this “fail to fetch issue” as a result there about 800 to 1000 records gets pulled to influxdb.
Seems like the same query which was working for other projects stopped working for these two projects as volume of changes are more in this case.

Reducing the time range didnt help, also tried updating the query but didnt help as this mass update to bugs happened at the same time, so could you please share some thoughts or workarounds how to handle this case. appreciate your help. Thank you

why are you pushing the jira data to influxdb? Why not query the jira api directly?

for one thing jira data changes constantly, how are you keeping track of deltas in your influxdb?

We have a dotnet service which pulls jira data into influxdb to display metrics on grafana dashboard.
One of the requirements is to display Bugs status overperiod(bug trending wrt) of time for example, Development and Escapedefects for a given quarter also for a given year.
Jira gives current status only.
In Grafana our query pulls last status of the jiraid

1 Like

how often does this push mechanism run and how does it handle stale data? meaning what happens to old data?

Just trying to see if it is a scaling issue. or it could also be a lack of indices in the filtered columns.

It pushes once in a day, and old data still exists in the influx as we have metrics driven by timeperiod

We are using community edition of Grafana, so is there a limitation on the features?

Lets assume that we need to fix the queries.
If we need to fix the queries used in the widgets, what can be done in this example?.

Query to pull total number of Development defects for a given project.
SELECT last(“value”) FROM “jira_Bug” WHERE (“projectkey” =~ /^$Group$/ AND (“status” != ‘Cancelled’ AND “status” != ‘Done’ AND “bugfoundby” = ‘Development’ AND “jiraid” !~ /^$ClosedDevBugs$/)) AND $timeFilter GROUP BY “Priority”,“jiraid”

Best to read key concepts: tags vs fields which can be indexed and which cannot and design accordingly.

  1. are the following columns fields or tags?
    projectkey, status, bugfoundby, jiraid

  2. You are also using regex filters !~, ~, which are not performant.

  3. You have historical data, not sure how much, which could affect performance. I would segregate those into another bucket/table or measurement and keep only recent stuff in your main table.

SELECT last("value") 
  FROM "jira_Bug" 
WHERE ("projectkey" =~ /^$Group$/ 
     AND ("status" != 'Cancelled' 
          AND "status" != 'Done' 
          AND "bugfoundby" = 'Development' 
         AND "jiraid" !~ /^$ClosedDevBugs$/)
       ) AND $timeFilter GROUP BY "Priority","jiraid"

Those are column fields in influxdb.
I have for last 6months and around 3 thousand records for selected projectkey

What happens when you run those same failing queries outside of grafana

This is another issue. It has to pull 3k then find last. When you actually want the total number which last function does not give you.

Grafana queries has limitation, that is the only way I know to pull latest record to avoid duplicates and there is transform function to get the count of records.