Grafana 8.0 incompatibilities

[vbontchev]

I am running Grafana on a Linux Ubuntu 16.04 machine. The other day it updated to version 8.0 and now my visualizations are broken or ugly.

Here is an example of how it looked before:

Here is how it looks now:

I can identify several problems:

1. Apparently the single-stat panel is not supported any more. It has been replaced by the horrible, ugly, no-good stat panel. At least it displays something (unlike some other panels that are completely broken; I’ll get to that in a moment) - but it is ugly. Several problems with it:

• The numbers are too large. OK, I can change the size of the text - but to what? Previously, I was using 70% font size. Here I am supposed to enter some absolute number (pixels?). “30” looks approximately like the old size - but I’m not sure. What number should I use here, in order to have the same size as before?

• The text is colored. OK, I can change the color scheme to “single color” and select the “text” color for it - and this mostly works but only for panels that contain a single number. What about those that contain a graph? To begin with, why didn’t the conversion use the color I had already set in the old panel? OK, I know what colors I used there, so I can change that. But why is the text colored the same as the graph?! I want only the graph to be colored. How do I change the color of the text?

2. The WorldMap panel simply no longer works. I use a MySQL database as a data source. It hasn’t changed. If I enter manually the request that I’ve told Grafana to use, I get correctly a table of data. But the Grafana panel is showing an error instead:

'failed to convert long to wide series when converting from dataframe: long series must be sorted ascending by time to be converted"

WTF does this mean? Googling the exact message yields nothing. The query I use in Grafana is something like

SELECT
  $__time(timestamp),
  country_iso_code AS metric,
  COUNT(country_iso_code) AS value
FROM
  connections,
  geolocation
WHERE
  $__timeFilter(timestamp) AND
  connections.ip = geolocation.ip
GROUP BY metric

When the macros are resolved, this translates to something like

SELECT
  UNIX_TIMESTAMP(timestamp) as time_sec,
  country_iso_code AS metric,
  COUNT(country_iso_code) AS value
FROM
  connections,
  geolocation
WHERE
  timestamp BETWEEN FROM_UNIXTIME(1622882715) AND FROM_UNIXTIME(1623487515) AND
  connections.ip = geolocation.ip
GROUP BY metric

If entered manually to MySQL, it returns data like this:

time_sec	metric	value
1623001011	BE	1
1623001003	GB	1
1623032218	NL	1
1622911279	US	22

What am I supposed to do, in order to make it work?

3. Then there is a pie chart panel, which simply doesn’t display anything any more (“no data points”), although not an error indication, either. I suspect that the problem is similar as the previous one.

Damn it, guys, why do the developers always have to break things when issuing a new version?! I’ve had problems literally with every update - but so far they were minor, like things in the user interface being moved around and made more inconvenient. But now our whole visualization is broken and the boss won’t be happy…

Any help?

[neteddie]

Same problem here: After the update my panel shows “No data” and the error message is “failed to convert long to wide series when converting from dataframe: long series must be sorted ascending by time to be converted”. The data source is MariaDB, too. Any help would be appreciated.

[vbontchev]

I am starting to get an inkling of what is happening and it looks like a bug in Grafana 8.0.

If I change the visualization from Worldmap to Table, I don’t get quite the same table that MySQL returns and is shown in my example above. In particular, instead of Unix epoch times, I get human-readable times. I guess this confuses the hell out of the visualizations (Worldmap, Pie Chart) that expect time series with epoch times.

And, indeed, I managed to fix the Pie Chart panel by making it use a Table instead of Time Series. Unfortunately, this doesn’t work for Worldmap, which insists on getting time series data.

This needs to be fixed ASAP. I’ll open an issue on GitHub.

[vbontchev]

I opened an issue on GitHub. The main developer confirmed that the problem is caused by the epoch times being converted to human-readable timestamps - this tends to confuse the Worldmap and probably everything else that uses time series data.

Hopefully, they will fix the MySQL data source soon.

[marcusolsson]

Thank you for taking the time to provide constructive feedback, despite the less than stellar experience with upgrading to the new version

We do try to test different combinations of panels and data sources before releases, but it sounds like we missed this one. We’re working on improving our test coverage for new releases, but we’ll continue to recommend backing up any dashboards before trying out new versions as bugs might still slip through.

As for the Stat panel, you might be interested to know that we’ve published the old Singlestat panel as a standalone plugin. If Grafana detects the standalone Singlestat during startup, then that will prevent your panels from being automatically migrated. Hopefully it’ll let you keep using Singlestat until you’ve had a chance to get familiar with the new Stat panel.

[vbontchev]

Thanks. Unfortunately, my panels were already migrated to the Stat panel. I’ll install the Singlestat standalone plugin but ideally, I’d like to use the new Stat panel, provided that I can make it do what Singlestat did before. Currently, the main problem with it is the colored text. Is it possible for the value shown on the panel to have a different color from the sparkline graph?

Speaking of new panels, please don’t discontinue the old Table panel before making sure that the new one supports the same features. In particular, I need pagination and the ability to specify font size and hate the endless scroll of the new Table panel.

I noticed that version 8.0.2 of Grafana has been released but, unfortunately, the MySQL data source issue is not fixed there; that’s a milestone for 8.0.3. I’m not sure I’ll be able to wait for that - the boss is pressing me to install version 7.5 and disable the automatic updates for this package…

[vbontchev]

OK, I installed the stand-alone Singlestat plugin, restarted the Grafana server, and I can see the plugin on the list of available plugins. However, when I try to change the Stat panel to Singlestat, the latter does not appear as available on the list of available panels.

Sigh…

[marcusolsson]

Grafana 8.0 migrates Singlestat to Stat the first time you start it. To prevent the migration, you’d have to have installed it before the first boot. If you do decide to restore your dashboards and revert to 7.5, it might be something for you to consider if you decide to upgrade at a later time.

The color of the text and background both use the Color scheme option so afaik it’s not possible to set colors individually.

Your concerns regarding the Table panel are understandable and I’ll make sure to forward them to the team.

[vbontchev]

@marcusolsson

Grafana 8.0 migrates Singlestat to Stat the first time you start it. To prevent the migration, you’d have to have installed it before the first boot.

Hold on, I don’t understand. Are you saying that it is not actually a panel one can use; it is only some kind of flag that prevents Grafana from upgrading? But if Grafana does’t upgrade, it means that it still can use the Singlestat panel, no? Can’t I just edit manually the JSON definition of the panel and change its type to singlestat after the upgrade? I remember that I was able to get the old Table panel with a hack like that…

[marcusolsson]

It is indeed a proper standalone panel. In fact, it’s the same old Singlestat that has been extracted from the Grafana repository into its own repository.

When you start Grafana 8.0 for the first time, it will check to see if the Singlestat plugin is installed. If it is, Grafana won’t perform the migration to the new Stat panel model, and simply update the panel type in the dashboard JSON from singlestat to grafana-singlestat-panel.

You’re correct that you could edit the dashboard JSON manually retroactively, but I believe the panel models for the old Singlestat and the new Stat are incompatible. If you manually set the type after the migration, you’d be using the new panel model with the old Singlestat, which likely wouldn’t work.

[vbontchev]

@marcusolsson

You’re correct that you could edit the dashboard JSON manually retroactively, but I believe the panel models for the old Singlestat and the new Stat are incompatible. If you manually set the type after the migration, you’d be using the new panel model with the old Singlestat, which likely wouldn’t work.

That’s not a problem. “Wouldn’t work” just means that the various options wouldn’t translate automatically. But there aren’t that many of them; I can re-configure them manually. I just did that with one of the visualizations - changed the panel type to grafana-singlestat-panel and re-configured it (use total instead of the default average, show spark lines, full height, and what color) - and it works just like it used to. Thanks for your help; this problem is solved.

Now only the Worldmap panel problem remains. Hopefully, it will be fixed soon, because the boss has gone off the reservation, wants us to ditch Grafana completely and to make our own visualization in PHP… This is crazy - it will take months (not just the visualization, we’d have to re-design the backend database, then modify the honeypot to use the new database… It will be a hell of a lot of work and completely wasted IMO… Not to mention that the result will look ugly.

[jwollman]

I’m getting this too after our upgrade to version 8. The fix would be to put an ORDER BY in your query to whatever column is your “time” column in the SELECT. That should fix this error if your query is a timeseries query (and not a table query).

But I agree, the upgrade broke a lot of things for us including all pie charts.

[vbontchev]

@jwollman

I’m getting this too after our upgrade to version 8. The fix would be to put an ORDER BY in your query to whatever column is your “time” column in the SELECT. That should fix this error if your query is a timeseries query (and not a table query).

No. Try it. It wont work. The problem is that the MySQL data source no longer returns Unix epoch times (which is what all plugins that use time series expect). Instead, it returns human-readable timestamp string (date, time) - which breaks everything that expects to see an integer there.

The pie charts are easily fixable, though. Just remove the part of the query that returns time and tell the panel to use table instead of time series. Sadly, this approach doesn’t work for the Worldmap panel.

[vbontchev]

OK, I have one more little problem.

One of my visualizations is a hourly bar chart. It still works but now it looks like this:

Look carefully at the legend. There is a word “value” there that’s not supposed to be present. The panel uses time series; I suspect that the problem will go away when the data source is fixed.

[jwollman]

@vbontchev

SELECT
  UNIX_TIMESTAMP(timestamp) as time_sec,
  country_iso_code AS metric,
  COUNT(country_iso_code) AS value
FROM
  connections,
  geolocation
WHERE
  timestamp BETWEEN FROM_UNIXTIME(1622882715) AND FROM_UNIXTIME(1623487515) AND
  connections.ip = geolocation.ip
GROUP BY metric

this is how we’d write your query (but we use postgres) this assuming that your timestamp column is some type of integer recording seconds from the epoch:

SELECT
timestamp as “time”,

– or
– date_trunc(‘day’, (TIMESTAMP ‘epoch’ + timestamp * INTERVAL ‘1 second’)) AS “time”,

country_iso_code AS metric,
COUNT(country_iso_code) AS value
FROM
connections,
geolocation
WHERE
$__unixEpochFilter(timestamp)
AND
connections.ip = geolocation.ip
GROUP BY 1, 2
Order by 1

[vbontchev]

jwollman, I’m afraid I don’t understand what you mean. Yes, the time is Unix epoch time, which means that it is long integer. And the problem is that instead of this, the MySQL data source is returning human-formatted time string.

Using timestamp AS "time", makes absolutely no difference and results in exactly the same error. Same with your second proposal, which I don’t even understand the purpose of the manipulation you’re doing there.

[vbontchev]

OK, Grafana 8.0.3 has fixed this particular issue. However, the bigger problem (the one that affects the Worldmap panel so badly) is still not fixed - I noticed that the fix has been moved as a milestone for version 8.0.4.

[vbontchev]

@vbontchev

The text is colored. OK, I can change the color scheme to “single color” and select the “text” color for it - and this mostly works but only for panels that contain a single number. What about those that contain a graph? To begin with, why didn’t the conversion use the color I had already set in the old panel? OK, I know what colors I used there, so I can change that. But why is the text colored the same as the graph?! I want only the graph to be colored. How do I change the color of the text?

This has been kinda fixed in Grafana 8.0.2. You can’t set the color of the value to whatever color you want - but you can make it not colored. From the panel settings, select Stat styles → Color mode → None. (The options are None, Value, Background, with Value being the default.)

[LucaAzalim]

Same problem here. After upgrading to 8.0.3, almost all my time series panels that use MySQL as the datasource stopped working.

Adding “ORDER BY created_at” to the query makes the error disappear, but now I see a completely empty graph.

Query:

SELECT created_at AS "time", sky_gold_log_tags.value as 'metric', $aggregation_query as "id" FROM sky_gold_log
JOIN sky_gold_log_tags ON sky_gold_log_tags.log_id = sky_gold_log.id AND sky_gold_log_tags.key = 'product_game'
WHERE $__timeFilter(created_at) AND action = 'REMOVE' AND source = 'BILL' AND valid = 1
GROUP BY metric
ORDER BY created_at

Was anyone able to fix this?

[LucaAzalim]

Adding “ORDER BY created_at ASC” to all queries fixed the problem I described above.