Dashboard Time Value Displays Incorrectly

Running Grafana 9.4.3 on RPi 4 running Bullseye.

I have a dashboard that reads various MariaDB tables and displays information, including the timestamp of the records that were retrieved for display. This has been working fine for months.

Today, and for some unknown reason, the timestamps are displaying as if the data is from the future by 4 hours. I’ve checked the settings on the dashboard and the server defaults, and they are set to display as the local browser time (Pacific).

I’ve read the DB tables directly and they show the correct timestamp relative to the current time. But when the timestamp field is displayed as part of the dashboard the time is shifted forward by 4 hours. The dashboard was working just fine.

Today I added a new panel and added a new datasource of MQTT, and when I applied that new panel things went wacky.

Any help on what is getting messed up would be appreciated.

So, I think I’ve figured it out. Maybe…

The server (RPi 4, Bullseye-64) that is running MariaDB is set to the “local” timezone (US/Los Angeles, although I’m in the Reno NV area). The dates stored in the DB tables show the proper local time timestamp on records (i.e if it’s 12:30pm PST when a record is inserted that’s what it shows).

The server (RPi 4, Bullseye-64) that is running Grafana is set up the same way, US/Los Angeles as the system date/time.

On the Grafana server, if I set the time Grafana time to US/Los Angeles (UTC-8) then times get display as 4:30 am (if the actual time is 12:30PM), so the server apparently takes the data records returned from MariaDB, assumes the timestamps are UTC, and backs them up by 8 hours yielding 4:30am. If I set the Grafana server to UTC only then things are assumed to be correct “as is” and the records show the correct time.

Unfortunately, this means my Grafana dashboard refresh pull down shows UTC time, without adjustment (i.e. it displays 20:30pm as the “time” when it should be showing 12:30pm) but then it does leave the displayed database records alone so the panel details show 12:30pm as they should.

Maybe it’s documented somewhere, but I’d think it should be more obvious that on data sources if you set the Grafana timezone to your local time, it apparently causes timestamps on data sources to be “adjusted” by the off set.

So, if there is some Grafana cheat sheet that explains how you should actually configure things I’d sure like to see it. I guess everyone should always use UTC time for source records, and then let Grafana adjust times based on the Grafana server setting. But that is problematic for people with historical DB tables holding anything but UTC timestamps. I guess I could add a calculation to the sql statements to add back the 8 hours (in my case), then set Grafana to UTC-8, and it would adjust properly and the displays would all work. Ugh…