Missing Data in Grafana Graphs Despite Continuous Netdata + InfluxDB Collection and Temporary Network Loss

Grafana InfluxDB
, ,
1

Issue Description:

The system, running Debian 10 on an Intel NUC, was operating normally. However, there was a network connection interruption at approximately 12:10 PM on December 27. Despite the machine remaining powered on, Grafana’s graphs stopped updating.

The data does not align correctly with the time axis in Grafana. There is a 29-minute gap in the displayed data from 12:06 to 12:35, even though InfluxDB contains the data for this period. The issue was resolved only after rebooting the machine.

Primary Issue:

Grafana’s graphs exhibit missing segments, despite the data being properly recorded in InfluxDB and the machine running during that time period.

  • Image

  • Image

  • Image

  • Image

  • Image

  • Data is present in InfluxDB, and the machine remained powered on.

  • Used InfluxDB shell (SELECT * FROM measurement) and confirmed that the data exists for the missing period.

Connected to http://localhost:8086 version 1.8.1
InfluxDB shell version: 1.8.1
> use netdata
Using database netdata
> precision rfc3339
> SELECT * FROM "netdata.system.uptime.uptime"
name: netdata.system.uptime.uptime
time                 host                      value
----                 ----                      -----
2024-12-27T11:59:58Z DEB-NUC11PAH 30207.4713333
2024-12-27T12:00:58Z DEB-NUC11PAH 30267.5
2024-12-27T12:01:58Z DEB-NUC11PAH 30327.5005
2024-12-27T12:02:58Z DEB-NUC11PAH 30387.1833333
2024-12-27T12:03:58Z DEB-NUC11PAH 30447.4155
2024-12-27T12:04:58Z DEB-NUC11PAH 30507.505
2024-12-27T12:05:58Z DEB-NUC11PAH 30567.5081667
2024-12-27T12:06:58Z DEB-NUC11PAH 30627.2443333
2024-12-27T12:07:58Z DEB-NUC11PAH 30687.3618333
2024-12-27T12:08:58Z DEB-NUC11PAH 30747.4713333
2024-12-27T12:09:58Z DEB-NUC11PAH 30807.3278333
2024-12-27T12:10:58Z DEB-NUC11PAH 30867.3898333
2024-12-27T12:11:58Z DEB-NUC11PAH 30927.4505
2024-12-27T12:12:58Z DEB-NUC11PAH 30987.215
2024-12-27T12:13:58Z DEB-NUC11PAH 31047.5
2024-12-27T12:14:58Z DEB-NUC11PAH 31107.4563333
2024-12-27T12:15:58Z DEB-NUC11PAH 31167.3141667
2024-12-27T12:16:58Z DEB-NUC11PAH 31227.5053333
2024-12-27T12:17:58Z DEB-NUC11PAH 31287.3616667
2024-12-27T12:18:58Z DEB-NUC11PAH 31347.2481667
2024-12-27T12:19:58Z DEB-NUC11PAH 31407.5
2024-12-27T12:20:58Z DEB-NUC11PAH 31467.3561667
2024-12-27T12:21:58Z DEB-NUC11PAH 31527.3281667
2024-12-27T12:22:58Z DEB-NUC11PAH 31587.415
2024-12-27T12:23:58Z DEB-NUC11PAH 31647.4481667
2024-12-27T12:24:58Z DEB-NUC11PAH 31707.4476667
2024-12-27T12:25:58Z DEB-NUC11PAH 31767.1603333
2024-12-27T12:26:58Z DEB-NUC11PAH 31827.5005
2024-12-27T12:27:58Z DEB-NUC11PAH 31887.485
2024-12-27T12:28:58Z DEB-NUC11PAH 31947.4573333
2024-12-27T12:29:58Z DEB-NUC11PAH 32007.2991667
2024-12-27T12:30:58Z DEB-NUC11PAH 32067.4161667
2024-12-27T12:31:58Z DEB-NUC11PAH 32127.4495
2024-12-27T12:32:58Z DEB-NUC11PAH 32187.2456667
2024-12-27T12:33:58Z DEB-NUC11PAH 32247.3033333
2024-12-27T12:34:58Z DEB-NUC11PAH 32307.5
2024-12-27T12:35:58Z DEB-NUC11PAH 32367.4711667
2024-12-27T12:36:58Z DEB-NUC11PAH 32427.444
2024-12-27T12:37:58Z DEB-NUC11PAH 32487.2991667
2024-12-27T12:38:58Z DEB-NUC11PAH 32547.4136667
2024-12-27T12:39:58Z DEB-NUC11PAH 32607.5086667
2024-12-27T12:40:58Z DEB-NUC11PAH 32667.2455
2024-12-27T12:41:58Z DEB-NUC11PAH 32727.3038333
2024-12-27T12:42:58Z DEB-NUC11PAH 32787.5
2024-12-27T12:43:58Z DEB-NUC11PAH 32847.5
2024-12-27T12:44:58Z DEB-NUC11PAH 32907.4431667
2024-12-27T12:45:58Z DEB-NUC11PAH 32967.2706667
2024-12-27T12:46:58Z DEB-NUC11PAH 33027.3575
2024-12-27T12:47:58Z DEB-NUC11PAH 33087.5043333
2024-12-27T12:48:58Z DEB-NUC11PAH 33147.4215
2024-12-27T12:49:58Z DEB-NUC11PAH 33207.1713333

Observations:

  • A data gap is visible in Grafana panels between 12:06 and 12:35, even though the data exists in InfluxDB.
  • The time axis appears to have skipped 29 minutes.
  • Normal graph rendering resumed after a system reboot.
  • The timestamp in the Grafana query is correct.

Queries are correct, and we also tried to reproduce this issue on a machine with the same configuration, software, and hardware, but there were no time axis gaps in Grafana graphs, even without an internet connection.

For example purposes, here is the query to verify uptime data:

SELECT max("value") FROM "netdata.system.uptime.uptime"  
WHERE time >= now() - 6h  
GROUP BY time(1m) fill(null)

Please check the attached screenshot for reference.
image

Environment Details:

  • OS: Debian 10 Buster with Backported Kernel 5.10.0-0.deb10.16-amd64
  • Grafana Version: 7.3.6
  • InfluxDB Version: 1.8.1
  • Netdata Version: v1.31.0-68-g2e88104dd

Setup:

  • Data Collection: Netdata
  • Time-Series Database: InfluxDB 1.8.1

Troubleshooting Steps Taken:

  1. Verified InfluxDB Data:
  2. Checked Network & System Uptime:
    • Confirmed that the system was running during the affected period.
  3. Restarted Machine:
    • Around 12:40 PM, the issue was resolved after unplugging and reinserting the power cable, followed by a system reboot.which restored normal functionality.
root@DEB-NUC11PAHS:~# last reboot
reboot   system boot  5.10.0-0.deb10.1 Tue Jan 21 10:17   still running  
reboot   system boot  5.10.0-0.deb10.1 Fri Dec 27 12:36   still running 

root@DEB-NUC11PAH:~# uptime
20:59:27 up 43 days, 10:42,  5 users,  load average: 2.16, 2.03, 2.21

Additional Information:

  • While the network was down, InfluxDB continued collecting data, yet Grafana displayed missing data.

  • After rebooting the machine, Grafana started displaying data normally again.

2

In results of InfluxDB query you can see data from 12:06 to 12:35 in UTC timezone, but Grafana you don’t see data from 12:06 to 12:35 in your local time zone, or I am wrong?

4

In results of InfluxDB query you can see data from 12:06 to 12:35 in UTC timezone, but Grafana you don’t see data from 12:06 to 12:35 in your local time zone

Yes, you are correct! its my actual issue