Grafana panels empty of data

Grafana InfluxDB
1

Hello,

I’m a newbie with Grafana, InfluxDB and Telegraf but I managed to get a good number of panels setup with data from NAS appliances across the organization using SNMP. Everything was working great for several weeks until this morning around 6:30am all the panels stopped displaying data. Nothing has changed as far as configuration, everything worked Friday and over the weekend and out of the blue stopped.

I check Telegraf and verified through a test that it is getting data from SNMP resources, I can run SNMPwalk and get info from any NAS. I checked InfluxDB and see that the latest data is as of this morning when everything stopped. In Grafana I can create a panel and see metrics for the InfluxDB server that is recent so it is specific to the SNMP data.

So I’m a little stumped. If Telegraf is feeding InfluxDB host metrics into the DB I’m not sure why it stopped dumping the SNMP NAS metrics. The Telegraf config is correct to connect to the InfluxDB host.

For this setup Grafana is running on CentOS VM and InfluxDB\Telegraf is running on another CentOS VM. Doesn’t help that I’m not a Linux guru either, just know enough to get by.

Any thoughts where I can look next?

2

I found the issue.

First, I couldn’t find info in the logs for telegraf because the logs didn’t exist in the /var/log/telegraf directory. I decided to tail the /var/log/messages file and noticed a telegraf error popped up indicating the the “max-value-per-tag” was higher than the 1000000 threshold so it would discard. I then found this value was in the influxdb.conf file and replaced the value with 0, restarted influxdb and all my panels started to populate with data.

I don’t know what the implications are of setting that to unlimited, I guess we’ll cross that bridge when we get to it.