Unable to pull metrics into Grafana using InfluxDB and Telegraf

Hello,

This is my first attempt to setup a Grafana (v5.2.4) dashboard using influxdb (v1.6.2) and telegraf (v1.7.4). I’m having trouble pulling metrics into Grafana from the server running telegraf.

Grafana and influxdb are installed on the same Ubuntu 16.04.5 server. The telegraf service is running on a Windows Hyper-V Server 2016 (core).

Both the Ubuntu server and the Windows Hyper-V server are on the same LAN. I’ve also added the following ports to iptables and Windows firewall, set to allow/accept: 8086 and 3000.

Both servers are able to ping one another through IP as well as hostname.

All services are verified running on their respective servers.

I’ve been monitoring the logs, but nothing stands out as to why I’m not seeing any metrics from the Hyper-V server in Grafana.

I’ve snipped out identifiable info in the logs below.

Apologies for how I’m dumping the log info below. I’m unsure how to use the forum tools to segment the output info below into its own, easily digestible quotes.

Telegraf logs:
2018-09-10T14:59:05Z I! Hang on, flushing any cached metrics before shutdown
2018-09-10T14:59:07Z I! Starting Telegraf v1.7.4
2018-09-10T14:59:07Z I! Loaded inputs: inputs.win_perf_counters
2018-09-10T14:59:07Z I! Loaded aggregators:
2018-09-10T14:59:07Z I! Loaded processors:
2018-09-10T14:59:07Z I! Loaded outputs: influxdb
2018-09-10T14:59:07Z I! Tags enabled: host=“snip
2018-09-10T14:59:07Z I! Agent Config: Interval:10s, Quiet:false, Hostname:“snip”, Flush Interval:10s

Grafana logs:
t=2018-09-10T10:14:13-0400 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2018-09-10T10:18:17-0400 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/datasources status=302 remote_addr=snip time_ms=1 size=29 referer=
t=2018-09-10T10:37:10-0400 lvl=eror msg=“Failed to get user with id” logger=context userId=2 error=“User not found”
t=2018-09-10T10:37:10-0400 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/api/datasources/proxy/1/query status=401 remote_addr=snip time_ms=21 size=26 referer=“http://snip:3000/d/EFVUyWoik/snip?panelId=2&fullscreen&edit&orgId=1”
t=2018-09-10T10:59:09-0400 lvl=eror msg=“Failed to get user with id” logger=context userId=2 error=“User not found”
t=2018-09-10T10:59:09-0400 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/api/annotations status=401 remote_addr=snip time_ms=3 size=26 referer=“http://snip:3000/d/EFVUyWoik/snip?panelId=2&fullscreen&edit&orgId=1”

Grafana Query Inspector output:
{
“xhrStatus”: “complete”,
“request”: {
“method”: “GET”,
“url”: “api/datasources/proxy/1/query”,
“params”: {
“db”: “GRAFANA”,
“q”: “SELECT mean("value") FROM "win_cpu" WHERE ("host" = ‘snip’) AND time >= now() - 6h GROUP BY time(10s) fill(null)”,
“epoch”: “ms”
},
“data”: null,
“precision”: “ms”
},
“response”: {
“results”: [
{
“statement_id”: 0
}
]
}
}

Active Telegraf outputs:
[[outputs.influxdb]]
urls = [“http://snip:8086”] # required
database = “GRAFANA” # required
precision = “s”
timeout = “5s”
username = “snip
password = “snip

InfluxDB setup:
CREATE USER “snip” WITH PASSWORD ‘snip’ WITH ALL PRIVILEGES
GRANT WRITE ON GRAFANA TO snip

Grafana datasource setup:

Check your logs:

Status code 401 = Unauthorized.

InfluxDB auth is your problem. Don’t use “Basic Auth” in your InfluxDB datasource configuration, unless you are really protecting InfluxDB with some additional sec. layer, where basic auth is implemented.

Hi Jan,

Thanks for taking the time to help.

I unchecked “Basic Auth” from my datasource config. I also restarted the grafana-server, influxdb, and telegraf services on their respective servers, and confirmed they restarted successfully, but the change did not work. Data points are still not flowing into Grafana.

I will add, the Grafana logs are no longer showing the errors you referenced. It’s been just about half an hour since making the change, and this is all that’s displaying in the logs at this time:

t=2018-09-10T12:17:27-0400 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=

Still no metrics into Grafana though.

Logs are your best friend + check with InfluxDB CLI if you have some data ready.

Ok, I can get into the Influx CLI. What commands should I be running to verify data is ready?

I worked through the CLI commands to find some info that looks pertinent.

That is only proof that InfluxDB is running. You need to verify that you have GRAFANA db data ready in the InfluxDB = ingestion path (telegraf-influxdb) is working correctly. Please take your time and be more familiar with InfluxDB and Grafana - both of them have nice documentation online.