Hello, I have that issue and I can’t fix it, the database is Influxdb, after some minutes all data sources give me (Network error:Bad Gateway)
All the other topics from grafana comunity didn’t helped me.
Any solution?
Thank you,
When this happens, what is the output of the command (as root):
netstat -lptn | grep 8086
Basically, is influx running?
Antony.
I have no result, is open only 8088 port, port 8086 is not showing.
So, show us the uncommented parts of your influx config file, and look for
anything in the log file when it starts up to tell you what it’s doing.
Antony.
This are uncommented parts:
[meta]
dir = “/var/lib/influxdb/meta”
[data]
dir = “/var/lib/influxdb/data”
wal-dir = “/var/lib/influxdb/wal”
max-series-per-database = 0
Influx is starting ok and when i start grafana also i have this error: “lvl=info msg=“Database locked, sleeping then retrying” logger=sqlstore error=“database is locked” retry=0” and then connection is refused.
Marian,
Also i have this errors:
Aug 10 11:41:11 aerosensors kernel: lowmem_reserve: 0 0 0 0
Aug 10 11:41:11 aerosensors kernel: Node 0 DMA: 14kB (U) 08kB 116kB (U) 032kB 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15892kB
Aug 10 11:41:11 aerosensors kernel: Node 0 DMA32: 924kB (UEM) 308kB (UEM) 3216kB (UEM) 732kB (UEM) 464kB (EM) 22128kB (UE) 17256kB (UM) 5512kB (UM) 611024kB (UM) 02048kB 14096kB (M) = 77888kB
Aug 10 11:41:11 aerosensors kernel: Node 0 Normal: 5874kB (UEM) 6068kB (UEM) 18816kB (UEM) 5332kB (UE) 764kB (UE) 12128kB (UEM) 36256kB (U) 26512kB (UM) 191024kB (UM) 12048kB (M) 0*4096kB = 57916kB
Aug 10 11:41:11 aerosensors kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 10 11:41:11 aerosensors kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 10 11:41:11 aerosensors kernel: 4140 total pagecache pages
Aug 10 11:41:11 aerosensors kernel: 1344 pages in swap cache
Aug 10 11:41:11 aerosensors kernel: Swap cache stats: add 27103878, delete 27102673, find 3581302/5492603
Aug 10 11:41:11 aerosensors kernel: Free swap = 0kB
Aug 10 11:41:11 aerosensors kernel: Total swap = 2097148kB
Aug 10 11:41:11 aerosensors kernel: 5242750 pages RAM
Aug 10 11:41:11 aerosensors kernel: 0 pages HighMem/MovableOnly
This are uncommented parts:
[meta]
dir = “/var/lib/influxdb/meta”
[data]
dir = “/var/lib/influxdb/data”wal-dir = “/var/lib/influxdb/wal”
max-series-per-database = 0
So, this sounds like you do not have anything in the “http” section
uncommented (such as “enabled = true”).
Try ensuring that it is enabled, and the bind port is 8086, and see if that
helps.
Influx is starting ok
What does the log file tell you about which ports it is binding to, and which
services it is starting up?
Antony.
I follow your instructions but is the same problem.
I saw this log about influx when its try to open service:
ts=2021-08-10T09:19:26.848665Z lvl=info msg=“Open store (end)” log_id=0VsqTMOG000 service=store trace_id=0VsqTMnW000 op_name=tsdb_open op_event=end op_elapsed=137653.815ms
ts=2021-08-10T09:19:26.855618Z lvl=info msg=“Opened service” log_id=0VsqTMOG000 service=subscriber
ts=2021-08-10T09:19:26.855653Z lvl=info msg=“Starting monitor service” log_id=0VsqTMOG000 service=monitor
ts=2021-08-10T09:19:26.855666Z lvl=info msg=“Registered diagnostics client” log_id=0VsqTMOG000 service=monitor name=build
ts=2021-08-10T09:19:26.855676Z lvl=info msg=“Registered diagnostics client” log_id=0VsqTMOG000 service=monitor name=runtime
ts=2021-08-10T09:19:26.855681Z lvl=info msg=“Registered diagnostics client” log_id=0VsqTMOG000 service=monitor name=network
ts=2021-08-10T09:19:26.855702Z lvl=info msg=“Registered diagnostics client” log_id=0VsqTMOG000 service=monitor name=system
ts=2021-08-10T09:19:26.855732Z lvl=info msg=“Starting precreation service” log_id=0VsqTMOG000 service=shard-precreation check_interval=10m advance_period=30m
ts=2021-08-10T09:19:26.855744Z lvl=info msg=“Starting snapshot service” log_id=0VsqTMOG000 service=snapshot
ts=2021-08-10T09:19:26.855758Z lvl=info msg=“Starting continuous query service” log_id=0VsqTMOG000 service=continuous_querier
ts=2021-08-10T09:19:26.855774Z lvl=info msg=“Starting HTTP service” log_id=0VsqTMOG000 service=httpd authentication=true
ts=2021-08-10T09:19:26.855806Z lvl=info msg=“opened HTTP access log” log_id=0VsqTMOG000 service=httpd path=stderr
ts=2021-08-10T09:19:26.855812Z lvl=info msg=“Auth is enabled but shared-secret is blank. BearerAuthentication is disabled.” log_id=0VsqTMOG000 service=httpd
ts=2021-08-10T09:19:26.855767Z lvl=info msg=“Storing statistics” log_id=0VsqTMOG000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2021-08-10T09:19:26.866223Z lvl=info msg=“Listening on HTTP” log_id=0VsqTMOG000 service=httpd addr=[::]:8086 https=false
ts=2021-08-10T09:19:26.866279Z lvl=info msg=“Starting retention policy enforcement service” log_id=0VsqTMOG000 service=retention check_interval=30m
ts=2021-08-10T09:19:26.867714Z lvl=info msg=“Listening for signals” log_id=0VsqTMOG000
ts=2021-08-10T09:19:26.898519Z lvl=info msg=“Sending usage statistics to usage.influxdata.com” log_id=0VsqTMOG000
I follow your instructions but is the same problem.
What does “netstat -lptn | grep 8086” show now?
I saw this log about influx when its try to open service:
ts=2021-08-10T09:19:26.866223Z lvl=info msg=“Listening on HTTP”
log_id=0VsqTMOG000 service=httpd addr=[::]:8086 https=false
That tells me that it has started an HTTP service listening on port 8086.
Antony.
Yes, the measurements are written in databases, but in grafana is the same error: “Bad gateway”!. I saw that CPU is above 95% used by influx, maybe if i increase the CPU of VM it will solve the problem ?
Marian,
Tell us what precisely you have entered for the database connection URL in
Grafana, so that it should find InfluxDB.
Also please confirm whether Grafana and Influx are running on the same machine
or not.
Antony.
Both are in the same machine, thank you so much for help until now, finally is worked, i increased de RAM and CPU and now is working normally.
Best regards,
Marian.
For the benefit of anyone who finds this thread in future and is having a
similar problem, please can you let us know:
-
what were the previous CPU and RAM settings of your VM (with the problem)
-
what have you changed these to (now working)?
Thanks,
Antony.
First change was ulimit in linux server machine from 1024 to 4096 because the first error in influx was "to many open files " and I change it with the following commands:
ulimit -Hn 4096
ulimit -Sn 4096
And then I increase the following resources:
CPU was 2 core 1 processor and now 5 core 1 processor.
RAM was 20 GB and now 30 GB.
Now is working without errors.
Marian,
