Upgraded linux OS from red hat 7 to red hat 9.7 broke our working grafana/nginx/influxdb stack

[asyed123med]

OS-version
NAME=“Red Hat Enterprise Linux”
VERSION=“9.7 (Plow)”

grafana version 10.2.6

InfluxDB v1.11.8

nginx version: nginx/1.20.1

i have inherited this stack and not that familiar with grafana or influxdb, this setup has been working for a while and after the upgrade of the OS, the application is not getting data from the influxdb

i have validated the database is up and functioning, it is connected to multiple remote machines and has established connectivity

root@msplap128(/var/log/grafana)# systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/etc/systemd/system/influxdb.service; enabled; preset: disabled)
Active: activating (start) since Thu 2026-05-21 10:58:42 CDT; 8min ago
Docs: InfluxDB OSS v2 Documentation
Cntrl PID: 83328 (influxd)
Tasks: 42 (limit: 190302)
Memory: 16.5G (peak: 28.5G)
CPU: 12min 20.190s
CGroup: /system.slice/influxdb.service
└─83328 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

also

root@msplap128(/var/log/grafana)# netstat -anp | grep -i influx
tcp 0 0 127.0.0.1:8088 0.0.0.0:* LISTEN 83328/influxd
tcp6 0 0 :::8086 :::* LISTEN 83328/influxd
tcp6 0 0 :::2003 :::* LISTEN 83328/influxd
tcp6 0 0 10.51.118.85:8086 10.50.17.220:45127 ESTABLISHED 83328/influxd
tcp6 0 0 10.51.118.85:8086 10.21.80.20:65316 ESTABLISHED 83328/influxd
tcp6 0 0 10.51.118.85:8086 10.50.23.35:41870 ESTABLISHED 83328/influxd
tcp6 0 0 10.51.118.85:8086 10.21.68.161:54363 ESTABLISHED 83328/influxd

the nginx webserver is running and working fine, since i an load the page

htts://nmon.corp.companyname.com/admin

and

systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-05-19 16:47:26 CDT; 1 day 18h ago
Process: 6242 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 6244 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 6246 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)

please see attachment for the grafana config and no data page

i think the authentication maybe and issue, or some kind of internal connectivity between grafana and the db has been messed up

i have access to all the log file, please let me where i can start to look at this

or if you have a quick ideas, i would appreciate it

i am very new to this forum

abbas

[infofcc3]

I replicated your environment locally (Rocky Linux 9.7, InfluxDB v1.11.8, Grafana 10.2.6) and found two root causes.the issue i got are
1 ->Wrong bind-address in InfluxDB config
InfluxDB config changed to listen on 127.0.0.1 which means grafana cannot reach
Check your current config by → grep “bind-address” /etc/influxdb/influxdb.conf

If it shows 127.0.0.1 run this ->sed -i ‘s/bind-address = “127.0.0.1:8086”/bind-address = “:8086”/’ /etc/influxdb/influxdb.conf

2 → File Permission Error
InfluxDB service user can no longer write to its own data directory.This causes the activating stuck state you have. Proof from journalctl →
buildtsi: mkdir /var/lib/influxdb/data/_internal/_series: permission denied
Fix the permissions-> chown -R influxdb:influxdb /var/lib/influxdb
chown -R influxdb:influxdb /var/log/influxdb
chmod -R 755 /var/lib/influxdb

3 → Reset and Restart InfluxDB
systemctl reset-failed influxdb
systemctl restart influxdb
sleep 5
systemctl status influxdb

4 → Verify Connection
curl -v http://localhost:8086/ping

5 ->Restart Grafana
systemctl restart grafana-server

[asyed123med]

@infofcc3

systemctl restart grafana-server

thank you Shankar for the steps

i have backed up all the files with

find . -type f -exec ls -l {} ;

and backed up the /etc/influxdb/influxdb.conf file

i will let you know how it comes out, but weirdly the db is taking a long time to start…
anway, will update you as soon i have some log files to share with you

abbas

[asyed123med]

Shankar

so the database does not see to start here are the messages from journalctl -u influxdb

May 26 14:09:01 msplap128 systemd[1]: Starting InfluxDB is an open-source, distributed, time series database…
May 26 14:09:01 msplap128 influxd[394361]: ts=2026-05-26T19:09:01.367666Z lvl=info msg=“configured logger” log_id=135QtUDl001 format=auto level=warn
May 26 14:29:01 msplap128 systemd[1]: influxdb.service: start operation timed out. Terminating.
May 26 14:29:01 msplap128 influxd[394361]: [tcp] 2026/05/26 14:29:01 tcp.Mux: Listener at 127.0.0.1:8088 failed failed to accept a connection, closing all listeners - acce>
May 26 14:29:01 msplap128 systemd[1]: influxdb.service: Failed with result ‘timeout’.
May 26 14:29:01 msplap128 systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
May 26 14:29:01 msplap128 systemd[1]: influxdb.service: Consumed 3min 7.981s CPU time, 28.4G memory peak.
May 26 14:29:02 msplap128 systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 1.
May 26 14:29:02 msplap128 systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
May 26 14:29:02 msplap128 systemd[1]: influxdb.service: Consumed 3min 7.981s CPU time, 28.4G memory peak.
May 26 14:29:02 msplap128 systemd[1]: Starting InfluxDB is an open-source, distributed, time series database…
May 26 14:29:02 msplap128 influxd[394846]: ts=2026-05-26T19:29:02.165872Z lvl=info msg=“configured logger” log_id=135S1lqG001 format=auto level=warn

in addition i will give the content of /etc/influxdb/influxdb.conf

grep -vE ‘^\s*(#|$)’ /etc/influxdb/influxdb.conf
reporting-disabled = true
[meta]
dir = “/var/lib/influxdb/meta”
[data]
dir = “/var/lib/influxdb/data”
wal-dir = “/var/lib/influxdb/wal”
index-version = “tsi1”
series-id-set-cache-size = 100
[coordinator]
log-queries-after = “10s”
[retention]
[shard-precreation]
[monitor]
[http]
bind-address = “:8086”
auth-enabled = true
suppress-write-log = true
https-enabled = true
https-certificate = “/etc/pki/influxdb/server.crt”
https-private-key = “/etc/pki/influxdb/private/server.key”
[logging]
level = “warn”
[subscriber]
[[graphite]]
enabled = true
database = “graphite”
retention-policy = “100days”
bind-address = “:2003”
protocol = “tcp”
consistency-level = “one”
templates = [
“puppetlabs.. .host.measurement”,
“puppetlabs...* .host.measurement.field”,
“puppetlabs.... .host.measurement.measurement.field*”,
]
[[collectd]]
[[opentsdb]]
[[udp]]
[continuous_queries]
[tls]

needless to say influxdb is not starting

root@msplap128(/tmp/ab)# systemctl status influxdb.service
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/etc/systemd/system/influxdb.service; enabled; preset: disabled)
Active: activating (start) since Tue 2026-05-26 14:29:02 CDT; 19min ago
Docs: InfluxDB OSS v2 Documentation
Cntrl PID: 394846 (influxd)
Tasks: 47 (limit: 190302)
Memory: 27.8G (peak: 28.5G)
CPU: 2min 57.888s
CGroup: /system.slice/influxdb.service
└─394846 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

May 26 14:29:02 msplap128 systemd[1]: Starting InfluxDB is an open-source, distributed, time series database…
May 26 14:29:02 msplap128 influxd[394846]: ts=2026-05-26T19:29:02.165872Z lvl=info msg=“configured logger” log_id=135S1lqG001 format=auto level=warn

seemed to just have timed out

[asyed123med]

@infofcc3

curl -v http://localhost:8086/ping

root@msplap128(/tmp/ab)# curl -v http://localhost:8086/ping

• Trying ::1:8086…
• Connected to localhost (::1) port 8086 (#0)

GET /ping HTTP/1.1
Host: localhost:8086
User-Agent: curl/7.76.1
Accept: /

• Mark bundle as not supporting multiuse
• HTTP 1.0, assume close after body
  < HTTP/1.0 400 Bad Request
  <
  Client sent an HTTP request to an HTTPS server.
• Closing connection 0

[asyed123med]

Shankar
please disregard the last few messages…

i have got the database to start, seems to be running also nginx is running grafana-server is running

seems i am getting authentication error from grafana-server datasouce to connect to the database

let me play with it some more and will put the results, i think i will need to get the username and password