Prometheus Java Agent Connection reset Errors

Grafana Prometheus
1

Hi,
I am having a strange issue while Prometheus trying to scrape data from my Tomcat hosts.
I am getting “Get “http://HOSTNAME:30018/metrics”: read TCP :54232-> :30018: read: connection reset by peer” randomly. I have 10 API hosts and at any time only 3 or 4 API hosts will respond to the GET query. The remaining will throw a connection reset error.
Doing a tcpdump on API host shows Tomcat is sending the RESET packet:
Output when I do curl http://localhost:30018/metrics
11:26:13.445191 IP xxxxx.30018 > xxxxxx.36506: Flags [R.], seq 1, ack 89, win 512, options [nop,nop,TS val 1244760697 ecr 1244700645], length 0

We are running Prometheus 2.45.0 version and prometheus agent - jmx_prometheus_javaagent-0.18.0.jar

Any advise on what could be causing this issue?
Thank you

2

If any one ends up with this issue, solution is to check Tomcat config files and setup params such as maxAge and switching to the new pool for each wars

factory=“org.apache.tomcat.jdbc.pool.DataSourceFactory”
maxAge=“14400000”