I am having a strange issue while Prometheus trying to scrape data from my Tomcat hosts.
I am getting “Get “http://HOSTNAME:30018/metrics”: read TCP :54232-> :30018: read: connection reset by peer” randomly. I have 10 API hosts and at any time only 3 or 4 API hosts will respond to the GET query. The remaining will throw a connection reset error.
Doing a tcpdump on API host shows Tomcat is sending the RESET packet:
Output when I do curl http://localhost:30018/metrics
11:26:13.445191 IP xxxxx.30018 > xxxxxx.36506: Flags [R.], seq 1, ack 89, win 512, options [nop,nop,TS val 1244760697 ecr 1244700645], length 0
We are running Prometheus 2.45.0 version and prometheus agent - jmx_prometheus_javaagent-0.18.0.jar
Any advise on what could be causing this issue?