Hey all we have had mimir and grafana agents deployed with no issues for the last few months.
Over the weekend i started noticing we were being flooded with the following log messages:
ProcessID: Source: Grafana Agent EventID: 1 Level: Warning Message: agent=prometheus instance=e161a8a57c2d31cb415b276fb63a5ea9 component=remote level=warn remote_name=e161a8-805a16 url=https://mimir.thetmg.com/api/v1/push msg=“Failed to send batch, retrying” err=“server returned HTTP status 500 Internal Server Error: failed pushing to ingester: rpc error: code = Unavailable desc = error reading from server: read tcp 172.20.0.5:33840->172.20.0.3:9095: use of closed network connection”
How can we fix this issue? i looked it up but could not see any answers related to it