We are trying to Collect Data from Clusters (Both Prometheus and Loki).
the issue that we are facing is that Prometheus is working and suddenly all the servers which are sending data to Prometheus using Alloy starts getting a “Server Returned HTTP status 400 Bad Request: out of Bounds”.
We rename the Prometheus folder ,restart the service ,it continues to work for some time and the same issue starts again .
this issue seems to affect only Prometheus and not Loki .
this issue doesn’t happen when we are running query or anything ,happens randomly .
has anyone faced this or is there anything we can do to solve this issue