Hi all,
I have deployed Static mode Kubernetes operator. And I am using this for pushing the pod logs to loki.
Recently I found out that grafana-agent-logs pod is taking a lot of memory and it is also not able to push the logs.
after checking the logs of the config-reloader container I found some error messages.
level=error ts=2024-03-26T06:33:34.799302016Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:33:34.799311483Z caller=reloader.go:384 msg="Failed to trigger reload. Retrying." err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:34:04.803619827Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
On force deleting this pod the error msg in the config-reloader container is gone and grafana-agent-logs pod’s memory also comes down, it also starts pushing logs to loki.
Can you help me with the reason for the issue and with a better solution, as the current solution is not very optimal.