Should failing frontends/ingestors restart in single-binary mode?

Grafana Loki
1

I keep managing to cause Loki to fall over in single-binary mode when logging large queries.

Doing so causes the following log lines:

level=warn ts=2022-02-22T09:14:34.704905388Z caller=pool.go:184 msg="removing frontend failing healthcheck" addr=10.227.4.227:9095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=warn ts=2022-02-22T09:14:36.15422671Z caller=pool.go:184 msg="removing ingester failing healthcheck" addr=10.227.4.227:9095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=warn ts=2022-02-22T09:14:36.154882028Z caller=pool.go:184 msg="removing ingester failing healthcheck" addr=10.227.4.227:9095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

These then never recover and Loki sits there running, but not performing anything useful.
Is this expected behaviour, or is there some configuration problem that’s likely causing this?

Thank you!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.