Unreachable Redis Sentinel in Loki

I got this error when I configured Loki to use Redis Sentinel nodes and in Loki components logs, I got these messages and errors:

level=warn ts=2023-12-31T17:46:51.26450088Z caller=experimental.go:20 msg="experimental feature in use" feature="Redis cache - chunksredis"

redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded
redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded
redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded

level=error ts=2023-12-31T17:46:51.765505306Z caller=redis_cache.go:32 msg=" error connecting to redis" name=chunksredis err="redis: all sentinels specified in the configuration are unreachable"
level=error ts=2023-12-31T17:46:51.765505307Z caller=redis_cache.go:32 msg="error connecting to redis" name=frontend.redis err="redis: all sentinels specified in the configuration are unreachable"

I thought maybe I had some network access issues or misconfiguration in my Redis Sentinels but when I use redis-cli from these hosts I can connect to these hosts!
I am using Loki microservice/distributed mode that I deployed on a K8S cluster via helm chart version: Loki-distributed-0.77.0 and app version:2.9.2.
What should I do?

1 Like