# Issues with reconnect, promtail towards loki-distributed

**URL:** <https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014>\
**Category:** Grafana Loki\
**Tags:** promtail\
**Created:** [January 15, 2024, 1:21pm UTC](https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014 "2024-01-15T13:21:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sojjan1337](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/sojjan1337/32/39317_2.png) [@sojjan1337](https://community.grafana.com/u/sojjan1337)\
**Post date:** [January 15, 2024, 1:21pm UTC](https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014/1 "2024-01-15T13:21:21Z")

</div>

Hi!

Using loki-distributed helm-chart 0.78.0 and promtail 6.15.3

Every-time i take down loki and up again Promtail cannot reconnect.  
I have to restart promtail again, then it works fine again.

Output from promtail:  
First i get 404 when loki closes, then when it’s up again i get 500.

```auto
`level=warn ts=2024-01-12T08:52:52.776069987Z caller=client.go:419 component=client host=` `loki.test.com` `msg="error sending batch, will retry" status=500 
tenant= error="server returned HTTP status 500 Internal Server Error 
(500): empty ring"`

```

Similar problems:

> <https://github.com/grafana/loki/issues/4202>
>
> \`promtail\_1 | level=warn ts=2021-08-23T07:22:20.680240681Z caller=client.go:344… component=client host=10.10.12.189:18080 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): empty ring"
> \`
> 
> Why I receive this error?
> After restart Promtail connection resumes

> <https://github.com/grafana/loki/issues/7614>
>
> Hello!
> 
> Faced a problem when installing a loki cluster.
> The cluster consists …of 5 nodes:
> 1) Nginx proxy
> 2) loki reader x2
> 3) loki writer x2
> 
> All have the same configuration file installed with a difference in the target item:
> \`\`\`
> auth\_enabled: false
> target: "read"
> 
> memberlist:
> node\_name: "loki-reader2.example.com"
> join\_members:
> - "loki-reader1.example.com"
> - "loki-reader2.example.com"
> - "loki-writer1.example.com"
> - "loki-writer2.example.com"
> dead\_node\_reclaim\_time: "30s"
> gossip\_to\_dead\_nodes\_time: "15s"
> left\_ingesters\_timeout: "30s"
> bind\_addr: 
> - "0.0.0.0"
> bind\_port: 7946
> 
> server:
> grpc\_listen\_port: 9095
> http\_listen\_port: 3100
> http\_server\_read\_timeout: "1h"
> http\_server\_write\_timeout: "5m"
> http\_server\_idle\_timeout: "5m"
> grpc\_server\_max\_recv\_msg\_size: 41943040
> grpc\_server\_max\_send\_msg\_size: 41943040
> log\_level: debug
> common:
> path\_prefix: /loki
> storage:
> filesystem:
> chunks\_directory: /loki/chunks
> rules\_directory: /loki/rules
> replication\_factor: 1
> ring:
> kvstore:
> store: memberlist
> 
> querier:
> max\_concurrent: 10
> query\_ingesters\_within: 5h
> query\_timeout: 1h
> engine:
> timeout: 5m
> 
> schema\_config:
> configs:
> - from: 2022-10-01
> store: boltdb-shipper
> object\_store: aws
> schema: v11
> index:
> prefix: index\_
> period: 24h
> 
> compactor:
> retention\_enabled: true
> retention\_delete\_delay: "1h"
> delete\_request\_cancel\_period: "1h"
> working\_directory: /loki/compactor
> shared\_store: s3
> retention\_delete\_worker\_count: 10
> compactor\_ring:
> kvstore:
> store: memberlist
> 
> storage\_config:
> boltdb\_shipper:
> active\_index\_directory: /loki/index
> cache\_location: /loki/index-cache
> shared\_store: s3
> aws:
> bucketnames: loki
> endpoint: https://s3.example.com
> access\_key\_id: -----------------
> secret\_access\_key: ----------------------
> insecure: true
> http\_config:
> idle\_conn\_timeout: 5m
> response\_header\_timeout: 0s
> insecure\_skip\_verify: true
> s3forcepathstyle: true
> 
> limits\_config:
> retention\_period: "4320h"
> per\_stream\_rate\_limit: "500MB"
> per\_stream\_rate\_limit\_burst: "1000MB"
> max\_concurrent\_tail\_requests: 10000
> ingestion\_rate\_mb: 100
> ingestion\_burst\_size\_mb: 100
> max\_entries\_limit\_per\_query: 100000
>   
> ingester:
> lifecycler:
> join\_after: "60s"
> observe\_period: "5s"
> ring:
> replication\_factor: 1
> kvstore:
> store: memberlist
> final\_sleep: "0s"
> max\_chunk\_age: "240h"
> \`\`\`
> All healthchecks pass, the service starts.
> According to the logs, Loki was able to locate all the members of the ring.
> But when trying to connect the cluster to grafana, a 500 error:
> 
> \`Loki: Internal Server Error. 500. empty ring\`
> 
> In the service logs I see the following:
> 
> \`\`\`
> ts=2022-11-07T13:12:13.926196155Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
> ts=2022-11-07T13:12:13.927323183Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=35.563µs Summary.QueueTime=0s
> ts=2022-11-07T13:12:13.927996495Z caller=spanlogger.go:80 user=fake method=query.Label level=info org\_id=fake latency=fast query\_type=labels length=1h0m0s duration=35.563µs status=500 label= throughput=0B total\_bytes=0B total\_entries=0
> level=debug ts=2022-11-07T13:12:13.931994093Z caller=grpc\_logging.go:46 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=105.52µs msg="gRPC (success)"
> level=error ts=2022-11-07T13:12:13.932669943Z caller=retry.go:73 org\_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = empty ring\\n"
> ts=2022-11-07T13:12:13.934128211Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
> ts=2022-11-07T13:12:13.934462835Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=13.737µs Summary.QueueTime=0s
> ts=2022-11-07T13:12:13.935224122Z caller=spanlogger.go:80 user=fake method=query.Label level=info org\_id=fake latency=fast query\_type=labels length=1h0m0s duration=13.737µs status=500 label= throughput=0B total\_bytes=0B total\_entries=0
> level=debug ts=2022-11-07T13:12:13.936167945Z caller=grpc\_logging.go:46 duration=42.182µs method=/frontendv2pb.FrontendForQuerier/QueryResult msg="gRPC (success)"
> level=error ts=2022-11-07T13:12:13.936833055Z caller=retry.go:73 org\_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = empty ring\\n"
> ts=2022-11-07T13:12:13.938052577Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
> ts=2022-11-07T13:12:13.938284912Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=45.012µs Summary.QueueTime=0s
> ts=2022-11-07T13:12:13.938793954Z caller=spanlogger.go:80 user=fake method=query.Label level=info org\_id=fake latency=fast query\_type=labels length=1h0m0s duration=45.012µs status=500 label= throughput=0B total\_bytes=0B total\_entries=0
> level=debug ts=2022-11-07T13:12:13.939859299Z caller=grpc\_logging.go:46 duration=44.651µs method=/frontendv2pb.FrontendForQuerier/QueryResult msg="gRPC (success)"
> level=error ts=2022-11-07T13:12:13.940325981Z caller=retry.go:73 org\_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = empty ring\\n"
> ts=2022-11-07T13:12:13.941421855Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
> ts=2022-11-07T13:12:13.941910948Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=7.473µs Summary.QueueTime=0s
> ts=2022-11-07T13:12:13.942403329Z caller=spanlogger.go:80 user=fake method=query.Label level=info org\_id=fake latency=fast query\_type=labels length=1h0m0s duration=7.473µs status=500 label= throughput=0B total\_bytes=0B total\_entries=0
> level=debug ts=2022-11-07T13:12:13.943278467Z caller=grpc\_logging.go:46 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=28.428µs msg="gRPC (success)"
> level=error ts=2022-11-07T13:12:13.943797077Z caller=retry.go:73 org\_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = empty ring\\n"
> ts=2022-11-07T13:12:13.944847003Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
> ts=2022-11-07T13:12:13.945103343Z caller=spanlogger.go:80 user=fake method=query.Label level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=8.999µs Summary.QueueTime=0s
> ts=2022-11-07T13:12:13.945650128Z caller=spanlogger.go:80 user=fake method=query.Label level=info org\_id=fake latency=fast query\_type=labels length=1h0m0s duration=8.999µs status=500 label= throughput=0B total\_bytes=0B total\_entries=0
> level=debug ts=2022-11-07T13:12:13.946855527Z caller=grpc\_logging.go:46 duration=50.382µs method=/frontendv2pb.FrontendForQuerier/QueryResult msg="gRPC (success)"
> level=error ts=2022-11-07T13:12:13.947119279Z caller=retry.go:73 org\_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = empty ring\\n"
> level=error ts=2022-11-07T13:12:13.947827961Z caller=stats.go:57 component=frontend msg="failed to record query metrics" err="expected one of the \*LokiRequest, \*LokiInstantRequest, \*LokiSeriesRequest, \*LokiLabelNamesRequest, got "
> \`\`\`
> 
> At the same time, the installation of loki on one server works correctly, logs are written and read.
> 
> Tell me, please, where to dig, I can not find any problems.
> 
> Also, I tried to connect the distributor to the loki, the logging does not happen either. A 404 error is thrown without any explanation:
> \`\`\`
> level=debug ts=2022-11-07T11:35:11.566374631Z caller=logging.go:76 traceID=7dd739ced5088cbe orgID=fake msg="POST /loki/api/v1/push (404) 87.265µs"
> level=debug ts=2022-11-07T11:35:11.653902837Z caller=logging.go:76 traceID=766c347a31371e33 orgID=fake msg="POST /loki/api/v1/push (404) 76.161µs"
> level=debug ts=2022-11-07T11:35:11.906294651Z caller=logging.go:76 traceID=58fe0b96a10b3d6b orgID=fake msg="POST /loki/api/v1/push (404) 177.982µs"
> level=debug ts=2022-11-07T11:35:11.922494924Z caller=logging.go:76 traceID=0f26b052f9416264 orgID=fake msg="POST /loki/api/v1/push (404) 88.356µs"
> \`\`\`
> At the same time, the s3 storage is available, another lock is written to it in parallel, which is located on a separate server.

I have also tried to add this i promtail:

```auto
config:
  clients:
    backoff_config:
    min_period: 500ms
    max_period: 20m
    max_retries: 100

```

---

<div class="post-metadata">

**Author:** ![sojjan1337](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/sojjan1337/32/39317_2.png) [@sojjan1337](https://community.grafana.com/u/sojjan1337)\
**Post date:** [January 16, 2024, 7:04pm UTC](https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014/2 "2024-01-16T19:04:02Z")

</div>

Also i can get this error in promtail:

```auto

HTTP status 404 page not found

```

then after loki comes up:  
`sending batch, will retry status=502 tenant= error"server returned HTTP status 502 Bad Gateway.`

It is using the same dns name to push logs, but somehow promtail cannot reconnect. What is different with loki when it goes down and up again, “reinstalled”?

---

<div class="post-metadata">

**Author:** ![sojjan1337](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/sojjan1337/32/39317_2.png) [@sojjan1337](https://community.grafana.com/u/sojjan1337)\
**Post date:** [January 19, 2024, 3:38pm UTC](https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014/3 "2024-01-19T15:38:02Z")

</div>

It seems that it works as it should, even though i get those errors. I still can ingest logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [January 18, 2025, 3:38pm UTC](https://community.grafana.com/t/issues-with-reconnect-promtail-towards-loki-distributed/112014/4 "2025-01-18T15:38:44Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.