Get error 500 on promtail

Hi,
I have installed loki on one server, virtual machine.

On other about 50 virutal machines I have installed promtail.
Promtail ship logs to loki but everyday promtail shows error 500.

Here is promtail logs which show logs sent well, but in one period of time get error 500:

promtail[5617]: ts=2022-02-23T11:26:36.521883703Z caller=log.go:168 level=info msg="Seeked /var/log/audit/audit.log - &{Offset:11538849 Whence:0}"
promtail[5617]: ts=2022-02-23T11:26:36.521940919Z caller=log.go:168 level=info msg="Seeked /var/log/syslog - &{Offset:696553 Whence:0}"
promtail[5617]: level=info ts=2022-02-23T11:26:36.521955028Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/syslog
promtail[5617]: ts=2022-02-23T11:26:36.522351468Z caller=log.go:168 level=info msg="Seeked /var/log/php7.3-fpm.log - &{Offset:0 Whence:0}"
promtail[5617]: level=info ts=2022-02-23T11:26:36.522387148Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/php7.3-fpm.log
promtail[5617]: ts=2022-02-23T11:26:36.522427892Z caller=log.go:168 level=info msg="Seeked /var/log/php8.0-fpm.log - &{Offset:56 Whence:0}"
promtail[5617]: level=info ts=2022-02-23T11:26:36.522510586Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/php8.0-fpm.log
promtail[5617]: level=info ts=2022-02-23T11:26:36.522558404Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/audit/audit.log
promtail[5617]: level=warn ts=2022-02-23T19:41:05.930110628Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"
promtail[5617]: level=warn ts=2022-02-23T19:43:03.930073177Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"
promtail[5617]: level=warn ts=2022-02-23T19:51:53.704492581Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://loki_server:3100/loki/api/v1/push\": context deadline exceeded"
promtail[5617]: level=warn ts=2022-02-24T01:11:25.930369748Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"
promtail[5617]: level=warn ts=2022-02-24T01:11:31.657114062Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"
promtail[5617]: level=warn ts=2022-02-24T04:18:44.228991182Z caller=client.go:349 component=client host=loki_server:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"
promtail[5617]: ts=2022-02-24T05:25:01.532012641Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/nginx/mydomain_access.log ..."
promtail[5617]: ts=2022-02-24T05:25:01.532142404Z caller=log.go:168 level=info msg="Successfully reopened /var/log/nginx/mydomain_access.log"
promtail[5617]: ts=2022-02-24T05:25:01.661156086Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/nginx/mydomain_error.log ..."
promtail[5617]: ts=2022-02-24T05:25:01.6612567Z caller=log.go:168 level=info msg="Successfully reopened /var/log/nginx/mydomain_error.log"
promtail[5617]: ts=2022-02-24T05:25:01.688348643Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/syslog ..."
promtail[5617]: ts=2022-02-24T05:25:01.688433086Z caller=log.go:168 level=info msg="Successfully reopened /var/log/syslog"
promtail[5617]: ts=2022-02-24T05:25:01.702682013Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/nginx/access.log ..."
promtail[5617]: ts=2022-02-24T05:25:01.70274376Z caller=log.go:168 level=info msg="Successfully reopened /var/log/nginx/access.log"

If I check logs on loki server, I can see this errors but I don’t know why that happen:

loki[1431265]: level=info ts=2022-02-23T19:40:35.879505714Z caller=table.go:91 table-name=index_19046 msg="skipping compaction since we have just 3 files in storage"
loki[1431265]: level=info ts=2022-02-23T19:40:36.059669181Z caller=compactor.go:460 msg="finished compacting table" table-name=index_19046
loki[1431265]: level=warn ts=2022-02-23T19:41:05.700895511Z caller=logging.go:72 traceID=567d86315149fcc4 orgID=fake msg="POST /loki/api/v1/push (500) 5.000410323s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 217; Content-Type: application/x-protobuf; User-Agent: promtail/; "
loki[1431265]: level=warn ts=2022-02-23T19:41:05.725686249Z caller=logging.go:72 traceID=3def77c58e75f274 orgID=fake msg="POST /loki/api/v1/push (500) 5.000490706s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 371; Content-Type: application/x-protobuf; User-Agent: promtail/; "
loki[1431265]: level=warn ts=2022-02-23T19:41:05.809220083Z caller=logging.go:72 traceID=0310cdaaac83434d orgID=fake msg="POST /loki/api/v1/push (500) 5.00107738s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 683; Content-Type: application/x-protobuf; User-Agent: promtail/; "
loki[1431265]: level=warn ts=2022-02-23T19:41:05.819559646Z caller=logging.go:72 traceID=007bd2020f437acd orgID=fake msg="POST /loki/api/v1/push (500) 5.000372162s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 371; Content-Type: application/x-protobuf; User-Agent: promtail/; "
loki[1431265]: level=warn ts=2022-02-23T19:41:05.823897457Z caller=logging.go:72 traceID=400a0c61736b9cd5 orgID=fake msg="POST /loki/api/v1/push (500) 5.001099127s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 370; Content-Type: application/x-protobuf; User-Agent: promtail/; "
loki[1431265]: level=warn ts=2022-02-23T19:41:05.883511709Z caller=logging.go:72 traceID=7c9b43100817080b orgID=fake msg="POST /loki/api/v1/push (500) 5.00057697s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Content-Length: 195; Content-Type: application/x-protobuf; User-Agent: promtail/; "

loki, version 2.4.1 (branch: HEAD, revision: f61a4d261)

promtail, version 2.4.1 (branch: HEAD, revision: f61a4d261)

Appreciate for any help. :slight_smile:

I get the same logs on this issue, is there any solution?

1 Like