Promtail logs failed to send logs to loki

Grafana Loki
,
1

We have an issue where promtail not sending all logs to loki. When the promtail is restarted the logs where visible in grafana but after sometime it is not sending any logs. We are scraping other logs from /var/log/pods/ directory but we have only issue with particular service.

we have used helm to install we have overridden only below config.

Including @tonyswumac as i can see he is active.

  snippets:
    pipelineStages:
      - cri: {}
      - multiline:
          firstline: '^([A-Za-z-_\.])+:.+'
          max_lines: 256
2

Honestly your configuration is so simple I don’t really know what the possible issue could be. Assuming your Loki instance is still receiving other logs just fine, the problem must be promtail. Was there any error log in promtail?

I would also recommend you to try Alloy and see if it gives you the same problem.

3

Thanks @tonyswumac for looking into it. Yea we can definitely try with alloy. Just giving some background same configuration worked fine and there were no issues until we upgraded the loki-stack which in turn upgraded promtail as well. After restart of promtail it starts sending the messages as usaual after some point it stops. Could it be related to our multiline regex and this interferes other folder logs.

4

I doubt it. Multiline configuration should stop logs from being sent. I’d recommend you to first narrow down the problem (for example, if Loki is working or not when promtail seems to be not sending logs. perhaps try an API call and manually send something to Loki). Enable debug log on promtail, and try out Alloy.