How to know if promtail is running and sending logs to central cluster

Grafana Loki
, ,
1

Hi all,
I am working with loki stack (promtail, loki, prometheus alertmanager, grafana) and using distributed system to scrape logs and send alert using alertmanager. Let me explain complete scenario :slightly_smiling_face:

  1. I have multiple clusters with different applications and promtail agent.
  2. A common cluster with loki, grafana, prometheus alertmanager.

Currently I am gettting logs from each cluster’s promtail via loki and sending alert to slack using alertmanager.
My problem is How will I know if any of the cluster’s promtail agent fails to scrap logs ?
How can I regularly monitor if my promtails are running fine in each cluster and we are getting logs ? :pensive:

Any Idea/method is appreciated. Thanks.

2

There are several things you can do, and it depends on the level of granularity you need.

  1. Promtail does expose metrics on its own. You can scrape metrics from promtail, send them to whatever metrics platform you are using, and generate alert from there. You should be able to see metrics on dropped logs or parse erros.

  2. If you are looking to identify when a promtail agent fails entirely, you can setup alerts and do a count based on each promtail source and alert when the count is low. You’ll also need to use a label to identify each promtail, of course.

1 Like