Loki send alert when error message comes in log

[ruman1996]

Hi,
I have nginx running in one ec2 instance and promtail is installed on that server, which sends logs to loki, which is running in another instance. I’m able to add loki as data source in grafana
and check live logs, query results and all. But I want to receive alert when some error message comes in nginx-logs, I have metrics monitoring and alerting active using prometheus and grafana.
Any reference or step by step document on implementing alerting for loki using alertmanager would help

[jlj77]

I’m new to Loki, but this sounds very much like one of the use cases I hope to implement soon. You might want to look at Ruler for this sort of functionality: Alerting | Grafana Labs

Eager to read what others suggest!

[ruman1996]

I was able to get alert based on rate of error, using old implementation of using grafana notification alert rules, but in this implementation I’m not able have dynamic labels and I want to even get which line triggered this alert ( error message in log) in email notification?
@torkel can you please add more details on loki 2.0-alertmanager configuration, because ruler config along with alert-rules is not sending alerts to alertmanager, any detailed video/guide can help.

[manishbadgujar]

Can anyone please guide me on this.

[ewelch]

The link @jlj77 provided is probably the best source for getting Loki working with a ruler. The alertmanager itself is just the Prometheus Alertmanager so you can search for any guides on how to best set that up in the Prometheus docs or other sources!

[ruman1996]

@ewelch I followed that configuration, restarted all the services ( prometheus, loki, alertmanager ) it didn’t work, I didn’t get notification. I’m attaching my config as well, we used loki as it had single store configuration now and went prod live, but now we are stuck in alerts, Any quick solution could help us.
Here are my configs:

loki-local-config.yml

---

limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h

chunk_store_config:
max_look_back_period: 0s

table_manager:
retention_deletes_enabled: false
retention_period: 0s

ruler:
storage:
type: local
local:
directory: /usr/local/bin/rules/
rule_path: /tmp/loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true

loki-alert-rules.yml

groups:

• name: example
  rules:
  • alert: HighThroughputLogStreams
    expr: (rate({job=“test”}|=“debug” [1m])) > 0
    for: 1m
    labels:
    severity: critical
    annotations:
    summary: High error from server rule

prometheus.yml

added alerts file

rule_files:

• “alert.rules.yml”
• “loki-alerts.yml”

[ruman1996]

@ewelch Can you please check, if I did any mistake in my config? I followed same guide https://grafana.com/docs/loki/latest/alerting/

[edemer75]

@ruman1996
Just found a solution with the same problem on my test environment.
Check ruler.storage.local.directory parameter description here Alerting | Grafana Labs. Put your rule files in the subdirectory /usr/local/bin/rules/YourTenantID or fake if you run loki as single instance (“auth_enabled: false” in local-loki-config.yml). After this you will see a log entry ‘caller=mapper.go:139 msg=“updating rule file” file=/tmp/loki/rules-temp/fake/rulefile.yaml’.
Hope, it is the solution.

[luciferinlove]

Try to set “for” less than the “rate” period. I hope it’ll help.