Creating Alerts

Here is my log pipeline:

{filename="/var/log/laravel/laravel.log"}|~"(?i)error"

When I try to create Alert rule I get the following error:

Failed to evaluate queries and expressions: failed to execute conditions: failed to query data: Run out of attempts while querying the server.

But when I try to run the same thing in Grafana Explore I get nice output which shows everything I need. Why can’t same be accomplished while creating rules?

Hey @kilio

Alerting rules can only be defined with metric queries. You could change your query to:

sum by (some_label) (rate({filename="/var/log/laravel/laravel.log"}|~"(?i)error"[1m]))

Loki uses Prometheus’ alerting rules code, so you can view more detail on alerting rules here:

1 Like

Hi @dannykopping,

This was very much helpful, however, while creating the Alert rule from the panel using the above provided metric query I get the following Health errors

Invalid format of evaluation results for the alert definition A: looks like time-series data, only reduced data can be alerted on.

Failed to query data: run out of attempts while querying data.

For the first error, I changed Query Type and defined time range but the second one doesn’t seem to be gone away even though I tried all the combinations with alerting if all values are null or no data received.

Which interface are you using? I’m not too familiar with the Grafana-based mechanisms for creating alerts unfortunately.

Hi @dannykopping,

Yes, I am focusing on Grafana alerts as Loki/Cortex rules can’t be defined. I facing the same problems as reported here, If Cortex/Loki managed alert Rule type is selected there is no data sources to select alongside even though Loki is added as Datasource.

I think you should ask over at the Grafana #support forum channel - we in the Loki team do not have much involvement on the frontend side of things

But technically rules (written in yaml format) placed in Loki container and ruler configured to send alerts to Alertmanager would need to work without even using Grafana right?

This is ruler config:

ruler:
  alertmanager_url: http://localhost:9093
  enable_alertmanager_v2: true
  enable_api: true
  enable_sharding: true
  ring:
    kvstore:
      store: inmemory
  rule_path: /loki/rules
  storage:
    type: local
    local:
      directory: /tmp/rules

This is rules.yaml:

groups:
    - name: should_fire
      rules:
        - alert: Error
          expr: sum by (job) (rate({filename="/var/log/laravel/laravel.log"}|~"(?i)error"[1m]))
          labels:
            category: app-logs
            severity: Critical
            environment: development
          annotations:
            action: action
            description: Application is logging errors
            impact: impact
            summary: Laravel application has error
            title: 'Alert: Application error'

I have this setup and when error is logged into laravel logs no alert is fired and I can’t understand why it is not.

In ruler config, replaced http://localhost:9093 with http://alertmanager:9093 and alert was fired to the right notification channel.