Bad response status 422 Unprocessable Entity

Grafana Loki
1

Hello everyone,

I’ve encountered an issue with the Loki ruler and my Mimir Alertmanager setup. Despite configuring the Loki ruler to forward alerts to the Mimir Alertmanager, it seems that the integration isn’t functioning correctly. I’m receiving error messages in both the Loki backend log and the Mimir backend. The errors read as follows:

Domain is altered for data privacy, service.io is just a placeholder!

In Loki backend log:

level=error ts=2024-04-04T09:48:16.641353044Z caller=notifier.go:532 user=anonymous alertmanager=https://mimir-backend.service.io/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 422 Unprocessable Entity"

In Mimir backend log:

ts=2024-04-04T09:48:53.401865562Z caller=notifier.go:530 level=error user=anonymous alertmanager=http://10.41.0.228:20330/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://10.41.0.228:20330/alertmanager/api/v2/alerts\": EOF"

Here’s some context: Both Loki and Mimir are running as services on my Nomad cluster, with Loki in SSD mode and Mimir in read-write mode. Additionally, these services are exposed via a Traefik proxy. I attempted to enable multi-tenancy on both Mimir and Loki, but this adjustment didn’t alter the error message.

Below are my configurations

Mimir config:


ruler:
  rule_path: /alloc/data/ruler
  alertmanager_url: https://mimir-backend.service.io/alertmanager
  ring:
    instance_addr: 10.41.0.228

ruler_storage:
  backend: local
  local:
    directory: /local/ruler

alertmanager:
  data_dir: /alloc/data/alertmanager
  external_url: https://mimir-backend.service.io/alertmanager
  sharding_ring:
    instance_addr: 10.41.0.228
    zone_awareness_enabled: true
    instance_availability_zone: eu-de-01

Mimir test rule is saved in /local/ruler/anonymous:

groups:
  - name: dead man switch
    rules:
      - alert: dead man switch
        expr: vector(1)
        labels:
          severity: none

Loki ruler config:

ruler:
  external_url: grafana.service.io
  datasource_uid: loki
  rule_path: /alloc/data/ruler
  alertmanager_url: https://mimir-backend.service.io/alertmanager
  enable_alertmanager_v2: true
  enable_sharding: true
  wal:
    dir: /alloc/data/ruler-wal
  storage:
    type: local
    local:
      directory: /local/ruler

Loki test rule is saved in /local/ruler/anonymous:

groups:
  - name: dead man switch loki
    rules:
      - alert: dead man switch loki
        expr: vector(1)
        labels:
          severity: none

Does anyone have insights into why this configuration is causing errors? Any help would be greatly appreciated.Thank you!

2

This is probably not Loki related. I would double check and make sure your alert manager is actually working.

If you need to share alertmanager between Loki and Mimir, it may be worth considering just standing one up outside of Loki and Mimir cluster, especially if you have auth enabled for either Loki or Mimir.

3

Hi there,

thanks for your response!
Unfortunately, I already tried that solution, but it resulted in the same outcome.

After some consideration, I decided to check the HTTP request sent by Loki to the Alertmanager, and luckily, I found the problem. Using tcpdump, I was able to examine the contents of the request and discovered the actual error message. It stated that the scheme of the “external_url” field was missing! I quickly fixed that, and my new ruler config looks like this now:

ruler:
  external_url: https://grafana.service.io
  datasource_uid: loki
  rule_path: /alloc/data/ruler
  alertmanager_url: https://mimir-backend.service.io/alertmanager
  enable_alertmanager_discovery: true
  enable_alertmanager_v2: true
  enable_sharding: true
  wal:
    dir: /alloc/data/ruler-wal
  storage:
    type: local
    local:
      directory: /local/ruler

Adding “http://” solved the issue! It would be helpful if the log message provided by Loki or the Alertmanager could indicate the specific HTTP error response instead of the generic error code 422. I will open an issue in the Loki repository regarding this, and hopefully, it can be addressed in future versions.