Loki spams deletion-related error messages

Grafana Loki
1

I don’t understand the log messages Loki is producing (see below) and I don’t know what to do about them … sorry, I’m currently setting up my first instance here and I have not found enlightenment in the docs.

It would be so cool if someone can give me a hint, I’m really stumped here

This is my loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h


limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h    # 7 days

And these are the logs in question:

2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.297347433Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296795124Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296777872Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296609762Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296596978Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296590897Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296572163Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.29642427Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296408721Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"	
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296398242Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"	
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296385428Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"	
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296281336Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296278882Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296258965Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296261049Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296081207Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296059828Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296032337Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296012561Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2

Having the same issue here /var/log/messages is spammed :frowning:

3

It sounds like v2.7.1 has a fix for this.

4

Error Version:

/ # /usr/bin/loki -version
loki, version 2.7.0 (branch: HEAD, revision: 1b627d880)
  build user:       root@953d84d26964
  build date:       2022-11-10T23:53:03Z
  go version:       go1.19.2
  platform:         linux/amd64

Fixed version:

/ # loki -version
loki, version 2.7.1 (branch: HEAD, revision: e0af1cc8a)
  build user:       root@00ddf1dbd331
  build date:       2022-12-09T23:31:00Z
  go version:       go1.19.2
  platform:         linux/amd64

I can confirm this looks to be fixed, the spamming has stopped :slight_smile: . I have only done about 15 min of testing, may need a longer period, but looks good so far.

5

The issue seems to be fixed indeed! Thank you so much!