Loki spams deletion-related error messages

I don’t understand the log messages Loki is producing (see below) and I don’t know what to do about them … sorry, I’m currently setting up my first instance here and I have not found enlightenment in the docs.

It would be so cool if someone can give me a hint, I’m really stumped here

This is my loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h


limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h    # 7 days

And these are the logs in question:

2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.297347433Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296795124Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296777872Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296609762Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296596978Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296590897Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296572163Z caller=delete_requests_client.go:201 msg="error getting delete requests from the store" err="Get \"http://127.0.0.1:3100/loki/api/v1/delete\": context canceled"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.29642427Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296408721Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"	
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296398242Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"	
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296385428Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"	
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296281336Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296278882Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296258965Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296261049Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296081207Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296059828Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
2022-12-06 12:57:46 ts=2022-12-06T11:57:46.296032337Z caller=spanlogger.go:80 user=fake level=error msg="failed loading deletes for user" err="unexpected status code: 404"
2022-12-06 12:57:46 level=error ts=2022-12-06T11:57:46.296012561Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"

Having the same issue here /var/log/messages is spammed :frowning:

It sounds like v2.7.1 has a fix for this.

Error Version:

/ # /usr/bin/loki -version
loki, version 2.7.0 (branch: HEAD, revision: 1b627d880)
  build user:       root@953d84d26964
  build date:       2022-11-10T23:53:03Z
  go version:       go1.19.2
  platform:         linux/amd64

Fixed version:

/ # loki -version
loki, version 2.7.1 (branch: HEAD, revision: e0af1cc8a)
  build user:       root@00ddf1dbd331
  build date:       2022-12-09T23:31:00Z
  go version:       go1.19.2
  platform:         linux/amd64

I can confirm this looks to be fixed, the spamming has stopped :slight_smile: . I have only done about 15 min of testing, may need a longer period, but looks good so far.

The issue seems to be fixed indeed! Thank you so much!

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.