Loki Log is flooded with "error deleting chunk" and "failed to delete key"

Grafana Loki
1

The logs of loki are flooded with “error deleting chunk” and “failed to delete key”. What can I do?

Here is a short excerpt:

2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490376481Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0007" value=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490366221Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490348506Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006" value=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490337836Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490314338Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u001c" value=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490292925Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490174581Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001" value=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490134295Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11	
level=info ts=2021-10-06T08:49:11.489476328Z caller=table_manager.go:324 msg="synching tables" expected_tables=15
2

You have retention configured and either file were missing when the file should have been removed via retention or the directory is not well configured.

3

You should not also run table manager and compactor.

4

I grep all absent names from log into file (docker version):
docker-compose logs --tail 1000000 loki | grep -P -o '/data/loki/chunks/[^:]+' > names.txt

And then create fake files with simple script:

File.open("names.txt", "r") do |f|
  f.each_line do |l|
    # Folder "data" is in current directory. If not, change path to appropriate.
    File.open("."+l.chomp, "w") {}
  end
end

Then Loki delete this files himself and error gone.

As I understand, Loki should skip error “no such file” if it want delete it.

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.