Loki Log is flooded with "error deleting chunk" and "failed to delete key"

The logs of loki are flooded with “error deleting chunk” and “failed to delete key”. What can I do?

Here is a short excerpt:

2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490376481Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0007" value=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490366221Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490348506Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006" value=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490337836Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490314338Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u001c" value=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490292925Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11	
level=warn ts=2021-10-06T08:49:11.490174581Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001" value=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11	
level=error ts=2021-10-06T08:49:11.490134295Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11	
level=info ts=2021-10-06T08:49:11.489476328Z caller=table_manager.go:324 msg="synching tables" expected_tables=15

You have retention configured and either file were missing when the file should have been removed via retention or the directory is not well configured.

You should not also run table manager and compactor.

I grep all absent names from log into file (docker version):
docker-compose logs --tail 1000000 loki | grep -P -o '/data/loki/chunks/[^:]+' > names.txt

And then create fake files with simple script:

File.open("names.txt", "r") do |f|
  f.each_line do |l|
    # Folder "data" is in current directory. If not, change path to appropriate.
    File.open("."+l.chomp, "w") {}
  end
end

Then Loki delete this files himself and error gone.

As I understand, Loki should skip error “no such file” if it want delete it.