My LOKI instance is using 1.5 million S3 objects and continues to grow despite many adjustments to retention. Retention is working, it’s not keeping data longer than expected, this is clearly not the problem. Something is driving a very high number of objects.
What does each individual object in a Loki S3 bucket represent and what can I do about that?
Most of the objects written by Loki are chunks. The frequency of Loki write is controlled by estimated size, idle time, or a maximum wait time. These configurations are all under limits_config (see Grafana Loki configuration parameters | Grafana Loki documentation).
Is there a reason you are concerned about object count in S3?
Our storage team enforces a 2 million object count limit. That is why it matters.
Which settings under limits_config dictate these settings? I don’t seem to see any that match your wording. They seem to be mostly settings to limit queries and limit incoming data, nothing about controlling how data is written.
update:
They appear to be under ingester section, not the limits section.
Personal opinion, it makes little sense to enforce such a small limit on object count for S3. I recognize the desire to not let things run rampant, but realistically the number should be something like 50 to 100 million. Might be worth a passing conversion with your storage team, but if you can’t convince them then you have a couple of options:
Change configuration to write chunks less often as mentioned above. max_idle_period is an easy change. Also, if you do change max_chunk_age you’ll want to make sure you also adjust the setting under querier to query your ingester for the same duration of max_chunk_age.
You also have the choice of using multiple S3 buckets as backend storage. However, I would recommend you to thoroughly test a migration method if you do choose to pursue this. I haven’t done this personally, but you may have to define a new storage and index scheme for a smooth migration.
Which querier setting needs to be adjusted if i change max_chunk_age?
About adding multiple buckets, you are proposing a new schema and unique storage config for that scheme that contains the multiple buckets so that the old data remains on the old single-bucket schema?
Also, from my reading into chunks today, it sounds like if you have dynamic labels in your data it creates many more chunks as well. As much as LogQL is handy for create those labels after they have been ingested, you can’t use grafana dashboard variables to filter data when the label was not there at ingestion time which makes dashboarding far less user friendly to anyone who doesn’t want to manually edit panel queries in order to filter data.
I suppose this could be done after the fact with recording rules, but I expect that will just re-create the problem.
Correct. Logs with unique set of labels are considered a log stream, and log stream would have their own chunk files. You are also correct in that you can’t use variables with non-label parsed values, which is a bit unfortunate.