I’m trying to migrate data from elastic to loki using http API. After sending 25 millions of logs to Loki (I didn’t receive any error when migrating), the grafana display labels that were available only at the historical data but doesn’t actually display anything.
I was previously testing the migration using less data, it took some hours, but the data was available. Now my migration was finished for more than 24h and no historical data is displayed.
This is a partial config of my loki:
compactor:
compaction_interval: 10m
retention_delete_delay: 2h
retention_delete_worker_count: 150
retention_enabled: true
shared_store: azure
working_directory: /var/loki/retention
limits_config:
allow_structured_metadata: true
enforce_metric_name: false
max_cache_freshness_per_query: 10m
max_entries_limit_per_query: 3000000
max_query_length: 700d
max_query_parallelism: 100
max_query_series: 3000000
reject_old_samples: false
reject_old_samples_max_age: 168h
retention_period: 320h
split_queries_by_interval: 24h
tsdb_max_query_parallelism: 512
By migration I guess you are exporting logs from ElasticSearch and then writing into Loki?
One thing that’s a bit tricky and not very obvious, is that when writing logs into Loki they have to be kinda in order for each chunk. There was a post that explained this well, but I can’t find it. Essentially you cannot write a stream of logs into Loki if a chunk with newer log already exists (a log stream is considered as logs that have identical set of labels).
For example, if you send three logs into Loki and they are out of order with a couple of seconds apart, this is accepted and will be written into one chunk. But if you write a log into Loki that’s a day before with labels {label="test"}, and newer logs already exist with the same label, then it’ll be discarded.
As a test, you can try to write to Loki with a brand new set of labels and see if that works or not.
Thanks for the reply @tonyswumac
By migration I guess you are exporting logs from ElasticSearch and then writing into Loki?
Yes, that’s correct.
is that when writing logs into Loki they have to be kinda in order for each chunk
I’m aware of that. In my migration process I send the elastic data to a database, and them ordered and paginate the database to send to loki.
But if you write a log into Loki that’s a day before with labels {label="test"} , and newer logs already exist with the same label, then it’ll be discarded
Even when using reject_old_samples: false if I have a 2024 log with {label="test"} and try to send the same label in 2021 loki will not allow?
It’s weird that I didn’t got any message acussing that the data would be discarded. There is some config workaround?
I don’t think there is any workaround for this. And if you think about it a bit it kinda makes sense, because Loki can’t go back and change a chunk that was already written into storage.
You can test this fairly easily, too, by either changing the labels to something entirely different, or write some logs into loki, wait for the chunk to be written, then write more logs into Loki with the same label but order timestamp.
1 Like
Thanks for explaining. I was able to test it, it really explains why I was able to make the migration on dev and not in prod.
While it makes sense conceptually, it’s odd from a product perspective that there’s no workaround. This makes data migration more difficult and consequently affects Loki’s adoption as well.
I think it’s just a side effect of the design. Personally I did run into this once before, but it wasn’t that big of a problem for me because migration is usually a one-time thing, and once you are aware it’s easy to work around this.
