Loki is installed via loki-stack helm chart (v2.4.1) on k8s. Component versions are not very fresh but loki-stack has these versions of loki software. Stack deployed with loki, grafana and promtail (one per node). Two days ago loki stops receive log messages. After restart the loki pod it becomes working again but logs for these two days now are missed in loki. Logs now are only in log-files.
Is any way to re-load/re-push logs from specified day/time to loki ?
Hmm… this may be the noob solution, but if you still have the logs, you could send them to promtail again. The exact way would depend a little on your setup, but one possible solution would be writing the logs again into a file promtail is observing (or creating a new file for dumping the old logs).
I would reccomend to test this with a single log line, in order to make sure loki saves them with the right timestamps. I’ve set up an rsyslog with active spooling, so that the logs gets stored and resend if there is some trouble with the pod. But, at least for my setup, the ingestion timestamp get written into loki in stead of the timestamp within the log.
It is too complex task … k8s logs from ~30 Namespaces from multiple pods in each NS … b.t.w. I have no direct access to logs because k8s hosted by DigitalOcean :-). I hoped that promtail has some feature/signal that means “please read all You logs again and re-push it to loki”. But docs says that there is no such feature.
Thank for Your reply but most probable I should forget about lost logs.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.