How can I tell Loki to remove a reference to a file it no longer has access to?

This is a cross post of my question on the closed issue 429.

I had the grafana-agent reading from a file that it did have access to but now does not. I updated my job to read the log from a new location which is working as expected, but the agent is attempting to update metadata about the prior log and filling up my sys logs. How can I get loki to completely forget about the previous file?

Here is the log output:

Jul  6 15:03:33 localhost grafana-agent[11276]: ts=2023-07-06T15:03:33.342184934Z caller=positions.go:206 level=warn component=logs logs_config=integrations msg="could not determine if log file still exists while cleaning positions file" error="stat /root/.pm2/logs/server-out-0.log: permission denied"

See my post on issue 429 for more configuration details

You could try to edit the positions file and remove the line for the old file. Or, you could try to touch a file at the old location (assuming it doesn’t exist anymore) and let it clean itself up.

May be a bug or feature request as well, not sure if this was intentional or not.

That worked for me thanks.

systemctl stop grafana-agent.service
sudo su -l grafana-agent -s /bin/bash
vim /tmp/positions.yaml
# remove lines to unused log files:
#   /root/.pm2/logs/server-error-0.log: "10901"
#   /root/.pm2/logs/server-out-0.log: "8404847"
systemctl start grafana-agent.service
1 Like