In my case, I’ll be pulling logs off a Pulsar topic, which I’ll want to run though Alloy and persist to Loki. Pulsar has an acknowledgment but since I don’t see a way for alloy to return an error when I push to Alloy directly, I can’t know if Loki might be down for some reason. I could also write the logs to disk first and have Alloy process that way, but, if Loki is down I can’t just let the log file grow indefinitely, so I’d have to roll the logs which introduces issues as well. How do others handle this? I could skip using alloy and push straight to loki, which if it failed, I could send a failed ack so that pulsar keeps the logs, but then I lose Alloy processing. What is the recommended way to handle this situation so that I’m guaranteed not to lose any logs if Loki is down?
I was thinking maybe just always try a health check call to loki after receiving the log off of Pulsar before continuing? (Granted there could still be a slight possibility of loki going down in the short time between the ping being successful and the push to alloy, or the append to file for alloy processing, but that could probably be an acceptable loss.) How do other deal with this?