Does Promtail offer any data delivery guarantees in the event Loki is unavailable? I see it has “retry then discard” approach, but I don’t like the sound of discarding data before it’s reached Loki.
There is a backoff setting that you can use to have logs be stored locally for certain period of time before it’s removed if destination is not available. For example, this is the backoff setting we use:
# Backoff schedule: # 0.5s, 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s(4.267m), 256s x 5 # For a total time of 1791.5s(29.85m) before logs are lost. backoff_config: min_period: 500ms max_period: 5m max_retries: 15
I don’t know the theoretical max limit for this configuration, however.