Loki logs are missing when using with cron job pod

I am overall using Loki pretty successful in my nodejs app with Winston.
However, only when I am using it in cron job pods which has short container life, I get only the first message in Grafana and the rest is missing.

This is my winston loki transport:

new LokiTransport({
            host: LOKI_URL,
            basicAuth: LOKI_BASICAUTH,
            json: true,
            labels: {
                job: "testjob",
                ns: MY_POD_NS,
                deployment: MY_DEPLOY_NAME,
                pod: hostname,
            },
            batching: false,
            level: LOG_LEVEL,
        })

I put console.log() right next to loki logs and I can see logs on console.
Any ideas where can I dig to solve this problem?

Hey @acidos

I’m just guessing here because I haven’t used Winston, but are you sure Winston is writing all the logs before the pod exits? Have you tried the gracefulShutdown option as specified here?

Sounds like you should be asking the Winston folks about this behaviour, because Winston is not (to my knowledge) a supported Loki client.

Hey @dannykopping
Yes, I have tried to use gracefulShutdown but no help. The last message before process exit is lost most of the time and looks like Winston is not having a chance to flush the buffer.

I will also try to ask Winston guys.
Thanks

1 Like

Best of luck!
Could I ask you to please report back here if you find a solution? It’d be great for future folks who run into this same issue.

1 Like