Hoping to get a little more visibility here than on the slack channel. My loki clusters are operating fine but I’m trying to move EC2 based applications to Fargate and having trouble with the firelens/fluentbit forwarding to Loki.
On EC2 I’ve got a local promtail watching the journald logs and forwarding them ‘as-is’ to Loki. Works great. In fargate I’ve followed the documentation for setting up a fluentbit container sidecar with my apps, and setting the logging configuration to use firelens forwarded to fluentbit. As the Loki team has acknowledged that they won’t be maintaining their custom fluentbit version with their own plugin I have been using the official fluentbit image and configuring the loki settings through firelens.
If I leave the fluentbit default line_format as json, I get log events in Loki, but we find the json format a bit obnoxious and want to use the key/value instead. However on switching line_format = key_value in the fluentbit configuration I am finding that no log lines ever show up at loki. Fluentbit shows no errors, and none of the Loki services show any errors. I can’t tell which of parties involved here (AWS, Fluentbit, Loki) are potentially swallowing the log lines.
Anyone have working firelens configuration with key_value enabled?