Compensation of Fluentbit Loki Output Plugin

[smandok]

Hi.

We’re trying to setup Fluent-bit shipping logs to Loki for visualization in Grafana.

As first test we set up a perfectly working Loki-instance with Fluentbit using the Fluent Bit Loki chart which is being used within the Loki-stack Helm-chart: The K8s-labels can be chosen in the Explore-function in Grafana and we can can simply see the unpoluted log-field-value of the logs.

But now we want to setup a Loki-distributed-architecture with the main goal of having an ha-concept. For that the corresponding Helm chart does not include the option to use Fluent-bit (like on the Loki-stack-chart). So now we’re trying to figure out how exactly the Fluentbit-chart has to be set up to get also the functionality like it is using the Loki-stack-chart.

Although there are some small differences in general between the configs of the Loki-Fluent-bit.conf and the Fluent-bit.conf, the main points are still the same:

Loki-fluent-bit.conf:

[SERVICE]
    HTTP_Server    On
    HTTP_Listen    0.0.0.0
    HTTP_PORT      2020
    Flush          1
    Daemon         Off
    Log_Level      warn
    Parsers_File   parsers.conf
[INPUT]
    Name           tail
    Tag            kube.*
    Path           /var/log/containers/*.log
    Parser         docker
    DB             /run/fluent-bit/flb_kube.db
    Mem_Buf_Limit  5MB
[FILTER]
    Name           kubernetes
    Match          kube.*
    Kube_URL       https://kubernetes.default.svc:443
    Merge_Log On
    K8S-Logging.Exclude Off
    K8S-Logging.Parser Off
[Output]
    Name grafana-loki
    Match *
    Url http://loki:3100/api/prom/push
    TenantID ""
    BatchWait 1
    BatchSize 1048576
    Labels {job="fluent-bit", cluster="hz-nb-dev-rt7-01"}
    RemoveKeys kubernetes,stream
    AutoKubernetesLabels false
    LabelMapPath /fluent-bit/etc/labelmap.json
    LineFormat json
    LogLevel warn

with the labelmap.json:

{
  "kubernetes": {
    "container_name": "container",
    "host": "node",
    "labels": {
      "app": "app",
      "release": "release"
    },
    "namespace_name": "namespace",
    "pod_name": "instance"
  },
  "stream": "stream"
}

Fluent-bit.conf:

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File parsers.conf
    Parsers_File custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On
[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Tag kube.*
    Mem_Buf_Limit 5MB
    Parser docker
[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    K8S-Logging.Parser Off
    K8S-Logging.Exclude Off
[OUTPUT]
    Name loki
    Host loki-distributed-distributor
    Port 3100
    Retry_Limit false
    Match *
    Labels cluster=xyz, job=fluent-bit
    Label_keys $kubernetes['host'], $kubernetes['pod_name'], $kubernetes['container_name']
    Remove_keys kubernetes,stream
    Line_format json
    Auto_kubernetes_labels off

So, do somebody has experience on that? How you can achieve the same easy and simple functionality of the (deprecated) Fluentbit Loki Output Plugin by using the origin Fluent/Fluent-bit-chart, especially for getting easily the log-field-value just by chosing the labels within Grafana?