Grafana-loki plugin for fluent-bit (core dumped)

If we run the plugin with the following command and configuration, we get this error.

fatal: morestack on g0
Trace/breakpoint trap (core dumped)

[root@chrzapp2369 fluent-bit]# /opt/fluent-bit/bin/fluent-bit -e /opt/fluent-bit/out_grafana_loki.so -c fluent-bit-test.conf
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/20 16:51:19] [ info] [fluent bit] version=2.2.0, commit=, pid=20314
[2023/12/20 16:51:19] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/20 16:51:19] [ info] [cmetrics] version=0.6.4
[2023/12/20 16:51:19] [ info] [ctraces ] version=0.3.1
[2023/12/20 16:51:19] [ info] [input:tail:tail.0] initializing
[2023/12/20 16:51:19] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
level=info caller=out_grafana_loki.go:64 id=0 [flb-go]="Starting fluent-bit-go-loki" version="(version=main-003a51b, branch=main, revision=003a51b86)"
level=info caller=out_grafana_loki.go:66 id=0 [flb-go]="provided parameter" URL=https://myLokiServer:3100/loki/api/v1/push
level=info caller=out_grafana_loki.go:67 id=0 [flb-go]="provided parameter" TenantID=
level=info caller=out_grafana_loki.go:68 id=0 [flb-go]="provided parameter" BatchWait=1.000s
level=info caller=out_grafana_loki.go:69 id=0 [flb-go]="provided parameter" BatchSize=1048576
level=info caller=out_grafana_loki.go:70 id=0 [flb-go]="provided parameter" Timeout=10.000s
level=info caller=out_grafana_loki.go:71 id=0 [flb-go]="provided parameter" MinBackoff=0.500s
level=info caller=out_grafana_loki.go:72 id=0 [flb-go]="provided parameter" MaxBackoff=300.000s
level=info caller=out_grafana_loki.go:73 id=0 [flb-go]="provided parameter" MaxRetries=10
level=info caller=out_grafana_loki.go:74 id=0 [flb-go]="provided parameter" Labels="[job=fluentbit]"
level=info caller=out_grafana_loki.go:75 id=0 [flb-go]="provided parameter" LogLevel=debug
level=info caller=out_grafana_loki.go:76 id=0 [flb-go]="provided parameter" AutoKubernetesLabels=false
level=info caller=out_grafana_loki.go:77 id=0 [flb-go]="provided parameter" RemoveKeys=[]
level=info caller=out_grafana_loki.go:78 id=0 [flb-go]="provided parameter" LabelKeys=[]
level=info caller=out_grafana_loki.go:79 id=0 [flb-go]="provided parameter" LineFormat=0
level=info caller=out_grafana_loki.go:80 id=0 [flb-go]="provided parameter" DropSingleKey=true
level=info caller=out_grafana_loki.go:81 id=0 [flb-go]="provided parameter" LabelMapPath=map[]
level=info caller=out_grafana_loki.go:82 id=0 [flb-go]="provided parameter" Buffer=true
level=info caller=out_grafana_loki.go:83 id=0 [flb-go]="provided parameter" BufferType=dque
level=info caller=out_grafana_loki.go:84 id=0 [flb-go]="provided parameter" DqueDir=/tmp/flb-storage/loki
level=info caller=out_grafana_loki.go:85 id=0 [flb-go]="provided parameter" DqueSegmentSize=500
level=info caller=out_grafana_loki.go:86 id=0 [flb-go]="provided parameter" DqueSync=false
level=info caller=out_grafana_loki.go:87 id=0 [flb-go]="provided parameter" ca_file=
level=info caller=out_grafana_loki.go:88 id=0 [flb-go]="provided parameter" cert_file=
level=info caller=out_grafana_loki.go:89 id=0 [flb-go]="provided parameter" key_file=
level=info caller=out_grafana_loki.go:90 id=0 [flb-go]="provided parameter" insecure_skip_verify=true
[2023/12/20 16:51:19] [ info] [sp] stream processor started
[2023/12/20 16:51:19] [ info] [input:tail:tail.0] inotify_fs_add(): inode=77 watch_fd=1 name=/var/log/messages
fatal: morestack on g0
Trace/breakpoint trap (core dumped)

This ist our fluent-bit.conf:

[SERVICE]
    flush        1
    log_level    info
[INPUT]
    name        tail
    path        /var/log/messages
    tag         log_generator
[OUTPUT]
    Name        grafana-loki
    Match       *
    URL         https://lmyLokiServer:3100/loki/api/v1/push
    labels      {job="fluentbit"}
    LineFormat  json
    insecure_skip_verify true
    LogLevel debug
    Buffer true

any idea?

I haven’t used fluentbit with Loki in a while, but this seems to be something related to fluentbit. I’d try enabling debug log and see what it says. Also, is your Loki server running on HTTPS?

We have found the problem.
The plugin must be compiled with the following version. The error occurs when you compile the plugin with the latest Loki version.

Loki → 2.8.7
Golang → 20.12

Thank you for taking the time to self answer. For people using the docker images grafana/fluent-bit-plugin-loki:main-e2ed1c0 is stable.

2 Likes