We have installed Loki-Grafana-Fluentbit without using Helm. We have 350+ application running on Kubernetes cluster.
Some of application produce too many lines of logs in a seconds. During that time we are facing the issue delay in logs from loki to grafana. This is happen in some of application. It is starting delay from 3 min and than so on… i assume that logs are not getting in batch-size from loki to grafana.
We are using Fluent-bit official image with output Loki with this ref documentation >> Loki - Fluent Bit: Official Manual
I am sharing ConfigMap file of Loki and Fluent-bit, I am not sure where i am missing.
- Fluent-bit Configmap file.
data:
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Buffer_Size 0
Merge_Log Off
#Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level debug
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE out_loki.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 128MB
Skip_Long_Lines On
Refresh_Interval 20
# Control the log line length
Buffer_Chunk_Size 4MB
Buffer_Max_Size 8MB
# Using the docker mode to deal with multiline messages emitted by docker
Docker_Mode On
Rotate_Wait 30
out_loki.conf: |
[Output]
name loki
match *
host loki.logging.svc.cluster.local
port 3100
tenant_id ""
labels job=fluentbit
auto_kubernetes_labels On
line_format json
parsers.conf: |
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
- Loki Configmap file:
auth_enabled: false
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
chunk_encoding: snappy
chunk_target_size: 1536000 #0
max_transfer_retries: 0
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
limits_config:
ingestion_rate_mb: 15
ingestion_burst_size_mb: 30
max_chunks_per_query: 2000000
max_label_name_length: 1024
max_label_value_length: 2048
max_streams_per_user: 10000
max_entries_limit_per_query: 5000
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
server:
http_listen_port: 3100
log_level: debug
grpc_server_max_recv_msg_size: 8388608
grpc_server_max_send_msg_size: 8388608
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 72h
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: filesystem