Grafana Loki + Promtail could not access logs in real time

Grafana Loki
1

Hi community,

I have setup a multiples promtail services on server to push logs to a centralised loki services hosted using docker. The problem has been arised when the amount of logs per second is greater than 100000 lines. The grafana dashboard shows error when trying to query the data from the loki. The error shown from loki logs are:

1
11888×351 43.3 KB

The dashboard errors are shown in attached:

image
image918×513 38.6 KB

image

The current loki configuration is as shown:

server:
http_listen_port: 3100
grpc_server_max_recv_msg_size: 16777216
grpc_server_max_send_msg_size: 16777216
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h

query_scheduler:
max_outstanding_requests_per_tenant: 8192
frontend:
max_outstanding_per_tenant: 8192
log_queries_longer_than: 10s
compress_responses: true
query_range:
parallelise_shardable_queries: true
align_queries_with_step: true
cache_results: true
limits_config:
split_queries_by_interval: 15m
max_query_length: 0h
max_query_parallelism: 32
ingestion_rate_strategy: local
ingestion_rate_mb: 32
ingestion_burst_size_mb: 64
max_streams_per_user: 0
max_entries_limit_per_query: 5000000
max_global_streams_per_user: 0
cardinality_limit: 200000

The server spec is relatively high for this use case:
It has 16 cores and 64gb in ram.

Is any configuration missed out or wrong that result in such error.

Thanks

2

Your screenshots are two different errors.

First is saying that if you want to match regex you have to at least use .+, which need at least one character, whereas .* can match empty string, which cannot be used by itself.

Second error empty ring can be a bunch of different things. Have you confirmed your loki instance is actually working? Any logs from Loki?

3

Thanks for the prompt reply. The loki logs from monitoring across few hours: It do shows a repetitive lines of logs shown in the attachment below:

image
image1699×94 8.75 KB

On top of that, from the zabbix graph from monitoring this server it do shows the memory consumption spike to the maximum and then the docker restart by itself which consists of the loki, grafana services hosted in docker.
image

Thats the things observed about the issue raised

4

Several ideas/hypothesis came across my mind while debugging this issues. The KV store is using the in-memory mode in the loki configuration. Does it contribute to the memory consumption issues? If so, what is the suggestion can be taken to solve this issue. As the grafana is expected to show a big amount of data (APPROXIMATELY 100000 lines per second) from loki which are the logs pushed from promtail.

Thanks,
Joe

5

  1. your error says you are out of space.

  2. If you are running a single instance you might consider increasing split_queries_by_interval to something bigger, maybe 12h.
    `

6

The suggestion of split_queries_by_interval has taken but there are errors like

image
image1095×288 12.3 KB

are shown.

When i trying to explore the datasource from grafana, it shows the empty ring error.
image

The memory consumption of the server spike quicker than previous configuration and the docker crashed & restarted.

image
image1097×37 2.64 KB