Grafana Loki + Promtail could not access logs in real time

Hi community,

I have setup a multiples promtail services on server to push logs to a centralised loki services hosted using docker. The problem has been arised when the amount of logs per second is greater than 100000 lines. The grafana dashboard shows error when trying to query the data from the loki. The error shown from loki logs are:

The dashboard errors are shown in attached:


The current loki configuration is as shown:

http_listen_port: 3100
grpc_server_max_recv_msg_size: 16777216
grpc_server_max_send_msg_size: 16777216
path_prefix: /loki
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
store: inmemory

- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
prefix: index_
period: 24h

max_outstanding_requests_per_tenant: 8192
max_outstanding_per_tenant: 8192
log_queries_longer_than: 10s
compress_responses: true
parallelise_shardable_queries: true
align_queries_with_step: true
cache_results: true
split_queries_by_interval: 15m
max_query_length: 0h
max_query_parallelism: 32
ingestion_rate_strategy: local
ingestion_rate_mb: 32
ingestion_burst_size_mb: 64
max_streams_per_user: 0
max_entries_limit_per_query: 5000000
max_global_streams_per_user: 0
cardinality_limit: 200000

The server spec is relatively high for this use case:
It has 16 cores and 64gb in ram.

Is any configuration missed out or wrong that result in such error.


Your screenshots are two different errors.

First is saying that if you want to match regex you have to at least use .+, which need at least one character, whereas .* can match empty string, which cannot be used by itself.

Second error empty ring can be a bunch of different things. Have you confirmed your loki instance is actually working? Any logs from Loki?

Thanks for the prompt reply. The loki logs from monitoring across few hours: It do shows a repetitive lines of logs shown in the attachment below:

On top of that, from the zabbix graph from monitoring this server it do shows the memory consumption spike to the maximum and then the docker restart by itself which consists of the loki, grafana services hosted in docker.

Thats the things observed about the issue raised

Several ideas/hypothesis came across my mind while debugging this issues. The KV store is using the in-memory mode in the loki configuration. Does it contribute to the memory consumption issues? If so, what is the suggestion can be taken to solve this issue. As the grafana is expected to show a big amount of data (APPROXIMATELY 100000 lines per second) from loki which are the logs pushed from promtail.


  1. your error says you are out of space.

  2. If you are running a single instance you might consider increasing split_queries_by_interval to something bigger, maybe 12h.

The suggestion of split_queries_by_interval has taken but there are errors like

are shown.

When i trying to explore the datasource from grafana, it shows the empty ring error.

The memory consumption of the server spike quicker than previous configuration and the docker crashed & restarted.