Static dashboard, timeseries doesn't work properly in Grafana -Loki

Grafana Loki
, ,
1

Hi,

I’ve configured several static panels to show the data in different time periods.

The dashbord is:

image
image1911×720 256 KB

For the first panel, the data appear very soon, but for the last 6 hours it take time and for de last day or last 24 hour the dashboard show an error like this:

image
image1775×320 13.1 KB

More in detail:
image
image947×247 51.5 KB

It takes more than 1.50 min. I’ve modified the data in the Grafana configuration to establish the limit for dataproxy timeout to 90. But it continues with the problem and it’s not normal to take so much time.

An extract loki configuration file is this:

image
image947×891 42.4 KB

I’ve made a lot of changes trying to solve this behavior, but I don’t know what the problem is. It seems to be a bottleneck but I don’t know where.

If I want to show the data for more than one day it’s impossible.
Any ideas about it?

2

What does your Loki cluster look like? How many readers / query frontends do you have, and how are the resource allocated?

Loki configuration would be helpful as well.

3

Hi, the thing is that there is only one Loki server connected to Grafana.

The number of persons that access Grafana to consult the data is not very high. (4-5 persons)
I identified the problem when I was alone connected to execute the query.
If there are more than two connected, the dashboards show an error of timeout

The thing is that in the promtail config file, there are 59 entries to filter the different web services I need to obtain the information
Let me explain:
I collect all the information from one server, so this server has a large promtail config file. Would it be a problem to collect the information? I see that in Loki the data is collected correctly.

An extract of promtail config file

image
image451×874 15.4 KB

The loki configuration file is: loki-local-config.yaml

image
image897×828 35.6 KB

image
image971×310 11.2 KB

(I’m sorry I don’t know how to attach yo the config in a better way)

4

Loki’s performance gain comes from distribution. If you are running a single instance there are a couple of things you probably want to do:

  1. Try to tune the chunk settings so each files are larger (so you have less files on your file system).

  2. Disable query split, or split it by a bigger time frame (for example split query by 12 hours instead of 30 minutes).

  3. Increase timeout as needed.

I participated in a couple of discussions with other users in similar situations, you might consider looking at my conversation history and see if anything there is helpful to you.

5

Hi, I’ve tried several options but when I try to Disable query split, none of the pannels show any information (it shows no data). So I have to configure it to show the data in the pannels.
I’ve increased the size: chunk_target_size: 3145728 up to 3MB but I still have the same problem.
Would you please, indicate where I can see other discussions so that I can continue testing with the posible solutions?
My server has this memory and CPUS:
image

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.