Static dashboard, timeseries doesn't work properly in Grafana -Loki

Hi,

I’ve configured several static panels to show the data in different time periods.

The dashbord is:

For the first panel, the data appear very soon, but for the last 6 hours it take time and for de last day or last 24 hour the dashboard show an error like this:


More in detail:

It takes more than 1.50 min. I’ve modified the data in the Grafana configuration to establish the limit for dataproxy timeout to 90. But it continues with the problem and it’s not normal to take so much time.

An extract loki configuration file is this:

I’ve made a lot of changes trying to solve this behavior, but I don’t know what the problem is. It seems to be a bottleneck but I don’t know where.

If I want to show the data for more than one day it’s impossible.
Any ideas about it?

What does your Loki cluster look like? How many readers / query frontends do you have, and how are the resource allocated?

Loki configuration would be helpful as well.

Hi, the thing is that there is only one Loki server connected to Grafana.

The number of persons that access Grafana to consult the data is not very high. (4-5 persons)
I identified the problem when I was alone connected to execute the query.
If there are more than two connected, the dashboards show an error of timeout

The thing is that in the promtail config file, there are 59 entries to filter the different web services I need to obtain the information
Let me explain:
I collect all the information from one server, so this server has a large promtail config file. Would it be a problem to collect the information? I see that in Loki the data is collected correctly.

An extract of promtail config file

The loki configuration file is: loki-local-config.yaml



(I’m sorry I don’t know how to attach yo the config in a better way)

Loki’s performance gain comes from distribution. If you are running a single instance there are a couple of things you probably want to do:

  1. Try to tune the chunk settings so each files are larger (so you have less files on your file system).

  2. Disable query split, or split it by a bigger time frame (for example split query by 12 hours instead of 30 minutes).

  3. Increase timeout as needed.

I participated in a couple of discussions with other users in similar situations, you might consider looking at my conversation history and see if anything there is helpful to you.

Hi, I’ve tried several options but when I try to Disable query split, none of the pannels show any information (it shows no data). So I have to configure it to show the data in the pannels.
I’ve increased the size: chunk_target_size: 3145728 up to 3MB but I still have the same problem.
Would you please, indicate where I can see other discussions so that I can continue testing with the posible solutions?
My server has this memory and CPUS:
image