Grafana on docker-compose behind Nginx proxy slow performance

docker
postgres
#1

Hi Guys,
I’m setuping Grafana on Ubuntu machine with 8 cores CPU and 32GB RAM.
My architecture include:

  • Grafana running on Docker container
  • Nginx proxy on another container
  • PostgreSQL as datasource running on another container

Now Grafana is connected to Postgres DB, with read-only role.
My dashboard has about 15 different panels and 3 main variable filters that all queries are depends on.

My issue is that grafana is performing very very slow, both on load time and on each panel refresh.

I’ve inspected the queries it makes and it looks that some of the queries returns after few seconds, while other after few minutes. I have another container running Django webapi that is used to insert the data to DB, and while grafana is running the queries to DB, the whole Host machine became very slow, preventing from other clients of the Postgres to run queries in parallel.

I’ve tested the same dashboard on another PC where grafana is installed directly on the main ubuntu OS, without docker, running Postgres the same way without docker, and the performance is way better, without any lag.

I’ll only add that the variable filters on my dashboard are creating very long queries when for instance “ALL” is selected on “customer” filter. Where customer is a multi-choice drop-down of 2K values.

I know that long queries can lead to performance issues, but with the same datasource on another machine of the same power, grafana performs very smooth, considering that it’s installed directly on the host, and not using nginx reverse proxy or docker container.

#2

Start/configure your containers in the host network --network host and you will have the same network performance as non dockerised app. You will loose network isolation of your containers, so you need to consider also security of that action.

#3

I don’t think there is a big difference in performance between host and isolated network modes, according to this test by [Michael Nikitochkin], the host network performing better by only 20%. So I’m not sure this is something to consider, considering that it will expose all my containers or at least Postgres and Grafana to the entire world.

#4

I agree, but you have complained about container performance, so I gave you the most obvious reason. You are using Grafana + nginx = connection is going 2x through docker userland proxy, so in theory you can get 40% TCP performance improvement + nginx operates on L7 layer, which adds additional latency. I see only one reason for reverse proxy in front of Grafana - TLS customization.

You need to investigate whole end to end / machine performance to find bottleneck and optimization. Anyway, this is not a question very related to Grafana, but to used architecture.

#5

I agree that this forum is not related to docker, but it look like in general grafana on Docker is performing very slow, even when not necessary behind reverse proxy.
I’ve noticed that when dashboards is rendering 10-20 panels at once (some queries are very long) the whole machine server became unusable for a couple of minutes or until all queries are processed.
I still not sure what exactly is making such heavy load on the system, Postgres/Grafana or Nginx or all together.
I’ll just add that when I’ve reduced the amount of payload that is sent via POST request to /api/tsdb/query, by improving the queries, I got significant boost on performance.