Diagnosis of Sluggish Dashboards

  • Grafana/Loki version and what operating system / components I am using:
    FreeBSD 13.1-RELEASE-p4
    Google Cloud SDK 331.0.0
    grafana-loki-2.6.1_4
    grafana9-9.2.4
    nginx-full-1.22.0_13,3

  • What I am trying to achieve?
    Dashboarding of a webserver’s access logs using a Google Cloud bucket to hold data for ‘Loki’.

  • How I am trying to achieve it:
    Our FreeBSD dashboard server (cf above) runs ‘Grafana’ and ‘Loki’ behind an nginx frontend, ‘Loki’ is configured to place dashboardable data (obtained from access logs on a 2nd, separate, dedicated FreeBSD webserver) in a Google Cloud bucket with indexing done on the dashboard server.

  • What has happened is:
    Dashboards indeed appear when the dashboard server’s Grafana service is browsed. However, performance is extremely sluggish, taking a minute or longer to render the dashboards. Login to the Grafana server is also very sluggish.

  • What I expect to happen:
    Much crisper response. What we have currently is not presentable to business teams.

  • I am not yet certain which configurations need to be posted with this topic.

  • The only errors in the Grafana UI or in related logs are occasional HTTP timeouts reported by the nginx that frontends Grafana.

  • The online instructions we’ve followed are the grafana.com documentation and other documentation source specific to setup of Loki for use with Google Cloud buckets.

Newbie Grafana admin here.

What troubleshooting we have done so far is elementary checking of the FreeBSD dashboard server itself (which runs the components listed under the first bullet item above). Examining ‘top’, ‘df’, ‘iostat’ reports and FreeBSD logs, we see no evidence of depletion of CPU/memory/disk resources. We’ve enabled Grafanadebug level logging and observe no obvious issues. We have noticed a good deal of disk write activity on the dashboard server when a delay is observed, which we conjecture is ‘Loki’ indexing activity (our understanding is even we’ve configured the data used for dashboarding to be located on a Google Cloud bucket, the index is created by Loki directly on the host running Loki (in our case, the FreeBSD dashboarding server).

Has any other forum members had experience with using a Google Cloud bucket to house data used for dashboarding, or had occasion to solve a Grafana dashboard presentation ‘sluggishness’ problem? Suggestions on how to troubleshoot?