Grafana dashboard return the query unstable

  • What Grafana version and what operating system are you using?
    Image grafana version grafana:10.2.3-ubuntu
    Thanos version: thanos:v0.32.5

  • What happened?
    I encountered a strange issue, grafana dashboard run query display chart unstable, sometime it show no data (as first video), my datasource is thanos querier multi tenency (query multi thanos sidecar in different aws account).

  • How are you trying to achieve it?
    I tried click query in prometheus + thanos querier several times, test curl HTTP API 1000 times also, it’s work normally (as second video). So that mean no network issue (I install grafana and thanos querier same host docker-compose) and maybe the problem come from grafana.

  • What did you expect to happen?
    Return chart dashboard without no data

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were
    There are no logs relate to this issue

Example:
First click run queries


Second click run queries

It’s quite unstable, but there are totally normal when I several click query in thanos querier, no problem here

Update:
I was simulated install new grafana, it’s work normally query from similar datasource. Seem like production grafana is bottleneck somewhere in query dashboard, but my data grafana is still small to bottleneck :thinking:

1 Like

Use query inspector (browser console) to find result/response code for those queries. Check Grafana log, Grafana usage (cpu, mem, …) metrics - they can provide some clue.

what is simulated new grafana install and where is it installed at, what kind of server and specs? Also where is your production grafana hosted on?

Thanks for your response, I’ve check Grafana log, no logs relate issue.
I run Grafana by docker compose and monitor via cadvisor, resources usage (cpu, mem, …) still ~3%

here is request response in old grafana, size 1.6kB is return data and 436B is no data, seem like something bottleneck.

here is request response in new simulator grafana, the packet is larger than old one. Quite confuse here

Hello @yosiasz , I install new grafana by docker-compose, host in AWS EC2 AL2023 type t3a.medium. Production grafana host in EC2 Amazon linux 2 type r6a.large. In fact, production still has many surplus spec (cpu, memory), but it share resource with prometheus, alertmanager, mysql8 (use for grafana), thanos querier, sidecar and receiver.
I’m suspecting the slow query from mysql8 (cpu, memory of mysql8 still 3%), but seems it’s not true because the issue happen only one thanos sidecar, other remain thanos sidecar is normal.

Here is my docker-compose file:

  grafana:
    image: grafana/grafana:10.2.3-ubuntu
    container_name: graf-10.2.3
    restart: always
    ports:
      - 3000:3000
    volumes:
      - /mnt/grafana/data:/var/lib/grafana
      - /mnt/grafana/config:/etc/grafana
    user: "0:0"
    command:
      - apt update && apt install -y librust-gobject-sys-dev libnss3-dev libdbus-1-dev libatk1.0-dev libatk-bridge2.0-dev libcups2-dev libdrm-dev libxkbcommon-dev libxcomposite-dev libxdamage-dev libxrandr-dev libgbm-dev libpango1.0-dev libasound2-dev && chown -R root:root /etc/grafana && chmod -R a+r /etc/grafana && chown -R grafana:grafana /var/lib/grafana && chown -R grafana:grafana /usr/share/grafana
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "10"
    depends_on:
      - mysql8

I’m sorry to hear that you’re experiencing difficulties.

Grafana can provide valuable information about certain issues through its own debug logs. These logs can help troubleshoot and identify the root cause of problems (especially if they are related to anything that can be configured via Grafana config file).

To enable debug logging edit the configuration file grafana.ini:

[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
level = debug

Then restart grafana for the setting to go into effect.

Replicate the problem and check Grafana logs. However, before posting the debug logs on the community forum, it is important to sanitize any private details such as passwords, tokens, IPs, names, … By including properly formatted debug logs (with sanitized information) and details about your installations (e.g. exact Grafana version, how it was installed, OS details, …), you greatly enhance the chances of receiving accurate assistance and solutions.

Thank you so much, I’ve relsoved this issue.
I use datasource as load balancer target 2 different backend. That why cause unstable query. :joy: