Graph disappear

tinman · October 19, 2022, 5:10am

Hi all.
noob at Grafana, so excuse the question.

I have a API endpoint located in AWS (Python/Flask).
The API is decorated with prometheus instrumentation.
when I call the API endpoint from my local machine, I can see the metrics recorded in Prometheus TSDB, so I got data flowing…

now in the Prometheus console via PromQL and in Grafana… for as long as I generate/call the end point (generate load). When my load generator stops… the dashboard/graph information in Grafana disappears,

it does not show me activity if I say show all, even if I increase the time window.

Although I can go into Prometheus and via PromQL query my activity and it’s there.Guessing it’s probably a error on my side in defining the Grafana dashboard… all I can think of

Deployment of Grafana is as part of Kube-Prometheus deployment as per: kube-prometheus/manifests at v0.6.0 · prometheus-operator/kube-prometheus · GitHub

thank you

tinman · October 20, 2022, 9:07am

anyone able to help?

G

tinman · October 22, 2022, 10:21am

anyone, please help, why would my graph in Grafana be dead… but showing / doing the same query in Prometheus shows values…
Same datasource.

tinman · October 24, 2022, 11:57am

… is this posting in the wrong section or something,
trying to understand why my dashboard is behaving as it does.
G

usman.ahmad · November 15, 2022, 4:26pm

Hi @tinman,

Thanks for opening this post.

Well, K8s and AWS both are a bit out of my skill set but I will try to get the thread moving so that someone else may help you out.

As I understood that the Graph disappear.

What panel visualisation you are using?

Also, did you try to increase the timeline e.g. from 30 minutes to let’s say 1 hour on the top bar of the panel might have helped?

Lastly, what do you see in the grafana backend logs? if there is nothing in the logs then increase the log level to debug mode and try to reproduce the issue and post the log output here

It might be more helpful to post some sort of screencast to understand it better.

tinman · November 15, 2022, 6:49pm

Ok, so I have 2 sources, called “issuer” and “verifier”

Left 2
rate(iamza_trustreg_txncounter_verifier_domain_count_total[5m])
rate(iamza_trustreg_txncounter_verifier_count_total[5m])
Right 2
rate(iamza_trustreg_txncounter_issuer_domain_count_total[5m])
rate(iamza_trustreg_txncounter_issuer_count_total[5m])

so while the process creating the fake workload runs I got the line/graph as per the 2 dashboards (top right, bottom left), and then on the next 5 sec refresh the dashboard would go black/empty as per the other 2 dashboards (top left/bottom right)

Grafana logs ?

if you might be willing to do a teams session, might be easier to show.

G

usman.ahmad · November 16, 2022, 10:53am

Thanks for the details,

Do you see any interesting messages in the logs once it gets refreshed after 5 seconds?

tinman · November 16, 2022, 11:45am

more a case of finding the grafana logs in the case of it being a container on a EKS cluster…
if anyone knows ?

G

tinman · November 17, 2022, 3:48pm

went digging for the logs, remembered how to get to them, with this being a EKS cluster deploy…
I’m running my workload generator at the moment.
Not sure if there are more logs… this does not seem like allot of log information for the 6 dashboards on the screen, 4 of which should be painting nice graphs atm.
G

 time_ms=2 size=109 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?panelId=16&edit&fullscreen&orgId=1&refresh=5s"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=3 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=7 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=3 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=9 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=0 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:44:41+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:44:41+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=2 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:48:24+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:48:24+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=0 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:48:24+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:48:24+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=3 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:48:24+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:48:24+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=0 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"
t=2022-11-17T15:48:24+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/v1/query_range remote_addr=127.0.0.1 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now" error="http: proxy error: context canceled"
t=2022-11-17T15:48:24+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/v1/query_range status=502 remote_addr=127.0.0.1 time_ms=0 size=0 referer="http://localhost:3001/d/csI2HVS4k/trus-cntr?orgId=1&refresh=10s&from=now-1h&to=now"

tinman · November 17, 2022, 3:52pm

Ignore the first 2 dashboards, first row, they should be at 0 atm. I havent created any load there the last hour.
bottom 2 rows should be active.
G

usman.ahmad · November 18, 2022, 10:23am

Thanks for the detailed reply.

Looking at the log error message i.e.

error="http: proxy error: context canceled"

After some research found out that this is a known issue in Prometheus and it is due to the use of multiple queries towards a Prometheus datasource.

The solution proposed is to change the refId to a unique value in the JSON, according to:

github.com/grafana/grafana

Prometheus - http: proxy error: context canceled

opened 11:19AM - 01 Oct 20 UTC

closed 03:05PM - 18 Nov 20 UTC

dupondje

area/datasource/proxy type/works-as-intended area/panel/data

**What happened**: After upgrading grafana from 7.1.5 to 7.2.0, some dashboar…ds don't load completely anymore. But the strange thing is that it seems to stop sending out ajax calls to the backend (Prometheus) to get its data. **What you expected to happen**: Dashboards load fine when downgrading to 7.1.5. **How to reproduce it (as minimally and precisely as possible)**: Open some random dashboard with a lot of Prometheus queries. **Anything else we need to know?**: The logs show the following: grafana-server[39288]: t=2020-10-01T13:02:59+0200 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/api/v1/query_range remote_addr=127.0.0.1 referer="xxx/system-metrics-linux?orgId=1&var-disk_device=All&var-interface=All&from=now-2d&to=now" error="http: proxy error: context canceled" **Environment**: - Grafana version: 7.2.0 - Data source type & version: Prometheus/Thanos - OS Grafana is installed on: Debian 10 - User OS & Browser: Firefox Arch Linux - Grafana plugins: / - Others:

Therefore please have a look there and maybe this fix can help you out (also check the other user comments as there is always something more explained to resolve it).

In case if it does help you then, we would appreciate it if you can open a bug report about this in our official GitHub repository using this link.

tinman · November 18, 2022, 5:20pm

hmmmm, where do I change this json, this refid.
it’s strange, this is 4 dashboards, or would each dashboard with its various line each be a separate query.

G

usman.ahmad · November 18, 2022, 6:53pm

Check this comment.

So basically, you also need to find the refId on all of your panels (you have currently 4 of them) and see which ones are getting duplicated.

For the ones which have duplicated refId, replace them with any other number e.g. from 222 to 223 and then update the Panel JSON.

How to edit JSON,

Click on the Panel title drop-down list → select Inspect → Panel JSON and there you go!!.

tinman · November 19, 2022, 4:39am

Hi hi
So did the RefID changes, had no impact, ps version is 6.7.4
I deployed a new copy of Grafana on my local machine, pointing to the same Prometheus data store.
Created the same 4 dashboards and it’s working as I’d expect it to, version 9.2.4
Note while recreating the dashboards I inspected the JSON and they all / always seem to be getting created with refID:A
might make sense to have this number auto increment or make it a random gen number if there is a issue with it.

tinman · November 19, 2022, 5:45am

a little update.
I upgraded my grafana on the cluster to 9.2.4
it’s working… better, seems to be more stable, still have graphs disappearing though, and then returning with a next refresh, as it stands it won’t be acceptable for a production monitoring environment that manages/monitors financial transactions/flows…
G

tinman · November 19, 2022, 6:00am

observation…
Just wanted to try a different visualisation, bar graphs based on rate of the various inputs…
well when i select data source and then metric my metric is not there…
although i can refresh my graph in prometheus dashboard/graph itself.
wonder if the problem is somewhere/somehow getting to the data itself.
G

usman.ahmad · November 23, 2022, 10:41am

Hi @tinman,

Well, I cannot tell the reason as sounds to be that it’s a bug or else in your case.

We would appreciate it if you can open a bug report about this in our official GitHub repository using this link.

Provide all the information (you can simply copy/paste your data + screenshot from here to there).

Lastly, paste the link to your GitHub issue here so that all other community users can also track it.

tinman · November 23, 2022, 11:47am

will see bout opening a bug report.
hmmm might be missing something myself, which GitHub issue ?
G

yosiasz · November 23, 2022, 4:03pm

What browser are you having this issue in? Can you try to use other browsers and see it issue persists?

tinman · November 23, 2022, 4:27pm

this was/is via Safari. will have a look using Firefox and advise.
G

Topic		Replies	Views
Dashboard data disappear Prometheus	0	676	October 26, 2022
How do we ask for help Grafana dashboard	4	557	November 16, 2022
All our dashboards disappeared Dashboards	2	561	November 17, 2022
Grafana Dashboard displaying empty graphs, not data coming Configuration	8	23507	February 2, 2024
Newbie question for Bind Dashboard & Prometheus Grafana	0	381	November 20, 2018

Graph disappear

Related topics