Using prometheus with Grafana

Hi,

Thanks for this great repo and idea! :slight_smile:

im trying to monitor K8S with Grafana and Prometheus Operator
most of the metrics are working as expected and I was able to see the dashboard with the right value, our system contain 10 nodes with overall 500 pods, now when I run query in Grafana to get all the graphs which contain all the nodes and all the pods on those nodess the Grafaa Ui doesn’t show anything even if I increase the interval to 15 min . I see just the continue indicator working but no response . (the data is from previous screen )

I remove all the limits and requests from Prometheus and Grafana
(when I monitor Prometheus and Grafana I didnt see high load on those components when i execute the query i.e. entering to the screen)

we are using Grafana 5.4.2 and Prom operator 0.29,
I try to search the net and I find the following
http://docs.grafana.org/installation/performance/
I increase the graphite to the latest version 1.1.5 without success,
Any idea, any hint what it could be ?

I remove all the limits so now sure what else can I do my system have lot of resources so this is not a problem, the problem is just on this query which I suppose its heavy but I dont know why it hanging

2019-03-11_18-40-15
btw when I see the network I see very large url in the query and some of them are with status cancelled .