What Grafana version and what operating system are you using?
OS: on azure Kubernetes
What are you trying to achieve?
Load testing our Grafana Deployment.
How are you trying to achieve it?
3 server instances of Grafana are deployed, Grafana Database is postgres (also deployed on azure Kubernetes), and 3 APIs functioning as the DataSource (also in azure Kubernetes).
Using JMeter to simulate 100 users accessing a dashboard each.
Endpoint used for JMeter requests: https:// “grafana link” /api/ds/query
Observing the results from the load test, we noticed a consistent spike in the requests time every 10 minutes.
This spike gets worse with more users simulated in the test.
In the above JMeter graph, is the result of a test with 100 users.
The test was configured so that at the 10 minute mark, all users would be making the 4 requests (1 per panel, 4 Panels in the Dashboard).
The first consistent spike occurred at the ~4 minute mark, being the smallest with the least users active at the time.
What did you expect to happen?
Response times being somewhat consistent.
Response Times not having a consistent spike every x(10) minutes.
Can you copy/paste the configuration(s) that you are having problems with?
We are using the following configuration via yaml:
- name: GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH value: username - name: GF_SERVER_SERVE_FROM_SUB_PATH value: "true" - name: GF_SERVER_DOMAIN value: <our domain> - name: GF_SERVER_ROOT_URL value: https://%(domain)s/grafana/ - name: GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS value: <our plugin> - name: GF_INSTALL_PLUGINS value: >- <our plugin>,https://grafana.com/api/plugins/grafana-piechart-panel/versions/latest/download;grafana-piechart-panel - name: GF_AUTH_GENERIC_OAUTH_AUTH_URL value: <generic auth link> - name: GF_AUTH_GENERIC_OAUTH_TOKEN_URL value: <generic auth link> - name: GF_AUTH_GENERIC_OAUTH_API_URL value: <generic auth link> - name: GF_AUTH_GENERIC_OAUTH_AUTHORITY_URL value: <generic auth link> - name: GF_AUTH_SIGNOUT_REDIRECT_URL value: <our redirect link> - name: GF_AUTH_GENERIC_OAUTH_SCOPES value: <our scopes> - name: GF_AUTH_OAUTH_STATE_COOKIE_MAX_AGE value: '300' - name: GF_AUTH_TOKEN_ROTATION_INTERVAL_MINUTES value: '100' - name: GF_SECURITY_COOKIE_SECURE value: "true" - name: GF_SECURITY_COOKIE_SAMESITE value: "strict" - name: GF_DATABASE_TYPE value: postgres - name: GF_DATABASE_HOST value: <DB link> - name: GF_DATABASE_USER valueFrom: secretKeyRef: name: <secret> key: username - name: GF_DATABASE_PASSWORD valueFrom: secretKeyRef: name: <secret> key: password - name: GF_DATABASE_SSL_MODE value: require - name: GF_DATABASE_MAX_IDLE_CONN value: '30' - name: GF_DATABASE_MAX_OPEN_CONN value: '30'
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly
Did you follow any online instructions? If so, what is the URL?
Did not follow any online instructions.