Grafana + K6 + Influx Data Error

Grafana k6
, ,
1

I currently have a k6 script that runs around 15 different scenarios and outputs that data to an instance of grafana deployed on kubernetes where I then have a deployed instance of grafana pull the data from influx into a dashboard for it to be displayed. What I have noticed is when I run my test to many times back to back the grafana dashboard starts to crash with the following error showing on all the panels and no data displaying.

Status: 500. Message: invalid: compilation failed: error @4:6-6:62: expected RPAREN, got EOF error @5:3-5:5: invalid expression: invalid token for primary expression: PIPE_FORWARD

Has anyone seen this error before? It is misleading because to me its saying the query is wrong but the query works as long as the test isnt ran multiple times back to back. I monitored my influx resources and it isnt coming close to maxing out on cpu and memory. Typically the issue fixes itself if I restart influx and wait a little bit. Is there a way to prevent this from happening as a whole or has anyone else found a workaround or something or this?

version=v2.7.10 commit=f302d9730c build_date=2024-08-16T20:19:28Z

2

Also, the _monitoring bucket in influx is showing write errors that occured during the k6 test yesterday.

I have confirmed that the pvc attached to influx also is not low on storage
/dev/rbd7 15G 1.2G 14G 8% /var/lib/influxdb2

3

Can you share the k6 script?

4

Not really, its over 1600 lines, the jist of it is its just adding load to multiple different endpoints and tracking response times for all the endpoints. Theres around 500-800 vus running at any given time