Grafana 9.4.7. Cloudwatch plug-in
I have a dashboard with 13 separate panels each with queries that use the AWS insight syntax to look for keywords in the Cloudwatch logs.
The syntax for all the queries is something like:
filter @message like /ERROR/
| stats count(*) as ERROR by bin(1h)
… OR …
filter @message like /ERROR/
| stats count(*) as ERROR
… depending on whether its a time series or a guage panel
Very often when the dashboard is refreshed one or more of the panels will error with
“t.data is not iterable”
However the error is often cleared if I refresh the dashboard again. The affected panel seems to be random.
… after refresh …
Your error is there:
I guess you are hitting some CW limits. Blind guess: Query concurrency.
Error moves around. Next image : - Same dashboard, same error, different panel. CW query concurrency is 30 insight queries but I have < 20. Sometimes it refreshes with no errors. No idea why.
Because each time another panel hits the limits.
In theory all panels (with CW logs query) may have this problem in the worst case (for example refreshing all panels every second by multiple users in multiple browsers). It is not a problem with the query but (very likely) how many CW queries your Grafana issues to AWS CW API per time period - it is not unlimited - check linked AWS doc please.
Check the error properly - it will gave you a clue - you showed only some small part Too many Re..., so don’t expect correct answer from minimal input, please.
Ah. I see you have completely misunderstood the problem. "Too many … " is not the error. It is the title of the panel in my dashboard. The error on both images is “t.data is not iterable”. There is no other information. Please read the initial description again. It is the same error in both images.
I get the same issue.
One way around it is to make one panel and use the inbuilt override features so you can have multiple outputs from a single query.
the queries get big and there are a lot of ephemeral fields but it’s helped me.
Downside is that yes, depending on your region, CloudWatch limits queries to 30 per second. HOWEVER, the start-query is limited to 5 per second. I don’t understand what the difference is in this…but suffice it to say, I frequently get the same “t.data is not iterable” error with no other info! Refresh the page and they all load fine
BIG issue is when someone else is looking at the same time then they all freeze lol
Attached is a single panel which classes a single query…
