AQL queries take 30+ min even on a 5-min time range, but CPU/RAM on both Grafana and QRadar are low — why?

Grafana
, , , ,
1

Hi everyone,

I have Grafana integrated with IBM QRadar SIEM (v7.5.0 UP12) using the ibm-aql-datasource plugin. The setup has 9 dashboards with 60+ panels total, each running AQL queries against QRadar.

The strange issue I’m seeing: even a query with a 5-minute time range (or any arbitrary range) can take 30+ minutes to return results. What’s odd is that resource usage doesn’t explain it at all:

  • Grafana server: CPU and Memory usage stay low during these slow queries, despite the server having decent specs.
  • QRadar server: CPU and RAM also stay low, and QRadar itself has high specs (this isn’t an under-provisioned All-in-One appliance).

So neither side appears to be resource-starved, yet queries crawl. A few relevant details that I think rule out the most obvious causes:

  • Concurrent users: only 1–2 people access these dashboards at a time — so this isn’t a high-concurrency load issue.
  • Auto-refresh: disabled. Dashboards are refreshed manually, so there’s no overlapping query burst from scheduled refresh intervals.
  • Despite this low usage pattern, queries still take 30+ minutes.

A few things I’m trying to rule out:

  1. AQL job queuing on QRadar — even with just 1-2 users, do queries from 60+ panels still queue server-side if triggered close together (e.g. loading a dashboard with many panels at once)? Is there a per-tenant or per-token concurrent search limit relevant here?
  2. Plugin-side timeout/retry behavior — the datasource has a configurable “Plugin Timeout” (default 5m). Could queries be silently timing out, retrying, and stacking up — which would explain low CPU but long wall-clock time?
  3. Ariel search engine I/O/disk bottleneck — since QRadar’s search engine relies heavily on disk I/O for the Ariel database rather than CPU/RAM, could the bottleneck be disk I/O or storage throughput on QRadar’s Ariel proxy, which wouldn’t show up as high CPU/RAM usage?
  4. Search job overhead per panel — does each panel submit a separate AQL search job (rather than batching), and if so, does QRadar process these sequentially rather than in parallel, even when resources are free?

Given that load is genuinely low (1-2 users, no auto-refresh), I’m leaning toward this being a QRadar-side search job/Ariel architecture limitation rather than a resource capacity issue — but would appreciate input from anyone who has run a similarly large dashboard setup (60+ panels) against QRadar via this plugin and identified the actual bottleneck.

Thanks !

2

Verify concurrent search limits first

With 60+ panels loading simultaneously, each submitting a separate AQL job, QRadar may be queuing excess searches server-side. Check your current concurrent search limit via:

QRadar Console → Admin → Ariel Query Server Settings(Check your QRadar Admin tab for Ariel or search-related settings, and refer to IBM documentation for your exact v7.5.0 UP12 version)

Compare against how many panels load at once. If the limit is lower than your panel count, queuing is likely contributing to the delay.

Check disk I/O before assuming a resource bottleneck

Low CPU and RAM does not mean QRadar is idle — Ariel searches can be storage I/O intensive. Verify:

bash

iostat -x 1 10
# Check /store partition: high %util or await values indicate I/O pressure

Verify plugin timeout behavior from logs

Check plugin logs to confirm whether a timeout triggers an automatic retry or returns an error. If retries occur while the original job is still running on QRadar, you can end up with compounding search jobs. If that is confirmed, adjusting the timeout is worth considering — but fixing the underlying query load matters more.

Consolidate panels → safest immediate action

Regardless of root cause, reducing the number of AQL jobs submitted on dashboard load helps. Combine similar queries:

sql

SELECT category, count(*) FROM events
WHERE category IN (1,2,3,4,5)
GROUP BY category LAST 5 MINUTES

This is the one change you can make right now without touching QRadar configuration.

4

Hello @irvanfebrianto1998
I just wanted to know did the solution work out for you? I can help further if you’re still running into any issues.

5

Hi @infofcc3

For now, I/O is still at a medium level, not critical — remaining RAM is 20GB and CPU is mostly idle and within normal range. I haven’t extended the Ariel search limit yet, but I did reduce the time range in the AQL queries, changing DESC LIMIT 10 LAST 3 HOURS to DESC LIMIT 10 LAST 30 MINUTES on each panel.

This was inspired by your suggestion to adjust the Ariel limit in QRadar — instead of changing it at the QRadar level, I adjusted the limit directly in the AQL queries.

That approach worked, but I’ll also try your suggestion to adjust the Ariel limit itself, since not all log sources have volume as high as firewall logs. Some servers still need to keep the 3-hour (or longer) range, and those queries are still running slow.

6

Glad to hear reducing the time window helped.

For the remaining 3-hour+ panels, check whether your AQL queries are filtering on indexed fields wherever possible.

It’s also worth avoiding SELECT * unless you truly need every field, since pulling fewer columns reduces I/O during the scan.

Since query execution happens on the QRadar/Ariel side, these optimizations are outside the scope of the Grafana plugin itself, which mainly handles connectivity and query submission.

If those optimizations still don’t bring the longer-range panels to an acceptable runtime then run the same AQL query directly in QRadar, outside Grafana. If it is also slow there that would suggest the bottleneck is QRadar/Ariel rather than the plugin

7

Thank you for the help! I understand this is still quite new for me — I’ve mostly been following the AQL queries that were originally written by the NOC team.

Regarding SELECT *, since all panels in my dashboard are currently using it, I’ll review them one by one and remove unnecessary fields where possible.

You were right — when I tested the same query directly in QRadar, it was also slow. I initially assumed the Grafana plugin was the bottleneck, but now it’s clear the issue is on the QRadar/Ariel side, not the plugin itself.

8

also read the documentation for QAdr rest api instead of carving your own queries. Maybe the rest api is more performant.