Grafana and Zabbix plugin issue

jamesw6 · June 11, 2026, 1:25pm

What Grafana version and what operating system are you using? Grafana version: v13.0.2, Ubuntu 26.04
What are you trying to achieve? Trying to add Zabbix as a data source for dashboards
How are you trying to achieve it? Setting up a new data source, clicking the “save and test” button never completes. If i attempt to add a dashboard widget and select the zabbix data source, it times out.
What happened? See above
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were. from the grafana/zabbix plugin log: logger=plugin.alexanderzobnin-zabbix-datasource t=2026-06-11T13:08:49.131701973Z level=error msg=“Partial data response error” pluginVersion=6.4.0 uname=admin dsName=alexanderzobnin-zabbix-datasource pluginId=alexanderzobnin-zabbix-datasource refID=A status=408 statusSource=downstream dsUid=efotdz8fi3chsf endpoint=queryData error=“query timeout: Query execution exceeded maximum allowed time (1m0s). Query was automatically terminated to prevent excessive resource consumption.”
logger=plugin.alexanderzobnin-zabbix-datasource t=2026-06-11T13:08:49.132162044Z level=error msg=“Plugin Request Completed” statusSource=downstream uname=admin dsName=alexanderzobnin-zabbix-datasource dsUid=efotdz8fi3chsf duration=1m0.001197991s endpoint=queryData pluginId=alexanderzobnin-zabbix-datasource pluginVersion=6.4.0 status=error
logger=plugin.alexanderzobnin-zabbix-datasource t=2026-06-11T13:12:01.631909184Z level=info msg=“Plugin Request Completed” dsName=alexanderzobnin-zabbix-datasource dsUid=efotdz8fi3chsf duration=15.892501ms pluginId=alexanderzobnin-zabbix-datasource statusSource=plugin endpoint=callResource pluginVersion=6.4.0 status=ok uname=admin".
Zabbix server is brand new, started this week. Resources are fine, CPU usage usually around 8% and 80% free memory.

infofcc3 · June 12, 2026, 8:57am

Your error is a query timeout → the plugin hit the default 1 minute execution limit. Fix these two things first:

1 → Increase query timeout:
Grafana → Zabbix data source → Additional settings → Query Timeout → set to 300

2 → Enable Trends:
Additional settings → Trends → Enable Trends → set After to 7d

If still timing out, check which setup applies to you:

If running on bare Ubuntu:
Run systemctl status zabbix-agent and confirm agent is running
Check Zabbix UI → Monitoring → Hosts → Zabbix server → Interfaces → confirm agent IP is correct

If running in Docker:

Run docker ps and confirm these 3 containers are running: zabbix-server, zabbix-agent, zabbix-web

Screenshot (592)1920×1080 250 KB
If zabbix-agent is missing:
docker run -d --name zabbix-agent --network <your-zabbix-network> -e ZBX_HOSTNAME="Zabbix server" -e ZBX_SERVER_HOST="zabbix-server" zabbix/zabbix-agent2:latest
If agent is running but still timing out: Zabbix UI → Monitoring → Hosts → Zabbix server → Interfaces → clear IP → set DNS name to zabbix-agent → Connect to DNS → Update

Screenshot (591)1920×1030 134 KB
Confirm Grafana is on the same Docker network: docker network connect <your-zabbix-network> <your-grafana-container>
Set data source URL to: http://zabbix-web:8080/api_jsonrpc.php

After all fixes Save & Test should show Zabbix API version and data will appear in your panels.

jamesw6 · June 12, 2026, 11:19am

Hey, thank you for the reply! I have tried with the settings you suggested, but it still hangs - I have noticed in zabbix itself it reports its CPU utilization as anywhere between 30 and 90% while this runs - if i close/refresh the page it drops. the vm is running on bare ubuntu, everything is running as it should, i’ve rebooted a couple of times and rechecked but it’s always the same. if i run top on the vm, i’ll see instances of mysql and php at the top of cpu utlization, but its never really maxed out - VM is is equipped with 4cpu and 8gb ram.

infofcc3 · June 12, 2026, 12:27pm

Thanks for the update. The timeout is being caused by high MySQL CPU load when Grafana queries Zabbix.

1 → Enable Trends (most impactful):
Zabbix data source → Additional settings → Trends → Enable Trends → set After to 7d
This uses pre-aggregated data instead of raw history — dramatically reduces MySQL load

2 → Narrow your default time range:
Set dashboard default time range to Last 1 hour instead of Last 6 hours
Wider time ranges = more MySQL rows = higher CPU

3 → Enable Direct DB Connection:
Zabbix data source → Additional settings → Direct DB Connection → Enable
This bypasses the Zabbix API entirely → faster on wide time ranges
You need to configure a MySQL data source in Grafana first → the plugin needs only read access to history, history_uint, trends and trends_uint tables

4 → Check MySQL slow queries:

bash

sudo mysql -u root -p -e "SHOW VARIABLES LIKE 'slow_query_log%';"

If slow_query_log = ON, tail the log path shown → it will show exactly which queries are spiking CPU.

Direct DB setup → Configure the Zabbix data source | Grafana Plugins documentation

jamesw6 · June 15, 2026, 9:10am

Hey, thank you for the reply! So, trends have been enabled with no change, i usually have the dashboards set to 1hr that i can see, it never completes, I have setup a mysql db connection, this saves and tests succesfully almost instantly, i added it to the zabbix plugin, but i get the same issue, it hangs on saving and testing and nothing ever loads in the dashboard, it seems it’s still reliant on the zabbix plugin. I’ve enabled slow queries and tailed the log while it runs, the only clue that has given is that there were 600k+ session records on the database. I cleared them down to the last 5 minutes, ran save and test and sure enough the session count starts rising again immediately. i’m not a db expert, claudeai is helping me out but it seems like the plugin is hammering the api and causes it to grind to a halt

jamesw6 · June 15, 2026, 9:53am

So with a bit more help from Claude, the solution was:

OPcache disabled — was causing empty API responses
Apache Authorization header — SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1 — Apache was stripping the auth header before it reached PHP
API token instead of username/password — prevented the sessions table from flooding with hundreds of thousands of records and grinding MySQL to a halt

bernardolankheet · June 15, 2026, 12:05pm

Hi @jamesw6

This is probably a bug.

It was reported in this issue:

github.com/grafana/grafana-zabbix

[CRITICAL BUG] 6.3.1 / 6.3.2: N+1 history.get queries per problem cause severe DB overload and PHP-FPM timeouts in large environments

opened 10:21AM - 25 May 26 UTC

christos-diamantis

<html><head></head><body> <h2>Summary</h2> <p>Versions <strong>6.3.1</strong> an…d <strong>6.3.2</strong> introduced a per-problem <code>history.get</code> API call pattern that generates a massive number of individual <code>SELECT</code> queries against the Zabbix <code>history_uint</code> (and <code>history</code>) tables. In large environments with many hosts and items, this causes:</p> <ul> <li>PHP-FPM workers timing out (10–13+ seconds per <code>api_jsonrpc.php</code> request)</li> <li>PostgreSQL saturated with slow <code>history_uint</code> range scans averaging <strong>2,000–6,000ms each</strong></li> <li>Zabbix web UI becoming unresponsive or unusable</li> <li>Cascading load on the database cluster</li> </ul> <p><strong>Rolling back to 6.3.0 immediately resolves all symptoms.</strong> This is a confirmed regression introduced in 6.3.1.</p> <hr> <h2>Environment</h2> Component | Details -- | -- Plugin version (broken) | alexanderzobnin-zabbix-app 6.3.1 and 6.3.2 Plugin version (working) | alexanderzobnin-zabbix-app 6.3.0 ✅ Zabbix version | 7.x Database | PostgreSQL 18 with TimescaleDB, Patroni HA cluster (3 nodes) Zabbix frontend | 2-node HA, PHP-FPM 8.x, pm = static, 150 workers per node Grafana nodes | 4 nodes behind F5 LB Scale | Very large environment: thousands of hosts and millions of items <hr> <h2>Suggested Fix</h2> <h3>Option 1 — Batch the <code>history.get</code> calls (minimum viable fix)</h3> <p>Instead of issuing one API call per problem, the plugin should:</p> <ol> <li>Collect all <code>(itemid, timestamp)</code> pairs for all visible problems</li> <li>Issue a <strong>single batched <code>history.get</code> request</strong> covering all itemids and the full timestamp range needed</li> <li>Map results back to individual problems client-side</li> </ol> <p>This is the same approach that worked correctly in 6.3.0 for operational data — the fix in 6.3.1 should have preserved batching semantics while correcting the value-at-timestamp logic.</p> <p>Alternatively, if per-timestamp accuracy requires individual calls, implement a <strong>request queue with concurrency limiting</strong> (e.g., max 5 parallel <code>history.get</code> calls at a time) to prevent thundering-herd DB load.</p> <h3>Option 2 — Make the feature opt-in via a plugin/panel setting (strongly recommended)</h3> <p>The historical-value-at-problem-creation-time feature is a correctness improvement that is valuable in small and medium environments. However, it is <strong>not needed by all customers</strong> and is <strong>actively harmful at scale</strong>.</p> <p>We strongly recommend exposing this as a toggle in the Problems panel options or in the datasource configuration, for example:</p> <pre><code>Problems Panel Options └── Operational Data └── [ ] Fetch historical item value at problem creation time (Warning: may impact performance in large environments with many active problems. Disabled by default.) </code></pre> <p>Or at the datasource level:</p> <pre><code>Datasource Settings └── Performance └── [ ] Enable historical value resolution for operational data Default: OFF </code></pre> <p>This way:</p> <ul> <li>Small environments that want accurate <code>{ITEM.VALUE}</code> at problem time can opt in</li> <li>Large environments with thousands of hosts and millions of items are not impacted by default</li> <li>The behavior of 6.3.0 is preserved for anyone who does not explicitly enable it</li> </ul> <p>This pattern is consistent with how other high-cost features are handled in Grafana plugins and avoids forcing a performance trade-off onto all users regardless of their scale.</p> <hr> <h2>Workaround</h2> <p>Roll back to version <strong>6.3.0</strong> until a fix is available:</p> <pre><code class="language-bash">grafana-cli plugins install alexanderzobnin-zabbix-app 6.3.0 systemctl restart grafana-server </code></pre> <hr> <h2>Additional Context</h2> <ul> <li>Database: PostgreSQL 18 + TimescaleDB, Patroni 3-node HA cluster</li> <li><code>pg_stat_statements</code> was used to capture query-level evidence</li> <li>PHP-FPM pool: <code>pm = static</code>, 150 workers per node (ruling out worker exhaustion as cause)</li> <li>DB connection count at time of issue: ~102/197 (ruling out connection exhaustion as cause)</li> <li>The issue is <strong>purely query volume/frequency</strong> introduced by the N+1 pattern in 6.3.1+</li> </ul></body></html>

An alternative is to reinstall an older version of the plugin.

grafana-cli plugins install alexanderzobnin-zabbix-app 6.3.0 systemctl restart grafana-server

Topic		Replies	Views
Zabbix plugin issue Grafana	0	470	September 18, 2019
Grafana 7 and Zabbix Plugin Grafana	0	348	June 4, 2020
Zabbix datasource in grafana Grafana datasource	1	241	February 29, 2024
Zabbix Plugin for Grafana Zabbix	5	3609	December 18, 2023
No longer receiving zabbix data Zabbix zabbix	4	5377	May 8, 2020

Grafana and Zabbix plugin issue

Related topics