RPC resource exhausted when querying a large data set

Grafana Configuration
, ,
1

Hello,

I’m facing an issue when trying to fetch a large data set from a PostgreSQL database using the Zabbix plugin with direct DB connection configured.

My Grafana instance is deployed using the latest available Docker grafana-oss image currently available (8.5.1).

I’m seeing in the log that it seems related to a limit of message size on gRPC:

logger=context traceID=00000000000000000000000000000000 userId=6 orgId=1 uname=XXXXXX t=2022-05-04T07:03:21.63+0000 lvl=eror msg="Failed to call resource" error="failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (32857275 vs. 16777216)" traceID=00000000000000000000000000000000

I searched the documentation and was able to locate a way to increase the limit value for the server part:

GF_SERVER_GRPC_SERVER_MAX_RECV_MSG_SIZE = 4194304000
GF_SERVER_GRPC_SERVER_MAX_SEND_MSG_SIZE = 4194304000

But I’m still facing the issue after this change. I located the following thread that speaks about a similar issue (How to query a 500MB trace?) and I tried to add the following environment variables without success:

GF_QUERIER_FRONTEND_WORKER_GRPC_CLIENT_CONFIG_MAX_RECV_MSG_SIZE = 4194304000
GF_QUERIER_FRONTEND_WORKER_GRPC_CLIENT_CONFIG_MAX_SEND_MSG_SIZE = 4194304000

Does anyone have an idea how to change this 16777216 limit ?

Thank you

2

@anon51471921 I’m not very familiar with Zabbix, but it looks like that message might refer to a configured limit on the database server if you’ve already changed the settings in your Grafana instance and it’s still complaining.

I found this old thread in the zabbix forums:

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/11204-zabbix-pre-1-6-x-6216-memory-error#post104945

3

Thank you for the pointer, but it’s not really related to this issue we were facing.

If anyone come to this issue looking for a solution, we ended up creating calculated items in Zabbix with a script in order to reduce the amount of data that Grafana has to fetch from the database. It’s not a perfect solution as it increase the amount of data that Zabbix stores in the database, but it’s working…

1 Like