RPC resource exhausted when querying a large data set

Hello,

I’m facing an issue when trying to fetch a large data set from a PostgreSQL database using the Zabbix plugin with direct DB connection configured.

My Grafana instance is deployed using the latest available Docker grafana-oss image currently available (8.5.1).

I’m seeing in the log that it seems related to a limit of message size on gRPC:

logger=context traceID=00000000000000000000000000000000 userId=6 orgId=1 uname=XXXXXX t=2022-05-04T07:03:21.63+0000 lvl=eror msg="Failed to call resource" error="failed to receive call resource response: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (32857275 vs. 16777216)" traceID=00000000000000000000000000000000

I searched the documentation and was able to locate a way to increase the limit value for the server part:

GF_SERVER_GRPC_SERVER_MAX_RECV_MSG_SIZE = 4194304000
GF_SERVER_GRPC_SERVER_MAX_SEND_MSG_SIZE = 4194304000

But I’m still facing the issue after this change. I located the following thread that speaks about a similar issue (How to query a 500MB trace?) and I tried to add the following environment variables without success:

GF_QUERIER_FRONTEND_WORKER_GRPC_CLIENT_CONFIG_MAX_RECV_MSG_SIZE = 4194304000
GF_QUERIER_FRONTEND_WORKER_GRPC_CLIENT_CONFIG_MAX_SEND_MSG_SIZE = 4194304000

Does anyone have an idea how to change this 16777216 limit ?

Thank you