Dear Community,
I’m using OpenTelemetry for several microservices, I see grafana-agent logs below from dashboard,
ts=2023-07-13T14:54:01.43025910Z caller=client.go:143 level=info component=logs logs_config=logs-config component=tailer msg "tail routine: started" path = /parentpath/logspath/microservice-1.log
ts=2023-07-13T14:54:01.43025920Z caller=client.go:143 level=info component=logs logs_config=logs-config component=tailer msg "tail routine: started" path = /parentpath/logspath/microservice-2.log
ts=2023-07-13T14:54:01.43025930Z caller=client.go:143 level=info component=logs logs_config=logs-config component=tailer msg "tail routine: started" path = /parentpath/logspath/microservice-3.log
ts=2023-07-13T14:54:01.43025940Z caller=client.go:143 level=info component=logs logs_config=logs-config component=tailer msg "tail routine: started" path = /parentpath/logspath/microservice-4.log
ts=2023-07-13T14:54:01.43025950Z caller=client.go:143 level=info component=logs logs_config=logs-config component=tailer msg "tail routine: started" path = /parentpath/logspath/microservice-5.log
ts=2023-07-13T14:54:34.43025926Z caller=client.go:369 level=warn component=logs logs_config=logs-config component=client host=********************** msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5765855 vs. 4194304)"
ts=2023-07-13T14:54:34.43025936Z caller=client.go:369 level=warn component=logs logs_config=logs-config component=client host=********************** msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4772855 vs. 4194304)"
ts=2023-07-13T14:54:34.43025946Z caller=client.go:369 level=warn component=logs logs_config=logs-config component=client host=********************** msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6375855 vs. 4194304)"
...... not count how many lines ......
As info from various channels, e.g. response-larger-than-the-max, looks like I should configure send and receive max size,
From logs looks like after a series of log file (e.g., microservice-1.log,microservice-2.log,microservice-3.log,etc) relevant operations , then start to get max exceed errors.
The doc tell that,
This can happen in multiple places, but it’s most commonly seen in the
query path, with messages between the querier and the query frontend
But I feel confused about the meaning of 4194304, when the max exceed error happens,
1, It could be the size of a single log file size or sum of multiple log files size?
Say, size(microservice-1.log) > 4194304 or size(microservice-1.log) size(microservice-2.log) + ... > 4194304
2, It could be the size of a single response in a log file or the size sum of multiple responses in a log file/multiple log files)?
Say, I have log.info(#large_response) in my codes while size(large_response) > 4194304 or size(#large_response_1) size(#large_response_2) + ... > 4194304
3, It would also happen when I query a large time-range from Grafana dashboard?
Thanks in advance if anyone has any idea what this.