Alloy grpc behind a Nginx Ingress Error FRAME_SIZE_ERROR 4740180

Grafana Alloy
,
1

Alloy grpc behind a Nginx Ingress Error FRAME_SIZE_ERROR 4740180

I try to deploy an Alloy via helm and create an Ingress by myself to test.
The client send metrics, logs, traces via Opentelemetry collector Java agent. It’s classic configration. I don’t share the config here.

This is my ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grpc-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  tls:
  - hosts:
    - grpc.api.example.com
    secretName: tls-secret
  rules:
  - host: grpc.api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grpc-service
            port:
              number: 4318

The Java client OkHttp send data, but there are a lot of errors:
ERROR io.opentelemetry.exporter.internal.grpc.GrpcExportor - Failed to export logs. The request could not be executed. Error message: FRAME_SIZE_ERROR 4740180

May I know if any idea how to fix this?

2

Try tweaking grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size to a bigger value and see if that helps.

3

Could you pls give a reference where Alloy can config the grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size

I didn’t find the param on otelcol.receiver.otlp | Grafana Alloy documentation

BTW, recently I deployed the otel-collector-contrib behind the Nginx Ingress, I enable the gRPC on it and I use the same Java client to do testing. the otel-collector-contrib can handle the gRPC.

4

Here is an issue for Java otel agent, I think they are the similar

5

The configuration is on Loki, I thought that’s where you had issue with.

If you are getting the error on Alloy then perhaps a bug report on github might be a good idea.

6

The issue was fixed. Let me share my result.

  1. Protocol Mismatch: I use a grpc+http protocol (I should use a https, I missed the s, it’s typo) to connect to a gRPC server(It’s actually a HTTP1.1 server, I though it should be gRPC +HTTP2.0 server) .

FRAME_SIZE_ERROR 4740180, the number is the decimal representation of the hexadecimal bytes 0x485454 (“HTTP”), which the server misinterprets as an excessively large frame size within the HTTP2 protocol.

  1. Nginx Ingress Configuration: for community Nginx Ingress, nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    but in my org, it changes to Nginx plus Ingress. To enable the gRPC, it should be:
    nginx.org/grpc-services: "grpc-svc-input-your-svc-here"