Grafana Agent auth handshake failed

tzaspelfinatix · November 16, 2021, 3:19pm

Hello

I’ve deployed a Grafana Agent for tracing and Grafana Tempo. Now I want to connect both with each other. The Agent is trying to export its (logged) data to Tempo Distributor on the jaeger grpc port (14250) and only returning me the following Errorlog: ts=2021-11-16T14:59:36Z level=error caller=exporterhelper/queued_retry.go:165 msg="Exporting failed. Try enabling retry_on_failure config option." component=traces traces_config=tempo-grafana-agent kind=exp │ │ orter name=otlp/0 error="failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not l │ │ ook like a TLS handshake\""

Did you ever happen to have this problem or know what the problem might be?

Best regards
Tom

joeelliott · November 16, 2021, 5:00pm

It seems you are attempting to connect to a non-TLS endpoint using TLS. Perhaps try setting insecure=true on your write configuration?

github.com

grafana/agent/blob/main/example/docker-compose/agent/config/agent.yaml#L89

    
      
            jaeger:
              protocols:
                thrift_http:
          attributes:
            actions:
            - action: upsert
              key: env
              value: prod
          remote_write:
            - endpoint: tempo:55680
              insecure: true
          batch:
            timeout: 5s
            send_batch_size: 100
          automatic_logging:
            backend: logs_instance
            logs_instance_name: default
            spans: true
            processes: true
            roots: true
          spanmetrics:

tzaspelfinatix · November 16, 2021, 9:14pm

Thanks again, it works. So I have to adjust the port from 14250 to 55680.

If there are any ways to test if tracing works? Unfortunately I don’t know how I can test if Grafana agent got a trace and if the agent works properly with Tempo.

Thanks
Tom

joeelliott · November 16, 2021, 9:23pm

The logs are good about reporting errors (as you’ve seen) if it is unable to forward traces to Tempo. I would also keep an eye on the following metrics emitted by the agent:

traces_receiver_accepted_spans
traces_exporter_sent_spans
traces_exporter_send_failed_spans
traces_receiver_refused_spans

system · November 16, 2022, 9:23pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.