Grafana-Agent Error - x509: certificate signed by unknown authority

Grafana Configuration
, ,
1

Hi, i’ve trying to implement grafana-agent using OTLP to exporter traces with .net 6, but I received this error:

"Exporting failed. No more retries left. Dropping data." error=“max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"” dropped_items=6"

Here is my codes:

grafana-agent.yaml

traces:
  configs:
  - name: default
    remote_write:
      - endpoint: tempo-us-central1.grafana.net:443
        basic_auth:
          username: MY_TEMPO_USER
          password: MY_TEMPO_PASSWORD

    receivers:
      otlp:
        protocols:
          grpc:

docker-compose.yaml

version: '3.4'

services:
  api:
    container_name: device-check
    image: ${DOCKER_REGISTRY-}api
    build:
      context: .
      dockerfile: API/Dockerfile
    ports:
      - "8080:80"

  grafana-agent:
    image: grafana/agent:latest
    container_name: grafana-agent
    command: "-config.file=/etc/agent-config.yaml"
    volumes:
      - ./agent-config.yaml:/etc/agent-config.yaml
    ports:
      - "12345:12345"

my services config in program.cs:

services.AddOpenTelemetry().WithTracing(builder =>
            {
                builder.AddOtlpExporter(options =>
                {
                    options.Endpoint = new Uri("http://grafana-agent:4317");
                })
                .AddConsoleExporter()
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("app"))
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation()
                .AddSource("my-source");
                
            }).StartWithHost();

Can anybody help me? I can’t identify what I’m doing wrong.

I tried to see a lot of examples on internet, but no example helped me.

2

First try to use latest image = image: grafana/agent:v0.30.1, because it is not clear which version you are using atm. Yes you have :latest tag, but that latest can be downloaded a years ago, when latest tag was different, so you have old image with old CA certs, which is causing your problem.

3

I changed to the image “grafana/agent:v0.30.1”, but the same error are occuring.
Is a possibility that error is occurring because i’ve trying to implement in my worker laptop, and probably my company has security politics?

4

Yes, deep packet inspection will use company CA certificate (for tempo-us-central1.grafana.net:443), which is not available in default common CA certs. You can disable cert verification (insecure option) or make used company CA cert available in used Docker image.

1 Like
5

Hey, the problem was occurring because my worker laptop had security politics. When I changed to my MAC, the problem didn’t exist more.

Thank’s for the help.

:slight_smile:

6

I like the concept of “security politics”.

It applies to so many things in the modern world.

Antony.