Issue with remote write to loki on grafana cloud - tls: failed to verify certificate: x509: certificate signed by unknown authority

Grafana Loki
1

I have a requirement to read kafka msgs and push to loki. I see a example repo on github for this and trying to implement grafana kafka example on my local. Created grafana cloud account to push logs to loki.

So far, all container are up and running, however, promtail is unable to push logs to loki. Here is the error msg:

2024-05-24 11:37:52 level=warn ts=2024-05-24T16:37:52.604389637Z caller=client.go:419 component=client host=<loki host> msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://<loki host>/loki/api/v1/push\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

Below are configs from promtail config and docker compose files:
promtail-config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/promtail-positions.yaml

clients:
  - url: ${GRAFANA_LOGS_WRITE_URL}
    basic_auth:
     username: ${USER_NAME}
     password: ${PASSWORD}

scrape_configs:
- job_name: varlogs
  static_configs:
  - targets:
      - localhost 
    labels:
      job: varlogs
      __path__: /var/log/*.log
- job_name: kafka
  kafka:
    brokers:
    - kafka_broker:9092
    group_id: grafana_group
    topics:
    - grafana
    labels:
      job: kafka
      host: ${HOSTNAME:promtail}
  relabel_configs:
      - action: replace
        source_labels:
          - __meta_kafka_topic
        target_label: topic
      - action: replace
        source_labels:
          - __meta_kafka_partition
        target_label: partition
      - action: replace
        source_labels:
          - __meta_kafka_group_id
        target_label: group

docker-compose:

# promtail consumes messages from a kafka topic and remote writes them to Grafana Logs
  promtail:
    image: grafana/promtail:3.0.0
    depends_on:
      - kafka_broker
    deploy:
      replicas: 2
    restart: always  
    volumes:
      - /var/log:/var/log
      - ./:/etc/promtail/
    command: [-config.expand-env=true, -config.file=/etc/promtail/config.yml]

Appreciate your help to resolve. Thank you

2

What’s the value of GRAFANA_LOGS_WRITE_URL please?

3

https://logs-prod-006.grafana.net/loki/api/v1/push

Trying to push logs to loki with this url.

4

That URL has valid certificate. I guess you are in the environment, where security guys are running deep packet inspection and using own CA cert.

5

Any options to bypass deep packet inspection or how to get this working?

6

Work around (not the solution) is to enable insecure_skip_verify: true to ignore cert verification issue.

Proper secure solution: get used CA certificates from security guys and make it available ca_file for the promtail in the container.

7

Hi jangaraj,

I did try insecure_skip_verify: true, however, I was seeing 401 unauthorized error. I was so deep into the issue, I ignored the error. I realized after your response, that if this is the workaround why didn’t it work for me? I generated a new API token and gave a shot, the magic worked. Thank you so much and appreciate for timely response. Apologies for not responding back in time.

8

401 error is not related to TLS config . That indicates wrong credentials.