Promtail is not consuming all the topics from kafka service of Aiven

Grafana Explore
,
1

I am using promtail agent to fetch the kafka topic messages and send to loki but I couldn’t see the topics in the loki datasource of grafana explore. Need help on the configuration part

2

welcome to forum @keerthika

can you share your config obfuscating any sensitive data

3

Hi Yosiasz,
Good to connect with you.
Please find the config below

  - job_name: aiven_kafka
    relabel_configs:
      - source_labels:
          - __meta_kafka_topic
        action: replace
        target_label: kafka_topic
      - source_labels:
          - __meta_kafka_topic
        target_label: topic
      - source_labels:
          - __meta_kafka_broker
        target_label: broker
      - source_labels:
          - __meta_kafka_partition
        target_label: partition
      - source_labels:
          - __meta_kafka_offset
        target_label: offset
      - source_labels:
          - __meta_kafka_group
        target_label: group
    kafka:
      brokers:            
        - xxx.com:22503
      topics:
        - ^test-topic-promtail-aks.*$
      # topics: 
      #   - ^FORMAT_SIGN_SEND.*$
      #   - ^GENERATE_NONCE.*$
      #   - ^GET_GAS_PRICE_AND_ESTIMATE_GAS.*$
      #   - ^PENDING_TRANSACTION_RECEIPT.*$
      authentication:
        type: ssl
        tls_config:
          insecure_skip_verify: true
          ca_file: /etc/secrets/stage-kafka/ca/ca.pem
          cert_file: /etc/secrets/stage-kafka/ssl/tls.crt
          key_file: /etc/secrets/stage-kafka/ssl/tls.key
          server_name: xxx.com
4

I could see there is lag in the topics consumer groups.
Any advise please