Cannot get JMX Metrics but can see them in jmxterm

Grafana Prometheus
1

Using Grafana 11.3.1 and Prometheus v2.55.0 with Docker on a Ubuntu Azure VM

Hello all & Happy New Year,

I’m currently trying to monitor a custom Connector inside a confluentinc/cp-server-connect:7.6.0 . I’m trying to get some JMX metrics from that connector that go beyond the standard metrics of these kafka connectors.

‘Full’ config is at the bottom.
I already added the jmx prometheus agent to the confluentinc/cp-server-connect:7.6.0 with
KAFKA_JMX_OPTS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9998:/usr/share/jmx_exporter/kafka-connect.yml

I also set the KAFKA_JMX_PORT: 9999 and KAFKA_JMX_HOSTNAME: 127.0.0.1 env variables for the connect.

I adjusted the kafka-connect.yml to include the custom metrics and also everything as a fallback like so:

lowercaseOutputName: true
rules:
  # Custom pattern for specific metrics
  - pattern: '^org\\.init\\.ohja\\.kafka\\.connect:connector=(?<connector>[^,]+),task=(?<task>[^,]+),type=(?<type>[^,]+),attribute=(?<attribute>.*)'
    name: 'ohja_kafka_connect_odatav2_${attribute}'
    labels:
      connector: '$connector'
      task: '$task'
      type: '$type'
    attrNameSnakeCase: true
    type: GAUGE
    help: 'Ohja custom metric for attribute: ${attribute}'

  # Fallback rule for all other metrics
  - pattern: '.*'
    name: 'fallback_$0'
    type: GAUGE

When looking in prometheus however I can see every jmx metric besides my custom ones from the connector.
So I ssh into my VM and try to find the metrics with JMXTerm and lo and behold there they are with the following steps:

java -jar jmxterm-1.0-alpha-4-uber.jar
Welcome to JMX terminal. Type "help" for available commands.
$>open 127.0.0.1:9999
#Connection to 127.0.0.1:9999 is opened
$>info -b org.init.ohja.kafka.connect:connector=Connector_David,task=0,type=odatav2-source-task-metrics
#mbean = org.init.ohja.kafka.connect:connector=Connector_David,task=0,type=odatav2-source-task-metrics
#class name = org.apache.kafka.common.metrics.JmxReporter$KafkaMbean
# attributes
  %0   - 0-entityset (double, r)
  %1   - 0-position (double, r)
  %2   - 0-service (double, r)
  %3   - 0-service-url (double, r)
  %4   - 0-topic (double, r)
  %5   - 0-uri-type (double, r)
  %6   - active-subscriptions (double, r)
  %7   - last-extraction (double, r)
  %8   - retries (double, r)
#there's no operations
#there's no notifications
$>

Going to IP_OF_MY_VM:9998 I can see:

# HELP org_init_ohja_kafka_connect_kafka_metrics_count_count total number of registered metrics org.init.ohja.kafka.connect:name=null,type=kafka-metrics-count,attribute=count
# TYPE org_init_ohja_kafka_connect_kafka_metrics_count_count untyped
org_init_ohja_kafka_connect_kafka_metrics_count_count 10.0

but no other metrics.
The full domain in jmxterm would be:

#domain is set to org.init.ohja.kafka.connect
$>beans
#domain = org.init.ohja.kafka.connect:
org.init.ohja.kafka.connect:connector=OData2SourceConnectorConnector_David,task=0,type=odatav2-source-task-metrics
org.init.ohja.kafka.connect:type=kafka-metrics-count
$>info -b org.init.ohja.kafka.connect:connector=OData2SourceConnectorConnector_David,task=0,type=odatav2-source-task-metrics
#mbean = org.init.ohja.kafka.connect:connector=OData2SourceConnectorConnector_David,task=0,type=odatav2-source-task-metrics
#class name = org.apache.kafka.common.metrics.JmxReporter$KafkaMbean
# attributes
  %0   - 0-entityset (double, r)
  %1   - 0-position (double, r)
  %2   - 0-service (double, r)
  %3   - 0-service-url (double, r)
  %4   - 0-topic (double, r)
  %5   - 0-uri-type (double, r)
  %6   - active-subscriptions (double, r)
  %7   - last-extraction (double, r)
  %8   - retries (double, r)
#there's no operations
#there's no notifications

But only the metrics count actually shows up in prometheus.

My full configuration is as follows:

Prometheus & Grafana in my docker-compose.yml

  prometheus:
    image: prom/prometheus:v2.55.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    depends_on:
      - connect

  grafana:
    image: grafana/grafana:11.3.1
    ports:
      - "3000:3000"
    volumes:
      - "/home/Admin/monitoring/grafana-data:/var/lib/grafana"
      - "/home/Admin/monitoring/grafana/provisioning:/etc/grafana/provisioning"

Connect in my docker-compose.yml:

  connect:
    image: confluentinc/cp-server-connect:7.6.0
    hostname: connect
    container_name: connect
    volumes:
      - "/home/Admin/docker-volume/docker-volume-confluent:/usr/share/confluent-hub-components"
      - "/home/Admin/monitoring:/usr/share/jmx_exporter/"
      - "/home/Admin/docker-volume/docker-volume-schemas:/app/schemas"
      - /home/Admin/monitoring/kafka-logs:/var/log/kafka-connect
    depends_on:
      - broker
      - controller
      - schema-registry
    ports:
      - "8083:8083"
      - "9999:9999"  # JMX port
      - "9998:9998"  #Testing
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      # CLASSPATH required due to CC-2422
      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-7.6.0.jar
      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.I0Itec.zkclient=ERROR,org.reflections=ERROR,com.sap.conn.jco=ERROR,org.init.ohja.kafka.connect=DEBUG
      #Specify a Log4j properties file at Kafka startup
      KAFKA_LOG4J_OPTS: "-Dlog4j.configuration=file:/usr/share/confluent-hub-components/connect-log4j.properties"
      _JAVA_OPTIONS: -Xmx6g -Xms6g -XX:MaxPermSize=1024m
      CONNECT_EXACTLY_ONCE_SOURCE_SUPPORT: enabled
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: 127.0.0.1
      KAFKA_OPTS: "-Dcom.sun.management.jmxremote=true \
             -Dcom.sun.management.jmxremote.authenticate=false \
             -Dcom.sun.management.jmxremote.ssl=false \
             -Djava.rmi.server.hostname=127.0.0.1 \
             -Dcom.sun.management.jmxremote.port=9999 \
             -Dcom.sun.management.jmxremote.rmi.port=9999"
      KAFKA_JMX_OPTS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9998:/usr/share/jmx_exporter/kafka-connect.yml

prometheus.yml:

global:
  scrape_interval: 10s
  scrape_timeout: 10s
  evaluation_interval: 10s

rule_files: ["rules/*.yml"]

scrape_configs:
  - job_name: kafka
    scrape_interval: 10s
    scrape_timeout: 10s
    scheme: http
    static_configs:
      - targets: ["broker:9200"]

  - job_name: kafka-connect
    scrape_interval: 10s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ["connect:9998"]

  - job_name: connect
    scrape_interval: 10s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ["connect:9999"]

  - job_name: ksql
    scrape_interval: 10s
    scrape_timeout: 10s
    scheme: http
    static_configs:
      - targets: ["ksqldb-server:7010"]

kafka-connect.yml:

lowercaseOutputName: true
rules:
  - pattern: '^org\\.init\\.ohja\\.kafka\\.connect:connector=(?<connector>[^,]+),task=(?<task>[^,]+),type=(?<type>[^,]+)'

    # Tells jmx_exporter to create separate metrics for each attribute name.
    # 'attrNameSnakeCase: true' converts `last-extraction` -> `last_extraction`.
    attrNameSnakeCase: true
    
    # Name for each metric; the placeholder ${attr} becomes the attribute name.
    name: "ohja_kafka_connect_odatav2_${attr}"
    
    labels:
      connector: "$connector"
      task: "$task"
      type: "$type"
    
    # Typically a gauge is fine for these custom attributes
    type: GAUGE
    help: "Ohja custom metric: ${attr}"

  - pattern: ".*"
    name: "fallback_$0"
    type: GAUGE

In Prometheus the status for kafka-connect is UP, while the status for connect is DOWN. Going to IP_OF_MY_VM:9998 shows me the metrics etc, going to IP_OF_MY_VM:9999 shows ERR_EMPTY_RESPONSE

I hope someone can help me with this as I am struggling to get these other metrics into Prometheus.

Best regards, Looking forward to a great year 2025
David

2

So after digging deeper I found out that Prometheus or the Prometheus Exporter skips the Bean entirely if one of the metrics (attributes?) inside it are not of type double. Some of the metrics were of type string but advertised themselves as doubles inside jmxterm which led to the Bean being skipped.
My solution was then to collect these with Logstash and Loki and the jmx input and do some complicated pipelines to get the metrics and what I need into grafana.