Greetings, I am testing Kafka monitoring using Prometheus using this dashboard: https://grafana.com/grafana/dashboards/10083. There are two panels that have no data points:
- Messages In vs Consumed. It uses these 2 queries:
– avg by (topic) (sum by (topic, kubernetes_pod_name) (rate(kafka_minion_group_topic_partition_offset{group="$consumer_group", topic="$topic"}[1m])))
– avg by (topic) (sum by (topic, kubernetes_pod_name) (rate(kafka_minion_topic_partition_high_water_mark{topic="$topic"}[1m]))) - Offset commits processed. It uses this query:
– avg by (kubernetes_pod_name) (rate(kafka_minion_internal_kafka_messages_in_success[1m]))
The queries have no syntax errors when testing from Prometheus (I substituted the variables with proper info). My colleague tells me that I have all of the metrics, but I cannot confirm this. I am new to Prometheus and am looking to use this instead of other tools we have. Thanks.