Track unused Kafka resources from Confluent metrics

  • What Grafana version and what operating system are you using?
    Grafana v9.4.3 latest installed from running on Windows 10

  • What are you trying to achieve?
    I’m fetching metrics from Confluent Cloud Metrics API /export endpoint for Kafka resources (cluster, topics, connectors, ksqldb, schema). I want to track resources that are not actively used, don’t have throughput or data in metrics. I want to find the following, so I can delete obsolete resources:
    → unused connectors with 0 throuhgput/messages processed.
    → unused clusters with 0 usage metrics, 0 topics/connectors/ksqldb.
    → unused topics with 0 messages produced/consumed.
    → unused ksqldb with 0 streams/tables or queries.
    → unused schemas and schema registry with 0 schemas or topics using it.

  • How are you trying to achieve it?
    I added resource IDs for all clusters, connectors, ksqlDB and schema registry in Prometheus config file. Grafana is set to use this data source. When I use any metric in Grafana, count only displays for which metrics data is coming. But it doesn’t show count of clusters etc. which has no data / metrics data.

  • What happened?
    I’m not at all getting data for unused resources, even if I put resource IDs of all existing clusters and other resources.

  • What did you expect to happen?
    If I put all resource IDs in the Prometheus config file, count of unused resources should also display. If there is no metrics data for any resource, that data should also show as 0 or not used.