If it’s not that, and you’ve rolled out the agent as a DaemonSet so there is an agent on the node running kube-state-metrics (in this case host filtering doesn’t matter), here are some things you can do to debug:
- Call the API to make sure the target shows up: https://github.com/grafana/agent/blob/master/docs/api.md#list-current-scrape-targets
- If it shows up and there are no errors, use
agentctl target-statsto make sure data is in the WAL (docs aren’t great here, might need to dig around) - Last resort: enable debug logging to see what’s going on: agent/configuration-reference.md at main · grafana/agent · GitHub
Other potential sources for this issue:
- K8s RBAC (I think unlikely given other stuff is working fine)
- K8s DNS (i think also unlikely)
- If you have any filtering rules in your config after
targets, the metrics could be getting dropped there. Hard to tell without sharing more of your config.
Hope this helps you get started resolving your issue. Please post your solution here so others can learn how you fixed this!