ServiceMonitor resolving incorrect service IP

Grafana Alloy
1

I have a simple ServiceMonitor to scrape some services:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: krane-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      krane/scrape-prom-metrics: "true"
  namespaceSelector:
    matchNames:
      - krane
  endpoints:
    - port: web
      path: /metrics

Alloy picks it up fine and also finds the target, but scraping fails because the endpoints are incorrect.

ts=2024-12-09T09:41:54.967799257Z level=debug msg="Scrape failed" component_path=/prometheus_operator_objects.feature component_id=prometheus.operator.servicemonitors.service_monitors scrape_pool=serviceMonitor/monitoring/krane-service-monitor/0 target=http://10.40.4.94:3000/metrics err="server returned HTTP status 404 Not Found"

It tries to scrape on 10.40.4.94 while it should be an address in the 10.44.x.x range. DNS resolution is correct if I run nslookup from a pod (in the same namespace).

I’m running the latest k8s-monitoring helm chart on GKE.

Does anybody have any idea what may cause this issue? Thanks!