Recover Kubernetes Nodes names and use them in prometheus.scrape component (metrics_path)

Grafana Alloy
1

Hello there,

I am scraping two Prometheus endpoints exposed by Kubernetes API.
Both of those endpoints are hidden behind the name of their respective nodes, which I need to recover, in case the nodes are rebooted and the names changes (for any reason).

I am trying to achieve the following:

  • Recover the node names via a discovery.kubernetes component and some relabelling
  • Wrap all of this in a custom component that would export the two names of the nodes
  • Use those exported names inside the metrics_path argument of the prometheus.scrape component.

Currently, it looks like this:

// Custom resource containing node names as variables
declare "filtered_node_names" {
  
  // Discover all nodes in the Kubernetes cluster
  discovery.kubernetes "nodes" {
    role = "node"
  }

  // Relabel nodes to extract their names
  discovery.relabel "node_names" {
    targets = discovery.kubernetes.nodes.targets

    // Extract the node name into a label
    rule {
      source_labels = ["__meta_kubernetes_node_name"]
      target_label  = "node_name"
      action        = "replace"
    }

    // Keep only the 'node_name' label
    rule {
      action = "labelkeep"
      regex  = "node_name"
    }
  }

  // Filter nodes containing 'deployments' in their name
  discovery.relabel "deployments_nodes" {
    targets = discovery.relabel.node_names.output

    rule {
      source_labels = ["node_name"]
      regex         = ".*deployments.*"
      action        = "keep"
    }
  }

  // Filter nodes containing 'cronjobs' in their name
  discovery.relabel "cronjobs_nodes" {
    targets = discovery.relabel.node_names.output

    rule {
      source_labels = ["node_name"]
      regex         = ".*cronjobs.*"
      action        = "keep"
    }
  }

  // Export the name of a node containing 'deployments' in its name
  export "deployment_node_name" {
    value = discovery.relabel.deployments_nodes.output
  }

  // Export the name of a node containing 'cronjobs' in its name
  export "cronjob_node_name" {
    value = discovery.relabel.cronjobs_nodes.output
  }
}

// Instantiate the custom resource
filtered_node_names "nodes" {}

// Scrape Kube-Deployments
prometheus.scrape "kube_deployments" {

  targets = [{
    __address__ = env("KUBERNETES_SERVICE_HOST") + ":" + env("KUBERNETES_SERVICE_PORT"),
  }]

  scheme = "https"
  tls_config {
      ca_file = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
      insecure_skip_verify = false
  }
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  
  scrape_interval = "10s"
  honor_labels = true

  metrics_path = "/api/v1/nodes/" + filtered_node_names.nodes.deployment_node_name[0].node_name + "/proxy/metrics/resource"

  forward_to = [prometheus.relabel.filter_namespace.receiver]
}

// Scrape Kube-Cronjobs
prometheus.scrape "kube_cronjobs" {
  targets = [{
    __address__ = env("KUBERNETES_SERVICE_HOST") + ":" + env("KUBERNETES_SERVICE_PORT"),
  }]

  scheme = "https"
  tls_config {
      ca_file = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
      insecure_skip_verify = false
  }
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"

  scrape_interval = "10s"
  honor_labels = true

  metrics_path = "/api/v1/nodes/" + filtered_node_names.nodes.cronjob_node_name[0].node_name +  "/proxy/metrics/resource"

  forward_to = [prometheus.relabel.filter_namespace.receiver]
}

Yes that’ s a lot of code.

Two problems here:

  • First I tried to directly export the strings instead of the array in the export blocks, but alloy keeps saying that the length of the array is 0, so it cannot access it’s content. This is very strange as the gui shows perfectly well the array, containing an object containing the key/value pair “node_name”
  • Secondly the prometheus.scrape cannot access the values exported by my custom component. It says it does not exist or is out of scope.

If you read this post entirely, well thank you very much !
Any help will be very much appreciated.

Best,
Thibault