I’m trying to make a dashboard that visualizes swarm metrics that scrape using remote Prometheus node.I have 4 swarm nodes running cadvisor and node-exporter as global service ,exposed via nginx on port 80 (Loadbalancer) .
My promthues config
- job_name: 'stage_swarm'
scrape_interval: 60s
scrape_timeout: 59s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- cadvisor.stage:80
- exporter.stage:80 ( metrics will be collected from remote swarm cluster node-exproter,via nginx(LB) )
I could see that node_meta has all the swarm nodes listed ,but am not able to visualize all the nodes separately .
How do I create a dashboard for all the host inside node_meta ? When I query using node_name or job ,I always get node count as 1 .
count(count(container_tasks_state{job="stage_swarm"}) by (container_label_com_docker_swarm_node_id)) – always 1
sum((node_memory_MemTotal - node_memory_MemFree - node_memory_Cached - node_memory_Buffers - node_memory_Slab) * on(instance) group_left(node_name) node_meta{job="stage_swarm"}) by (node_name)
sum(node_memory_MemTotal * on(instance) group_left(node_name) node_meta{job="stage_swarm"}) – always wrong. Same issue for all metrics.
Thanks