Migrating from smokeping to prometheus / grafana

[mirtobusico]

Hi all,

I’m new to Grafana and I’m trying to replace Smokeping with probes / Prometheus / Grafana /dashboards.

Details of what I have done are below.

The documentation of the the packages appear to be frozen at least two years ago.

The Grafana documentation and video seems to refer to an old Grafana version and the interface is different.

I know that I have a lot to learn; but any help/reference/documentation would be greatly appreciated.

My framework:

• What Grafana version and what operating system are you using?

I’m using Grafana 13.0.1-2 on Majaro

• What are you trying to achieve?

I’m trying to migrate from Smokeping to scrapers / Prometheus / Grafana

• How are you trying to achieve it?

I installed

- prometheus-smokeing prober

- prometheus 3.11.3-1

- grafana 13.0.1-2

- this SmokePing dashboard

• What happened?

The system seems to work; but results are different from smokeping

Trying to create a new dashboard using the previous dashboard as a model don’t work because I’m not able to understand the queries and the panel setting

• What did you expect to happen?

To learn how to use Grafana to manage my data

• Can you copy/paste the configuration(s) that you are having problems with?

I’m using the default configurations that come with the packages. I only added the ping targets

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No errors, only results different from what I expected

• Did you follow any online instructions? If so, what is the URL?

The official Prometheus and Grafana (OSS) documentation and the github main page of the packages

[infofcc3]

Welcome @mirtobusico to the community
Smokeping prober v0.11.0 changed metric names. Dashboard 22471 uses old names → that’s why results look wrong.

Old (dashboard expects)	New (v0.11.0 actual)
smokeping_rtt_seconds_bucket	smokeping_response_duration_seconds_bucket

Step 1: Install and Start Prober Correctly

bash

yay -S prometheus-smokeping-prober

Start with --privileged flag (required for RTT histogram data):

bash

sudo tee /etc/systemd/system/smokeping_prober.service << 'EOF'
[Unit]
Description=Smokeping Prober
After=network.target

[Service]
ExecStart=/usr/local/bin/smokeping_prober --privileged 8.8.8.8 1.1.1.1 google.com
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now smokeping_prober

Verify metrics are flowing:

bash

curl -s http://localhost:9374/metrics | grep response_duration | head -5

---

Step 2: Prometheus Config

Add to prometheus.yml under scrape_configs:

yaml

  - job_name: 'smokeping'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9374']

Restart Prometheus:

bash

sudo systemctl restart prometheus

Verify at http://localhost:9090/targets — smokeping job shows UP.

3 → Fix Grafana Dashboard Panel Queries

In Grafana 13 → hover over panel → click ⋮ menu → Edit → switch to Code mode → replace the query.

Latency (median):

promql

histogram_quantile(0.5,
  sum by (le) (
    rate(smokeping_response_duration_seconds_bucket[$__rate_interval])
  )
) * 1000

Jitter →

promql

(histogram_quantile(0.95,
  sum by (le) (rate(smokeping_response_duration_seconds_bucket[$__rate_interval])))
- histogram_quantile(0.5,
  sum by (le) (rate(smokeping_response_duration_seconds_bucket[$__rate_interval])))
) * 1000

Set unit to milliseconds (ms) in right sidebar → Standard options → Unit.

Packet Loss →

promql

100 * (1 - (
  rate(smokeping_responses_total[$__rate_interval])
  /
  rate(smokeping_requests_total[$__rate_interval])
))

[mirtobusico]

Thanks a lot

I’ll try to verify and repeat every step and report here.

BTW

I don’t know how to find the dashboard version (when installed) on the site it says

And the query for “response time (by bucket)” seems to already use the new names

sum by (le) (
  rate(smokeping_response_duration_seconds_bucket{host="${hostname:raw}",ip="${host:raw}",job="$job"} [$__interval])
) * 2

The prometheus-smokeping-prober version is 0.11.0-2

Now I’ll verify every step.

[mirtobusico]

Hi @infofcc3 here what I’ve done.

My starting point is Smokeping that, for google dns 8.8.4.4, gives an average around 15.9 ms

STEP 1

Original systemd service from Manjaro package in /usr/lib/systemd/system/prometheus-smokeping-prober.service

[Unit]

Description=Prometheus style smokeping

After=network-online.target

Wants=network-online.target




[Service]

ExecStart=/usr/bin/prometheus-smokeping-prober --config.file="/etc/prometheus/smokeping_prober.yml"

ExecReload=/bin/kill -HUP $MAINPID

DynamicUser=true




NoNewPrivileges=true

ProtectSystem=full

ProtectKernelModules=true

ProtectKernelTunables=true

PrivateTmp=true

LockPersonality=true

ProtectHostname=true

ProtectHome=true

ProtectControlGroups=true

ProtectKernelLogs=true

PrivateDevices=true

RestrictRealtime=true

CapabilityBoundingSet=

MemoryDenyWriteExecute=true

CapabilityBoundingSet=CAP_NET_RAW

AmbientCapabilities=CAP_NET_RAW




[Install]

WantedBy=multi-user.target

Not sure if –privileged is needed

My configuration file in /etc/prometheus/smokeping_prober.yml contains

---

targets:

- hosts:

- cloudflare-dns

- google-dns

- nexxt-int

- p9-principale-ext

- mostro

- kit-studio

- bianco500

interval: 1s # Duration, Default 1s.

network: ip # One of ip, ip4, ip6. Default: ip (automatic IPv4/IPv6)

protocol: icmp # One of icmp, udp. Default: icmp (Requires privileged operation)

size: 56 # Packet data size in bytes. Default 56 (Range: 24 - 65535)

Originally there were another two lines in this file that I had to remove in order to access the prober from the 192.168.1.0 lan

source: 127.0.1.1 # Souce IP address to use. Default: None (automatic selection)

tos: 0x00 # Packet ToS field in decimal, hexadecimal or binary format. Default: 0 (Range: 0-255)

To use only names for the targets without relaying on DNS I have in /etc/hosts (the VM I’m using have “netmon-vm” hostname)

# Standard host addresses

127.0.0.1 localhost

::1 localhost ip6-localhost ip6-loopback

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

# This host address

127.0.1.1 netmon-vm




1.1.1.1 cloudflare-dns

8.8.4.4 google-dns

192.168.1.254 nexxt-int

192.168.1.50 p9-principale-ext

192.168.1.10 mostro

192.168.1.40 kit-studio

192.168.68.30 bianco500

With these settings I can access the probe metrics

The curl command for the “google-dns” host gives

[mirto@netmon-vm ~]$ curl -s http://netmon-vm:9374/metrics | grep google-dns
smokeping_requests_total{host="google-dns",ip="8.8.4.4",source="",tos="0"} 325924
smokeping_response_duplicates_total{host="google-dns",ip="8.8.4.4",source="",tos="0"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="5e-05"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0001"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0002"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0004"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0008"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0016"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0032"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0064"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0128"} 0
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0256"} 319266
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.0512"} 320651
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.1024"} 321685
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.2048"} 324263
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.4096"} 324315
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="0.8192"} 324377
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="1.6384"} 324379
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="3.2768"} 324379
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="6.5536"} 324379
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="13.1072"} 324379
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="26.2144"} 324379
smokeping_response_duration_seconds_bucket{host="google-dns",ip="8.8.4.4",source="",tos="0",le="+Inf"} 324379
smokeping_response_duration_seconds_sum{host="google-dns",ip="8.8.4.4",source="",tos="0"} 5782.676779012143
smokeping_response_duration_seconds_count{host="google-dns",ip="8.8.4.4",source="",tos="0"} 324379
smokeping_response_ttl{host="google-dns",ip="8.8.4.4",source="",tos="0"} 116
smokeping_send_errors_total{host="google-dns",ip="8.8.4.4",source="",tos="0"} 0
[mirto@netmon-vm ~]$

So it seems that it is gathering the metrics

STEP 2

The prometheus configuration file is /etc/prometheus/prometheus.yml to which I added the smokeping job

# my global config

global:

scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).




# Alertmanager configuration

alerting:

alertmanagers:

- static_configs:

- targets:

# - alertmanager:9093




# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

# - "first_rules.yml"

# - "second_rules.yml"




# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: "prometheus"




# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.




static_configs:

- targets: ["localhost:9090"]

# The label name is added as a label `label_name=<label_value>` to any timeseries scraped from this config.

labels:

app: "prometheus"



- job_name: smokeping

static_configs:

- targets: ["localhost:9374"]

The query url

http://netmon-vm:9090/query?g0.expr=smokeping_response_duration_seconds_bucket+%7Bhost%3D%22google-dns%22%7D&g0.show_tree=0&g0.tab=table&g0.range_input=1h&g0.res_type=auto&g0.res_density=medium&g0.display_mode=lines&g0.show_exemplars=0

shows

That seems to have gathered the correct data

STEP 3

To start form a simple case, I’ll try to replicate the “Response time by bucket” panel

From Drilldown > Metrics > All metrics I select the proposed “smokeping_response_duration_seconds_bucket” example panel And added it to an empty dashboard

The default panel generated is this

Setting the Unit to “Seconds” and adding a filter for “google-dns” I obtain

According to the original dashboard I changed the query options

Obtaining something similar to the original dashboard panel

Remarks

In the original panel the query is (I don’t understand why the formula is multiplied by 2)

sum by (le) (

rate(smokeping_response_duration_seconds_bucket{host="${hostname:raw}",ip="${host:raw}",job="$job"} [$__interval])

) * 2

In my panel the query is

sum by(le) (rate(smokeping_response_duration_seconds_bucket{host="google-dns"}[$__rate_interval]))

What is the difference between $__interval and $__rate_interval ?

In the original panel the “values” correspond to the scale on the left but gives only one value

In my panel seems that the values follow the scale on the left but give an interval bucket value (is this correct?)

What is the difference between Value and Bucket?

And this is for a single host.

But if I want to monitor multiple hosts as in smokeping I don’t know how to do this.

In my understanding the heatmap cannot show multiple time series. Is this correct?

Any hint on how to proceed?

[infofcc3]

Your setup is correct and data is flowing perfectly.

$__interval vs $__rate_interval
Use $__rate_interval → it’s the correct one for rate() functions. It automatically adjusts to be at least 4x your scrape interval, preventing gaps. $__interval can be too small and cause empty graphs.

* 2 in original query
It compensates for Min interval = 5m set in Query options, which halves the data points. Your query without * 2 is correct. To get a smoother heatmap set in Query options: Min interval 1m and Max data points 500.

Value vs Bucket in heatmap tooltip
Your panel is correct. “Bucket” shows the count of responses that fell within that latency range → which is exactly what you want for a heatmap.

4. Multiple hosts in one heatmap
You’re right → a single heatmap cannot show multiple hosts. So create one row per host, or use a variable $hostname to switch between hosts via dropdown.

To add a dashboard variable in Grafana 13: Dashboard Settings → Variables → Add variable → set type to Query, query:

label_values(smokeping_response_duration_seconds_bucket, host)

Then use {host="$hostname"} in your panel queries. This creates a dropdown showing all your hosts.

Prometheus template variables | Grafana documentation

[infofcc3]

Thanks @mirtobusico

[mirtobusico]

Hi @infofcc3 thanks to your suggestions I was able to create a dashmoard with 3 panels:

• all hosts response time - seconds (like smokeping without smoke)
• one host response time - bucket (like smoke in smokeping)
• one host response time seconds (like the first panel but only for one host

Again thanks for your time