How to collect metrics when user is out in the field

Grafana Prometheus
1
  • What Grafana version and what operating system are you using?

Grafana 11 on Fedora 40.

  • What are you trying to achieve?

Trying to understand how metrics can be collected when a user doesnt have a dedicated internet connection back to Prometheus/Grafana environment.

  • How are you trying to achieve it?

By asking the question in this forum.

Hi,

I have a scenario where I have users that have their lap tops out in the field. The users will go to areas where there isnt a dedicated internet connection. However I would still like to collect metrics and then once the user(s) have re-establish an internet connection, then collected metrics are then available for visualisation in Grafana. I dont know if this can be done when prometheus is part of the solution (and it’s not a requirement for me) so I’m hoping someone might point me in a direction (fourm, link, book, etc) that may shed some light on solving the problem.

I presume this isnt an unusual use case but I’ve had difficulty finding out how other people are solving this problem. The only thing I have read is the usage of VictoriaMetrics due to a push architecture that it can use. However I’m not sure this is the approach that should be taken. Thoughts?

2

Hi @hoovermeows

In Node-RED (which can be installed and run pretty much anywhere, including the users’ laptops), you can temporarily store the data when the connection to Promethus
or (as I prefer) InfluxDB is not available by using some of these approaches:

1. Use a Local File for Temporary Storage

  • Use the file out node (example here) to write incoming data to a local file (e.g., JSON or CSV).
  • When InfluxDB is available again, another flow would read the stored data and send it to InfluxDB.

2. Use a Local SQLite Database

  • Store incoming data in a small SQLite database.
  • A background job moves the stored data to InfluxDB when available.

3. Use a Queue with a Delay and Retry Mechanism

  • Use the [node-red-contrib-simple-message-queue](node-red-contrib-simple-message-queue (node) - Node-RED) node.
  • If the InfluxDB write fails, queue the data and retry until successful.
  • Not sure how many hours/days/weeks of data you might store, or if this approach has size limits.
2 Likes
3

Configure vmagent at every laptop in the field, for collecting metrics from local apps running on the laptop, and pushing the collected metrics to the centralized storage such as VictoriaMetrics or Mimir. Vmagent buffers the collected samples until the connection to the remote storage is re-established. I’d recommend using VictoriaMetrics, since it supports ingestion of samples with old timestamps and with out of order timestamps (aka backfilling) - see these docs.

1 Like
4

Thank you Valyala.

I think I need to take VictoriaMetrics for a spin.

1 Like