Metrics are deleted if device is offline for more than 15m

Grafana Cloud
, ,
1

I have a remote device which has Grafana Agent installed and connected to Grafana Cloud which receives some Prometheus metrics. The device might be without internet access for an extended period of time (e.g. 2-3 hours) before connecting again. In this case I see that the metrics are being deleted and when there is internet again these 2 hours of data are not filled up.

I have tried updating the min_wal_time to 3h which is what I understood that would help for that here but I am still losing this data. I am using the default wal_directory: /tmp/grafana-agent-wal. Here is a simplified version of my Grafana Agent config:

server:
  log_level: info

metrics:
  wal_directory: /tmp/grafana-agent-wal
  wal_cleanup_period: 2h
  http_idle_conn_timeout: 1h
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: <psw>
        username: <username>
      url: <url>
    min_wal_time: 3h
    max_wal_time: 6h
    wal_truncate_frequency: 2h
    write_stale_on_shutdown: true
    scrape_configs:
    - job_name: job-prometheus-metrics
      static_configs:
      - targets:
        - localhost:8000
        labels:
          instance: instance-name
      scrape_interval: 5s
  global:
    scrape_interval: 30s

logs:
  configs:
  - name: default
    clients:
    - basic_auth:
        password: <psw>
        username: <username>
      url: <url>
    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: custom-job
      static_configs:
      - targets:
        - localhost
        labels:
          instance: instance_name
          job: custom-job
          __path__: /file_location.smt

integrations:
  node_exporter:
    enabled: true
    instance: device
  prometheus_remote_write:
  - basic_auth:
      password: <psw>
      username: <username>
    url: <url>

How can I keep this data locally till they are uploaded to Grafana Cloud, so I will not lose them?

2

Hi @ioka !

If you expect to have such long timeframes where data cannot be forwarded, I think it would be best to store the data locally and then connect that data source to Grafana Cloud for queries, visualizations, and alerts.

The data propagation model used by the Grafana Agent (recently renamed to Alloy) and other types of Prometheus remote_write functionality is intended for real time data scraping and forwarding.

Alternatively, you can try out the “agentless” Metrics Endpoint integration for Grafana Cloud however consistent public connectivity will still be necessary for the best results.

3

maybe have something local for those time periods and sync to Cloud later on.
but that defeats the whole purpose of Cloud solution