Prometheus metrics are removed if device has no internet for more than 30m

Grafana Configuration
,
1

Hello,

I have a remote device running grafana agent and sending prometheus metrics to a grafana cloud instance. The device might face internet disconnections for around 2 hours and in that case I would like to keep the data and send them when the internet connection is restored.

I have tried updating the min_wal_time to 3h which is what I understood that would help for that here but I am still losing this data. I am using the default wal_directory: /tmp/grafana-agent-wal . Here is a simplified version of my Grafana Agent metrics config:


metrics:
  wal_directory: /tmp/grafana-agent-wal
  wal_cleanup_period: 2h
  http_idle_conn_timeout: 1h
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: <psw>
        username: <username>
      url: <url>
    min_wal_time: 3h
    max_wal_time: 6h
    wal_truncate_frequency: 2h
    write_stale_on_shutdown: true
    scrape_configs:
    - job_name: job-prometheus-metrics
      static_configs:
      - targets:
        - localhost:8000
        labels:
          instance: instance-name
      scrape_interval: 5s
  global:
    scrape_interval: 30s