Retention Policy - Dashboard Prometheus

Hi there.

I’m a noob to Grafana and I’ve been tasked to edit our current platform and Extend the retention period.
What I’ve managed to figure out is that the data base is prometheus.
It’s using the “Graph (Old)” visualization.

Now if we change the retention period in the top right it only gives us data for about 15 days.
How do i extend this to 6 month or a year?

Thanks

Hi

If using prometheus you should locate the prometheus file, where the retention period can be set. Mine is located here: /etc/default/prometheus (no file extension)

As shown below. I have set my retention time to 2years.
I believe you could set it to 180d (for aprox 6months)
But also 6m (im in doubt if it should be 6M) for 6 months.

Remeber you need some extra disk space.

cat /etc/default/prometheus

Set the command-line arguments to pass to the server.

ARGS="–storage.tsdb.retention.time=2y"

/ Carsten

Hey Carsten

I didn’t have this file, can’t find it.
I’m assuming this is on the grafana host not the machine I’m pulling data from.
That being said I don’t see the file on either machine.

I’ve found /etc/systemd/system.conf
But don’t see any arguments there that could help with the retention.

Any help is appreciated.

Thanks

Hmm

If i go to directory /etc/default/ as shown below. i got the file. Try the below:
ls -c -ltd – prometheus ← will list all files with prometheus in filename
There the top file is the one you need.

nano prometheus
will let you edit it, and change retention. You might have to sudo to be able to edit it.

carsten@carsten-pc:/etc/default$ ls -c -ltd – prometheus
-rw-r–r-- 1 root root 3503 Sep 14 10:57 prometheus
-rw-r–r-- 1 root root 3471 Sep 14 10:46 prometheus.org
-rw-r–r-- 1 root root 1200 Sep 14 10:40 prometheus-pushgateway
-rw-r–r-- 1 root root 3197 Sep 14 10:40 prometheus-alertmanager
-rw-r–r-- 1 root root 8114 Sep 14 10:40 prometheus-node-exporter

Let me know how it goes.

/ Carsten

I only have the first option.

There was no file in there so i created “prometheus” in there.
i’ve edit to contain
-storage.tsdb.retention.time=2y

Is there something in there I need to do to enable it?
Also how do i specify what dashboard in grafana to apply this to, if I don’t want to apply it to all of them?

Hmm

You should have the file and directory there if prometheus is installed on that machine. Have you tried with a search for the file. Dont know what system you are running on. But could try a find in ubuntu. And you are sure its prometheus you are running and not influxDB ?

image

So our license server is what we pulling the data off.
On there it looks like prometheus-2.28.1 was installed and the client was setup.
there have been a few prometheus service restarts.
we have a /etc/prometheus folder with /console, libraries, consoles, prometheus.yml
But there is nothing in the /etc/default folder

Would it maybe be the yaml file or do i add the prometheus file in here somewhere?

Hi,

maybe this will help…

Below the file of /usr/lib/systemd/system/prometheus.service

[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/

[Service]
Restart=always
User=prometheus
EnvironmentFile=/etc/default/prometheus
ExecStart=/usr/bin/prometheus $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no
LimitNOFILE=8192

[Install]
WantedBy=multi-user.target

You can see the EnvironmentFile that pointed to /etc/default/prometheus
So, to config the retention time, just edit the /etc/default/prometheus as follow:

ARGS="--storage.tsdb.retention.time=2y"

To make it permanent, just run this:

systemctl daemon-reload
systemctl restart prometheus

Regards,
Fadjar Tandabawana

Thanks @fadjar340
That was helpful.
I’ve found this on the target server.
As Prometheus.server doesn’t exist on the Grafana server.
I’m assuming i need to create the EnvironmentFile and put in the recommended default location?
Is there somewhere i can find an example of the Environment file?

[root@server01 systemd]# cat /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target 

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.listen-address=0.0.0.0:9090 \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Hi @memorist23 ,

Therese several options in the prometheus.service file, in my snippet there’s using EnvironmentFile, but in your snippet, no need to have EnvironmentFile, just add the retention config

In my config, I use this as EnvironmentFile

In your config, just use this

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target 

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.listen-address=0.0.0.0:9090 \
--web.console.libraries=/etc/prometheus/console_libraries \
--storage.tsdb.retention.time=2y

[Install]
WantedBy=multi-user.target

[/quote]

Then your retention time goes to 2y

Regards,
Fadjar Tandabawana

1 Like

Hi all
So I’ve run in to a bump in the road. The retention period no longer works and the system has defaulted back to 15 days.
I’ve done more digging on the server I’ve found and installer for prometheus-2.28.1.linux-amd64.tar.gz
But still can’t find an Environment file or why this is resetting.
Could I create a new one in a new location without breaking the rest of my setup or is there still something I’m missing?
I could do a locate prometheus and upload the directories I have if that helps?