How to configure data retention/expiry

Hi all,

I cannot make sense of the InfluxDB documentation it seems. How can I configure InfluxDB to e.g. simply delete everything older than a year. That would perfectly satisfy my needs for a first try. No downsampling or anything fancy needed, IMHO.

Thanks,
Patrick

To create a 1 year retention policy on an existing database you can use
CREATE RETENTION POLICY "one_year" ON "databasename" DURATION 52w REPLICATION 1 DEFAULT
Actually that is 52 weeks. Then when you write to a measurement tell it to use that retention policy. A database can have multiple retention policies.

This page is a good reference

[Edit] In fact, since I included DEFAULT on the policy that will be the default one to use for that database, so you would not need to specify it explicitly when writing/reading.
Note that any existing data will not be affected by this as it will have (presumably) been written with the infinite retention period, so if you want that to expire you will have to manually delete it or move it to the new policy, which you can do using something like
SELECT * INTO "the_db"."one_year"."the_measurement" FROM "the_db"."autogen"."the_measurement" GROUP BY *
then after you have checked that it is there ok
DROP RETENTION POLICY "autogen" ON "home"
or you can just drop the individual measurements in autogen if appropriate.

2 Likes

Still having a hard time wrapping my head around what all this means but that’s more than enough to set me on the right path. Thanks.

I hoped there was a config file directive … you know, ops people, Ansible, … and stuff. I hate semantics in databases. :wink:

Patrick

I searched for a way to delete all the data with the wrong retention policy and I found this: https://docs.influxdata.com/influxdb/v1.7/query_language/database_management/

DELETE WHERE time < '2016-01-01'

So now I have to make up my mind if I do it the Influx way or just set up a cron job.

Kind regards,
Patrick

There are ansible modules for at least most of this, in fact coincidentally I have just started looking at them. https://docs.ansible.com/ansible/latest/modules/list_of_database_modules.html#sts=Influxdb

1 Like

Implemented the policy in Influx. I assumed that in

DROP RETENTION POLICY "autogen" ON "home"

“home” is supposed to be my database name? I used that and it seemed to work.

See, when one is completely new to this - I was not even aware I was dealing with something that could manage multiple “databases”. I never created one consciously.
All I did was configure a (what I thought) Graphite compatible listener in a couple lines via google&paste:

[[graphite]]
  enabled = true
  database = "graphite"
  retention-policy = ""
  bind-address = ":2003"
  protocol = "tcp"
  consistency-level = "one"

  separator = "."

  templates = [
    "servers.* .hostname.resource.measurement*",
  ]

So I was looking for the magic incantation I could put into the “retention-policy” parameter there.

Thanks again, more interesting stuff to research.
Patrick

Sorry, missed changing the database name in the drop retention example.

I don’t use Graphite so have had to setup the db and so on manually. That is one of the problems using a higher level system that does things magically for you, when you need to get your hands dirty it is not always clear where in the garden you need to dig.

I don’t either. I use Influx as a replacement for Graphite because I have a system that feeds into Graphite and is not configurable in that regard. And the Graphite stack was like … uh … :wink:

This could be interesting to you @pmhausen

We are replacing InfluxDB v.1.x at the moment as it is end-of-life (and because we ran into performance issues that are kind of caused by how we ship metrics…).

The replacement is VictoriaMetrics and it also supports Graphite plaintext protocol
The query language in Grafana does change though, so if that is an issue, it might not be for you. The good news is that the query language changes to PromQL which is much nicer IMO :slight_smile:

Forgot… It is very easy to configure :slight_smile:

@b0b No FreeBSD package, mostly Docker. Thanks, but won’t fly.
Influxdb was as simple as pkg install influxdb

Kind regards,
Patrick