How to Use a Prometheus Label as a Timestamp for Grafana

[mazzybig]

Hi everyone!

I have my own metrics that looks like:

my_metric{id="object1",date="2021-10-11T22:55:54Z" } 1
my_metric{id="object2",date="2021-10-11T22:20:00Z" } 4

I want to make a graph with label ‘date’ by X-axis and metric value by Y-axis. There should be value points for different IDs.
In other words, I want to change the default timeline to my new one.

Are there some ideas how to do it or should I change my metrics?

Thank you

[mattabrams]

Are you using Prometheus @mazzybig? Please specify datasource.

One gotcha I can see here is that Prometheus labels are strings, so your timestamp, date, would need to change its data type to work as a time field

[mazzybig]

Yes, I’m using Prometheus. I create my own metrics using bash-script that saves them to prom-file for node-exporter.

your timestamp, date, would need to change its data type to work as a time field

I couldn’t get this point. Can you please explain what should I do?

As I said earlier, I can create my own metrics in any format. So now I’m looking for a better approach.

[mattabrams]

so you are using the textfile collector that ships with node-exporter? Whenever Prometheus scrapes that file, it will add a timestamp to the scraped metrics. What is the pre-existing timestamp and how / where does it come from?

[mazzybig]

@mattabrams

so you are using the textfile collector that ships with node-exporter?

Yes, exactly

@mattabrams

What is the pre-existing timestamp and how / where does it come from?

I get them from third-party API. These are the dates and times of some kind of events. Moreover, several events with different times may be written in one text file.
So, I need to visualize these events on a timeline.

[mattabrams]

@mazzybig yes, if I understand correctly, then you’ve run up against a known limitation with Prometheus. It just wasn’t designed to ingest time-series data where the timestamp is a label. See this issue comment and the related blog post for workarounds:

In a table, formatting a timestamp field (pulled from Prometheus metric label value) as any form of "Date" unit yields NaN

[mazzybig]

Thank you!
It makes sense

[kalashkimi]

Hey have you got anyway to do that? (Custom time stamp on x axis)