How to combine multiple timeseries into one

Grafana Prometheus
1

I have these metrics in prometheus tsdb(output from curling the exporter)

# HELP custom_test_metric A test metric with a past timestamp
# TYPE custom_test_metric gauge
custom_test_metric{step="step1"} 42.0 1741899227000
custom_test_metric{step="step2"} 45.0 1741899230000
custom_test_metric{step="step3"} 48.0 1741899827000

As you can see I added timestamps into each metrics. The reason I am doing this is: my data generation is not always a fix interval. For a 15 second scrape interval, I may generate 1, 3, or 10 values, each is guarantee to have a unique timestamp. So in my exporter/collector, I just keep on using label stepN for the Nth value.

I’d like to show them in one timeseries in grafana, instead of n, since they are actually the same one metric (with no labels). The timestamp is guaranteed to be unique for each of these metric. Something like without the step label, like this:

custom_test_metric{} 42.0 1741899227000
custom_test_metric{} 45.0 1741899230000
custom_test_metric{} 48.0 1741899827000

Is this doable? If it is, then my prometheus scrape interval does not need to be related/dependent on the how fast the data is collected. That will be a big win since one scrape job may collect different metric at dramatically different intervals.

Background: for each prometheus scrape interval, I may have more than one value. So in the collector, I added step label for each metric. But they are really the same metric (with same set label value). I can’t just give a small scrape interval since the data generation is not at fixed interval. So I create the above step1/step2/step3/step… label with timestamp and hopefully grafana can combine them and show them as one timeseries, using something like transform. Is this doable?

I tried prometheus re-write/relabel rules but could not make it work.

Thank you!

2

Hi, can you aggregate those series into one with aggregations like sum / max/ avg? Like
sum(custom_test_metric). What it will do is it will take every sample at the same timestamp and sum it into one resulting series. I’m not sure about the interval though - Grafana must pass interval to the datasource (prometheus in your case) meaning how often a point should be marked (by default it’s 15s). You could try to make Grafana interpolate those by setting interval to 1s but I’m not quite sure the consequences.

3

Thanks for replying. It doesn’t seem to work for me.
To be more specific, here is an example. This is part of my collector, which is scraped/called by the prometheus server once per minute:

        # Simulating past timestamps (40 seconds ago)
        timestamp_ms = int(time.time() - 40)

        random_int = random.randint(1, 100)
        gauge.add_metric(["test","step1"], random_int + 40, timestamp=timestamp_ms)
        gauge.add_metric(["test","step2"], random_int + 45, timestamp=timestamp_ms+20)
        gauge.add_metric(["test","step3"], random_int + 50, timestamp=timestamp_ms+40)
        yield gauge

So basically the collector sends back to prometheus 3 metrics each have 5 in value difference and 20s in timestamp difference. These are correctly put into prometheus tsdb as you can see from the promql:

custom_test_metric{instance="cai-cloud", job="testers", label="test", step="step1"}
99 @1741987564
56 @1741987624
42 @1741987684
104 @1741987744
 
custom_test_metric{instance="cai-cloud", job="testers", label="test", step="step2"}
104 @1741987584
61 @1741987644
47 @1741987704
109 @1741987764
 
custom_test_metric{instance="cai-cloud", job="testers", label="test", step="step3"}
76 @1741987544
109 @1741987604
66 @1741987664
52 @1741987724
114 @1741987784

The problem, however, seems to be in the way I display them in grafana. Here is the query:
custom_test_metric{instance="$Cloud"}. If I set the min interval to 60s, 3 identical (shift by value 5) lines are shown. I was hoping the 3 lines should be horizontally shifted by 20s. How do I ask grafana to honor the timestamp in prometheus tsdb?

If I set the min interval to 20s, 3 timeseries lines are shown which is correct per-series-wise, but there are extra data points (duplicated) shown as well. If I do sum(), then these duplicated data points are used to get the sum of 3 values. The problem seems to be grafana adds timestamps for each metric for each the set min interval. Again, how to ask grafana to honor the timestamp in prometheus tsdb and not adding extra data points?

I guess only when this problem is solved then maybe we can try sum().

My apology if I am missing some very fundamental knowledge of grafana. And thanks for your help!