Filtering todays logs from midnight to now UTC in Loki

Grafana Loki
,
1

I need to filter the logs only for today. The logs have timestamps in UTC. I can filter logs with a hard-coded date string as follows.

{job="myjob"} |~ `2025-04-02.*`

Given, I need to generate that string in the filter expression dynamically, I attempted the following

 {job="myjob"} |~ `{{.timestamp | toDate \"02-01-2006\"}}`

However, that didn’t work. Any suggestions on how to generate a string dynamically to filter the logs?

2

Apart from the above string matching approach happy to explore other avenues. All I need is to get logs from midnight to now.
PS: I do not have the dashboard support. Just Loki expressions emitting metrics to Prometheus.

3
  1. You should always make sure the timestamp of your logs are accurate when ingesting into Loki.
  2. Assuming timestamp of your logs are accurate, you should specify time as part of Loki API and let Loki do the work for you. See Loki HTTP API | Grafana Loki documentation, specifically start and end.
4

Thanks for your reply Tony.
The issue here is that I do not have the luxury of invoking the Loki via API.
I have to use Loki recording rules that get triggered, say every 5 minutes. They need to consider the logs only from midnight to now.
Loki expressions in recording rules can do this for hard-coded strings. The issue here is that it cannot do it for dynamically constructed strings.

5

Are you actually trying to setup a recording rule, or are you just looking to run a query?

If you are actually looking to setup a recording rule, then I would recommend you to reconsider because running recording rule frequently against a long period of time isn’t the most efficient and might cause you problems. But if this is what you want, because recording rules are supposed to metrics query they usually have an interval at the end (for example, let’s say sum(count_over_time({app="foo", env="production"} |= "error" [5m])) by (app), you can simply change the interval from 5m to say 12h or something like that. But you can’t control this finely.

If you are just looking to run a query… you should really try and use the API. If the API is blocked from you, you should find out why it’s blocked because it makes no sense. Not to mention that ruler won’t actually give you logs, it gives you metrics.

6

Thanks again Tony.
I think what you’re describing is the option later I have resorted to.
I have Loki recording rules running in short intervals, e.g., 1m, emitting metrics to Prometheus. From Prometheus I have setup its own recording rules relying on the metrics emitted by Loki. But then in Prometheus too only allows fixed/static values for its range vector selector, e.g., [24h] Querying basics | Prometheus
For a moving window of 24 hours in the past this will work. But not sure how I can calculate the count/increase from midnight to now.

7

To give you the full picture, I need to read these metrics (midnight to now) from a client application via the Grafana HTTP API. The client application is not under my control. So my intention is to perform all these calculations using recording rules in Prometheus and store as a time series so that any time the client reads, it gets the value, e.g., the count of requests from midnight to now is N. If we have set the interval for 5 mins then the series have captured these values to the closest 5 mins interval. Then the client application can call the Prometheus API (proxy via Grafana HTTP API) using the query name as the parameter.

9

Use now/d to now always for today’s logs

image
image1317×600 94.8 KB

Note: You can’t use dynamic date like {{.timestamp}} inside Loki’s |~ regex because LogQL doesn’t support Go templating. This templating only works for dashboard variables or panel expressions, not inside log queries.

10