How to count over time all logs over last 10 business days during business hours?

Grafana Loki
1

I want to count_over_time logs matching an expression over the last 10 business days only between 10:00 and 17:00 hours. At best, would be if the count would split for every 1 hour.

I am executing the roughly following, not real code::

for daysago in range(20):
   query = """
                   sum by (job) (count_over_time(
                        {source="%s", job=~"%s"}
                       | job !~ ".*TEST.*"
                        | job !~ ".*\\..*"
                        |= ": [GC ("
                        [1h]
                 ))
   """ % (source, job)
   result = query_range(
      query=query,
      since=datetime.timedelta(hours=7),
      end=datetime.datetime.combine( datetime.date.today() - datetime.timedelta(days=daysago), datetime.time(17)),
  )

As I understand, “end” is the ending time, i.e. 17:00 on a particular day and “since” should be set to 7 hours.

The api requests look like the following:

/loki/api/v1/query_range?query=%0A++++++++++++sum+by+%28job%29+%28count_over_time%28%0A++++++++++++++++%7Bsource%3D%22test%22%2C+job%3D~%22QM2.%2A%22%7D%0A++++++++++++++++%7C+job+%21~+%22.%2ATEST.%2A%22%0A++++++++++++++++%7C+job+%21~+%22.%2A%5C%5C..%2A%22%0A++++++++++++++++%7C%3D+%22%3A+%5BGC+%28%22%0A++++++++++++++++%5B7h%5D%0A++++++++++++%29%29%0A++++++++++++&end=1712869200000000000&since=25200s

However, the resulting value is extremely big. I do not understand where is it coming from, however from my experiments it looks to be related to “step” of the API, as-if loki has calculated [1h] range over multiple ranges and summed up. Setting step to 1 hour, results in no results - the api response is empty array.

How do I get count_over_time split for every hour over specific range of dates? Thanks.

2

Couple of changes I’d try:

  1. Change your API endpoint to instance query (/query) instead of query_range.
  2. Change the interval in your query to [7h] instead of [1h] (might even make that a variable).
  3. Because you are now using instance query, you’d want to use the time at the end of the day so [7h] includes the past 7 hours according to query time.