Grouping not allowed for count_over_time aggregation

[atjbramley]

Hi,

I’m trying to use Loki to generate visualisations from a service’s event logs. I have written a script that authorises and parses the JSON API of the service, and dumps the results into a file on disk. Using jq I can either choose to analyse these logs in .csv or .json format - and I think that .csv is going to be easier.

Here’s an example of the logs:

event,1108,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:48.175Z,10,xxx.xxx.172.92
event,1107,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:46.856Z,10,xxx.xxx.172.92
event,1100,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:40.6766667Z,10,xxx.xxx.172.92
event,1115,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:38.63Z,10,xxx.xxx.172.92
event,1107,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:36.167Z,10,xxx.xxx.172.92
event,1108,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:33.908Z,10,xxx.xxx.172.92
event,1107,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:32.908Z,10,xxx.xxx.172.92

I’ve scraped the logs into Loki using promtail, and added labels using the following pattern

`<event>,<eventType>,<itemId>,<collectionId>,<groupId>,<policyId>,<memberId>,<actingUserId>,<installationId>,<date>,<device>,<ipAddress>`

I’d now like to create a visualisation of the various event types - say a timeseries chart of eventType=1000, a graph of the most accessed itemIds etc.

I’m a bit stuck at this point - a query that I thought should work:

count_over_time({filename="/var/xxxlogs/eventLogs/event_logs.csv"} 
  | pattern `<event>,<eventType>,<itemId>,<collectionId>,<groupId>,<policyId>,<memberId>,<actingUserId>,<installationId>,<date>,<device>,<ipAddress>` [1m]) by (eventType)

gives a grouping not allowed for count_over_time aggregation error.

I’d be most grateful if anybody could give me some ideas or suggestions as to how to produce visualisations based on this data. At the moment my knowledge is limiting me to logs!

[tonyswumac]

The error is correct, count_over_time does not have a grouping part. You can include your query with a sum function, like this:

sum by (eventType) (
  count_over_time(
    {filename="/var/xxxlogs/eventLogs/event_logs.csv"} 
      | pattern `<event>,<eventType>,<itemId>,<collectionId>,<groupId>,<policyId>,<memberId>,<actingUserId>,<installationId>,<date>,<device>,<ipAddress>`
      | __error__=""
    [1m]
  )
)

[atjbramley]

Hi Tony,

Thanks - I’ve managed to come up with working queries now.

I’ve now run into the issue that because I’m using a script to import, the log entries are being timestamped with the time they were scraped rather than the time present in the log line.

I’m attempting to battle this with regex and the promtail config file - I’ll have a play over the weekend and open a new thread if I can’t figure it out.

Thanks once again.