Creating a custom timestamp conversation with Promtail Pipelines

Grafana Loki
,
1

Hi!

I’m trying to use Pipelines to define a timestamp from logs that are presented in a .csv file.

I have some log examples as shown:

event,1107,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-15T18:17:13.3233333Z,10,111.222.333.444
event,1100,1412cf19-7234-4456-87e8-af4e012b8ff7,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-15T18:10:40.44Z,10,1.2.3.4
event,1300,,e9c76946-b644-4451-8695-af4e012b562d,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-15T18:09:51.1266667Z,10,111.222.333.444
event,1000,,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-15T18:09:32.2446508Z,10,111.222.333.444
event,1108,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:48.175Z,10,111.222.333.444
event,1107,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:46.856Z,10,111.222.333.444
event,1100,0deba616-9f81-488f-81c1-af4a01040347,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:46:40.6766667Z,10,111.222.333.444
event,1115,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:38.63Z,10,111.222.333.444
event,1107,f812baad-6e31-4fac-8c8a-af4a0103a7f4,,,,,83cd55a9-95bf-4eb5-a221-af4900c54bf7,,2022-11-11T15:45:36.167Z,10,111.222.333.444

I have written the following regex to extract labels:

  pipeline_stages:
  # https://grafana.com/docs/loki/latest/clients/promtail/stages/regex/
    - regex:
        expression: '^(?P<event>\w{1,5}),(?P<eventType>\d\d\d\d),(?P<itemId>[^,]*),(?P<collectionId>[^,]*),(?P<groupId>[^,]*),(?P<policyId>[^,]*),(?P<memberId>[^,]*),(?P<actingUserId>[^,]*),(?P<installationId>[^,]*),(?P<date>[^,]*),(?P<device>\d\d),(?P<ipAddress>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$'
    - labels:
        eventType:
        itemId:
        collectionId:
        groupId:
        memberId:
        actingUserId:
        date:
        ipAddress:

This seems to work OK, but because the .csv is read all at once, the timestamp is created for the time that promtail scraped the file, not the time recorded in the log entry:

Screenshot 2022-12-20 at 16.35.45
Screenshot 2022-12-20 at 16.35.452046×882 41.6 KB

I believe that the below config should take the ‘date’ label, and log it as a time stamp:

    - timestamp:
        source: date
        format: 2006-01-02T15:04:05.000000Z

However, when doing this the timestamp does not change.

Would greatly appreciate any pointers as to where I’m going wrong!

2

Your configuration look right, if I had to guess I’d say it’s the inconsistent number of digits in the fraction of second field.

Promtail time stamp uses Go’s time parse function. You can follow the function call here time package - time - Go Packages and write a little Go function and do some spot test yourself.

3

Thank you Tony,

I’ll probably write a little script to drop the milliseconds as the simplest way around this.

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.