Loki unable to parse timestamp sent? (what timestamp format does Loki expect?)

Grafana Explore
,
1

Hi there, I am sending logs to loki using vector.dev as the log shipper.
I am having a hard time getting loki to accept/parse the time field sent.

I parse my applications logs own timestamp in vector before shipping them to loki.
I am formatting my timetamps to rfc3339 before sending them off to loki, but loki seems to just set the timestamp to time received of the logs, not the parsed timestamp from my logs.
I have set the loki sink in vector to allow unordered-writes, and I’ve set the chnk age in loki to 72 hours.

Loki version: 2.6.1

If I run a query using logcli agains my single loki instance, I can see the timestamps do not match what I sent to loki (expanded json for readability)
LogCli output below:

2022-10-20T22:59:13+00:00 {} 
{
  "environment": "dev",
  "file": "/var/log/haproxy.log",
  "message": " <fake-ip-goes-here>:1741 [20/Oct/2022:22:59:12.808] Fake message goes here",
  "owner": "company",
  "platform": "web",
  "source_type": "file",
  "time": "2022-10-20T22:59:12+00:00"
}

In the above, we can see that there is a 1 second difference, which leads me to think Loki was unable to parse the timestamp in the time field, and just reverted to setting the timestamp as time received

I am unable to find any documentation that states what timestamp format Loki expects in the time field. The documentation I find only relates to date parsing stages in promtail (timestamp | Grafana Loki documentation)

In the Loki configuratoin file, I have enabled unorderd_writes in order to allow historical dates.

limits_config:
  unordered_writes: true
2

I’m also struggling with this. We want to move to using RFC3999Nano but need to confirm it won’t break anything first.

BTW - vector VRL may lose precision when converting to 19-digit epoch time

$ .log.time = "2022-11-09T00:32:03.123456789Z"
"2022-11-09T00:32:03.123456789Z"

$ .timestamp = to_unix_timestamp(to_timestamp!(.log.time), unit: "nanoseconds") / 1000000000
1667953923.1234567
3

The Vector Loki sink POSTs a stream of log messages using Loki’s push api (HTTP API | Grafana Loki documentation)

Each entry in the stream is expected to have a nanosecond-precision epoch timestamp before the entry. Vector uses the .timestamp field if it is present, otherwise now(). We were able to use our custom field .log.time to update .timestamp in VRL using

.timestamp = .timestamp || to_unix_timestamp(to_timestamp!(.log.time), unit: "nanoseconds") || now()
4

Hey @wadefitzpatrick ,

I am able to convert the time field present in log using below transofmration

. = parse_json!(.message)
        parsed_timestamp,err = parse_timestamp(.time, "%Y-%m-%d %H:%M:%S.%f")        
        .timestamp = to_unix_timestamp(to_timestamp(parsed_timestamp), unit: "nanoseconds")

raw logs-
{"dvc_ip":"10.0.0.2","src_ip":"10.0.0.91","transport":"tcp","time":"2023-02-19 23:59:49.396641","packets":1,"bytes":1096,"vlan":0}

But in loki , I see timestamp being added as a field