Promtail : How to set timestamp when there's '@' in JSON log line object key

Hi.
I use Promtail to tail a log that contains JSON-formatted lines. I used to use the default timestamp (time Promtail reads the line). When I tried to set the timestamp from the log line instead, it didn’t work for a JSON object key starting with a @ like @timestamp.
I tried escaping the @ with an entity, single and double quoting the value, but Promtail kept rejecting the config or falling back to the time of receipt, so I think the limitation is in Promtail and not in faulty YML.
@ seems commonly used for timestamp keys in JSON (logstash json_event, ECS) so maybe someone else has solved this - or I have missed something.
My YML looks like this:

 - job_name: xyz
   pipeline_stages:
   - json:
      expressions:
        time: "@timestamp"
   - timestamp:
      source: time
      format: RFC3339Nano

Setting the timestamp from any key name without @ worked fine.

Is this something that needs an enhancement, or is there a way to configure @? I’d prefer not to change my JSON but that seems to be the workaround.

Hi @anderssakoul

Did you try adding a backslash before the @?

 - job_name: xyz
   pipeline_stages:
   - json:
      expressions:
        time: "\@timestamp"

Promtail accepted “\timestamp” (not very useful!) but wouldn’t start if configured with "\@timestamp". I’ve tried @ and @ and \u0040 as well.

Please send the error message you received

for @timestamp or "@timestamp" the error is “Unable to parse config: promtail.yml: yaml: line 23: found character that cannot start any token”, reasonably enough according to the YAML spec. If I try \@timestamp for escaping I get “Unable to parse config: promtail.yml: yaml: line 23: found unknown escape character”. If I try \U0040 I get “Unable to parse config: promtail.yml: yaml: line 23: did not find expected hexdecimal number” For @, @ Promtail won’t start but nothing is output in the service log.

Oddly, only \U0040 got me a “did not find expected hexdecimal number”, and lower case \u0040 just got a silent failure to start.

Trying to escape a capital A instead of @ doesn’t work either : \U0041 A - though I’m not sure whether or not it’s supposed to, since it’s not a character that’s normally invalid in YAML.

I see.

Try this:

timestamp: '"@timestamp"'

Note that’s a single quote wrapping the quoted field: '"@timestamp"'

We use JMESPath in promtail to parse JSON expressions. You can test this out on https://jmespath.org/.

image

That seems to work : genius! Thanks for your help.

My pleasure :slight_smile:

Please mark the question as solved so future folks can find the answer, too.
Happy Loki-ing!

1 Like

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.