Promtail Config - Unable to overwrite the log line scrapped timestamp with the Log generated timestamp

Grafana Dashboards
, , ,
1

  • What Grafana version and what operating system are you using?
    I am using Grafana version 9.3.6.

  • What are you trying to achieve?
    Using Grafana I should be able to view Historical log data based on the timestamp when the log were created and not the timestamp when the log was extracted or scrapped. This allows me to generate Time series graph based on the Application log timestamp.

  • How are you trying to achieve it?
    Collect Logs from Application server using Windows Version of Promtail 2.8.2 and push it to Loki DB.

  • What happened?
    I use the timestamp stage in action stage to convert the extracted log timestamp to be the timestamp of a log line before it is sent to Loki

  • What did you expect to happen?
    The Time field in Loki should reflect the extracted log timestamp and not the scraped timestamp of the log line.

  • Can you copy/paste the configuration(s) that you are having problems with?
    scrape_configs:

  • job_name: Test.Project.28
    static_configs:
    • targets:
      • localhost
    • labels:
      Environment: LOCAL
      HostIP: 10.10.10.100
      HostName: ServerName
      ServerType: WebServices
      ProjectName: Test.Project.28
      path: D:\Program Files\Application\Logs\Test.Project\TimeStampLogTest.log
      pipeline_stages:
    • multiline:
      firstline: ‘^[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}’
      max_wait_time: 5s
      max_lines: 10000
    • regex:
      expression: ‘^[(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}).\d+] (?P[.]) (?P[.]) (?P[.]) - (?P[(?s:.)])’
    • labels:
      LogTimestamp:
      LogThreadId:
      LogLevel:
      ClassName:
      LogMessage:
    • timestamp:
      source: LogTimestamp
      format: 2006-01-02 15:04:05
      location: “America/Chicago”

image

In the picture from Grafana explore window, you can see the Log Line Scrapped timestamp and log line timestamp and they are not the same.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No Errors in config or output

  • Did you follow any online instructions? If so, what is the URL?

I have been struggling with this issue for days and unable to find a solution. Any help greatly appreciated.

Thanks
Sarath

2

Hi,
Just checking if anyone could provide some help on this issue. I am unable to overwrite the log line scrapped timestamp with the Log Created timestamp. I have used all formats and location tags in the timestamp stage and nothing really helps. Please let me know if you have any suggestions.

Thanks