# Grafana wont load log lines after adding timestamp in promtail config

**URL:** <https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006>\
**Category:** Configuration\
**Created:** [September 26, 2022, 1:07am UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006 "2022-09-26T01:07:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![inftom1](https://avatars.discourse-cdn.com/v4/letter/i/a87d85/32.png) [@inftom1](https://community.grafana.com/u/inftom1)\
**Post date:** [September 26, 2022, 1:07am UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006/1 "2022-09-26T01:07:40Z")

</div>

- What Grafana version and what operating system are you using?  
grafana/loki:2.6.1  
grafana/promtail:2.6.1  
grafana/grafana:9.1.6-ubuntu
- **What** are you trying to achieve?  
My log lines currently have time stamps in the following format `20/Sep/2022:09:52:57 +0000`. Currently the timestamps on grafana are being read at scrape (default). I want them to populate with the actual time.
- **How** are you trying to achieve it?  
I am adding a timestamp stage with a custom format.
- What happened?  
Adding the timestamp stage causes logs to stop appearing when queried.
- What did you **expect** to happen?  
Logs would query all the same but with the correct timestamp.
- Can you copy/paste the configuration(s) that you are having problems with?  
promtail config

```auto
server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: debug

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

# local machine logs

- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__ : /var/log/*.log
  pipeline_stages:
  - match:
      selector: '{job="varlogs"}'
      stages:
      - json:
          expressions:
            time: time_local
      - timestamp:
          source: time
          format: '02/Jan/2006:15:04:05 -0700'
          fallback_formats: ['02/Jan/2006:15:04:05 -0700', 'RFC3339']
          action_on_failure: skip

```

Loki Config

```auto
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: debug

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  #ingestion_rate_mb: 10
  #ingestion_burst_size_mb: 50
  reject_old_samples: false
  #per_stream_rate_limit: 10MB
  #per_stream_rate_limit_burst: 50MB

```

Promtail logs

```auto
level=debug ts=2022-09-26T01:38:19.527634648Z caller=manager.go:265 component=discovery msg="Starting provider" provider=static/0 subs=map[local:{}]
level=debug ts=2022-09-26T01:38:19.527901298Z caller=manager.go:299 component=discovery msg="Discoverer channel closed" provider=static/0
level=info ts=2022-09-26T01:38:19.529054758Z caller=server.go:288 http=[::]:9080 grpc=[::]:39077 msg="server listening on addresses"
level=info ts=2022-09-26T01:38:19.52931686Z caller=main.go:121 msg="Starting Promtail" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)"
level=debug ts=2022-09-26T01:38:24.528096622Z caller=filetargetmanager.go:278 msg="new target" labels="{ __address__ =\"localhost\"}"
level=info ts=2022-09-26T01:38:24.528193572Z caller=filetargetmanager.go:338 msg="Adding target" key="/var/log/*.log:{job=\"varlogs\"}"
level=info ts=2022-09-26T01:38:24.534297631Z caller=filetarget.go:278 msg="watching new directory" directory=/var/log
level=debug ts=2022-09-26T01:38:24.534345771Z caller=filetarget.go:318 msg="tailing new file" filename=/var/log/test.log
ts=2022-09-26T01:38:24.535117494Z caller=log.go:168 level=info msg="Seeked /var/log/test.log - &{Offset:0 Whence:0}"
level=info ts=2022-09-26T01:38:24.53510711Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/test.log
level=debug ts=2022-09-26T01:38:24.536870553Z caller=json.go:182 component=file_pipeline component=stage type=json msg="extracted data debug in json stage" extracteddata="map[filename:/var/log/test.log job:varlogs timestamp:17/Sep/2022:09:52:57 +0000]"
level=debug ts=2022-09-26T01:38:24.537055305Z caller=json.go:182 component=file_pipeline component=stage type=json msg="extracted data debug in json stage" extracteddata="map[filename:/var/log/test.log job:varlogs timestamp:17/Sep/2022:09:52:58 +0000]"
level=debug ts=2022-09-26T01:38:24.537138191Z caller=json.go:182 component=file_pipeline component=stage type=json msg="extracted data debug in json stage" extracteddata="map[filename:/var/log/test.log job:varlogs timestamp:17/Sep/2022:09:52:58 +0000]"
level=debug ts=2022-09-26T01:38:24.537679344Z caller=json.go:182 component=file_pipeline component=stage type=json msg="extracted data debug in json stage" extracteddata="map[filename:/var/log/test.log job:varlogs timestamp:17/Sep/2022:09:52:58 +0000]"

```

Sample Log Line

```auto
{"time_local":"17/Sep/2022:09:52:57 +0000","request":"https://test.xyz/v1/client/relay","method":"POST","protocol":"HTTP/1.1","status":"200"}

```

- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us **exactly** what they were.  
I did not spot any errors.
- Did you follow any online instructions? If so, what is the URL?  
[timestamp | Grafana Loki documentation](https://grafana.com/docs/loki/latest/clients/promtail/stages/timestamp/)

---

<div class="post-metadata">

**Author:** ![yosiasz](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/yosiasz/32/45847_2.png) [@yosiasz](https://community.grafana.com/u/yosiasz)\
**Post date:** [September 26, 2022, 3:22am UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006/2 "2022-09-26T03:22:49Z")

</div>

Could you specify fhe exact timestamp format, the exact RFC?

> <https://github.com/grafana/loki/blob/main/docs/sources/clients/promtail/stages/timestamp.md>

---

<div class="post-metadata">

**Author:** ![inftom1](https://avatars.discourse-cdn.com/v4/letter/i/a87d85/32.png) [@inftom1](https://community.grafana.com/u/inftom1)\
**Post date:** [September 26, 2022, 4:25pm UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006/3 "2022-09-26T16:25:05Z")

</div>

I’m using a custom timestamp format  
`02/Jan/2006:15:04:05 -0700`

I tested here [Go Playground - The Go Programming Language](https://go.dev/play/) using this code

```auto
/ Golang program to illustrate the usage of
// time.Parse() function

// Including main package
package main

// Importing fmt and time
import (
	"fmt"
	"time"
)

// Calling main
func main() {

	// Declaring layout constant
	const layout = "02/Jan/2006:15:04:05 -0700"

	// Calling Parse() method with its parameters
	tm, _ := time.Parse(layout, "20/Sep/2022:09:52:57 +0000")

	// Returns output
	fmt.Println(tm)
}

```

---

<div class="post-metadata">

**Author:** ![inftom1](https://avatars.discourse-cdn.com/v4/letter/i/a87d85/32.png) [@inftom1](https://community.grafana.com/u/inftom1)\
**Post date:** [September 27, 2022, 9:07am UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006/4 "2022-09-27T09:07:23Z")

</div>

heres something interesting when running this command, that specifies the config file, it actually shows the converted timestamp in blue.  
`cat /var/log/test.log | promtail --stdin -log.level=debug --dry-run --config.file=/etc/promtail/promtail-config.yml --client.url ` `http://127.0.0.1:3100/loki/api/v1/push`

this is what my docker compose is set to run. so not sure why its not populating the same inside of grafana/loki frontend.

---

<div class="post-metadata">

**Author:** ![armesh49](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@armesh49](https://community.grafana.com/u/armesh49)\
**Post date:** [November 18, 2022, 6:23am UTC](https://community.grafana.com/t/grafana-wont-load-log-lines-after-adding-timestamp-in-promtail-config/73006/5 "2022-11-18T06:23:40Z")

</div>

I found that Grafana Loki has a huge delay on displaying logs with custom timestamp if the custom timestamp is older than 2 hours from “now”.

So just wait like 3 hours then check again in the UI. If it didn’t work you can restart Loki and wait another 3 hours.

I am running Grafana Enterprise, Loki & Promtail in Ubuntu 18 Virtual Machine.