When I used Grafana explorer to view logs, I found the collected logs were not order by time.
The first time is when I start Promtail, and the second time is the actual logging time. I tried to erase the first time but failed
My original log files are stored in different files at different levels, and use Promtail to collect logs and upload them to Loki
Origin log pattern like below:
2024-07-18 09:00:47 | quartzScheduler_Worker-2 | INFO | c.jybfgs.iotx.upa.service.config.aspect.LogAspect | execute job finish:{"classMethod":"com.jybfgs.iotx.upa.service.protocl.p645.service.impl.DeviceOperateServiceImpl.nextActionJob","requestParams":{"total":3,"index":0},"timeCost":59}
The time label configured by Promtail also matches the log content.
Why does this happen? How can I display the logs in chronological order?
Environment:
Grafana 10.4.2
Loki 2.8.2
Promtail 2.1.0
-----promtail config------
server:
http_listen_port: 9080
grpc_listen_port: 0
grpc_server_max_recv_msg_size: 8388608
grpc_server_max_send_msg_size: 8388608
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
- job_name: upa
pipeline_stages:
- regex:
expression: ‘^(?P\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}).*’
- labels:
time:
- timestamp:
source: time
format: “2006-01-02 15:04:05 UTC”
static_configs:
- targets:
- localhost
labels:
app: upa
type: normal
level: info
__path__: /var/log/upa/*info*log
-------Loki config-------
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 3110
grpc_server_max_recv_msg_size: 1073741824
grpc_server_max_send_msg_size: 1073741824
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0
max_chunk_age: 20m
schema_config:
configs:
- from: 2023-07-03
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
query_scheduler:
max_outstanding_requests_per_tenant: 10000
storage_config:
boltdb:
directory: /mnt/software/loki/index
filesystem:
directory: /mnt/software/loki/chunks
limits_config:
enforce_metric_name: false
reject_old_samples: false
#reject_old_samples_max_age: 168h
ingestion_rate_mb: 16
ingestion_burst_size_mb: 16