Why after set promtail pipline stage regex ,then the entry log will be duplicated?

I met some stranger question. When I Set the promtail configuation as pipline_stage to regex incoming log’s timepstamp and level,like this
server:
http_listen_port: 9080
grpc_listen_port: 0
http_server_read_timeout: 20s
http_server_write_timeout: 10s
grpc_server_max_recv_msg_size: 1048576
grpc_server_max_send_msg_size: 1048576
grpc_server_max_concurrent_streams: 512
log_level: info
positions:
filename: /app/docker/promtail/application-positions.yaml
sync_period: 10s
target_config:
sync_period: 10s
clients:

  • url: http://loki-ali.hfinside.com/loki/api/v1/push
    tenant_id: prod
    scrape_configs:
  • job_name: consul
    static_configs:
    • targets:
      • localhost
        labels:
        app: ‘fpc’
        ip: 172.19.38.72
        path: /app/docker/fpc/logs/*.log
        pipeline_stages:
    • regex:
      expression: ‘(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}) (?P[A-Z]+) .*’
    • labels:
      timestamp:
      level:

after this configuation,the entry log will be dupilcated and displayed on grafana
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
Log labels

app fpc

ip 172.19.38.72

filename /app/docker/fpc/logs/rocketmq_client-2022-08-04.0.log

level INFO

timestamp 2022-08-04 00:00:00.031
Detected fields

ts 2022-08-05T13:44:51.811Z

tsNs 1659707091811610186
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00
2022-08-05 21:44:51
2022-08-04 00:00:00.031
2022-08-04 00:00:00.031 INFO [MQClientFactoryScheduledThread] RocketmqClient - [PULL_TPS] [PROD_C_RMQ_PA_ACCT_IN_RESP_1025@PROD_Q_PA_FPC_INACCT_RESP_NOTIFY] Stats In One Minute, SUM: 26 TPS: 0.43 AVGPT: 1.00

there are 9 row same log send to loki server?

but when I delete the pipeline_stage in configuration,everything become to normal,just get 1 row on the grafana

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.