Reading of few logs missed by promtail due to frequent log rotations

Grafana Loki
,
1

Few log lines are missed by promtail due to frequent file rotation of log file. May be the time when log file is compressed to .gz file, promtail misses recent few log lines during compression process. Kindly suggest how to increase read throughput of promtail so that logs are not missed.
here is the promtail config:

---
# Loki Config
# (version=2.8.4, branch=HEAD, revision=89d282c43)
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
  http_listen_conn_limit: 0
  grpc_listen_network: tcp
  grpc_listen_address: ""
  grpc_listen_port: 0
  grpc_listen_conn_limit: 0
  tls_cipher_suites: ""
  tls_min_version: ""
  http_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: true
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 2m0s
  grpc_server_max_recv_msg_size: 4194304
  grpc_server_max_send_msg_size: 4194304
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_connection_idle: 2562047h47m16.854775807s
  grpc_server_max_connection_age: 2562047h47m16.854775807s
  grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
  grpc_server_keepalive_time: 2h0m0s
  grpc_server_keepalive_timeout: 20s
  grpc_server_min_time_between_pings: 5m0s
  grpc_server_ping_without_stream_allowed: false
  log_format: logfmt
  log_level: info
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  log_request_at_info_level_enabled: false
  http_path_prefix: ""
  external_url: ""
  health_check_target: null
  disable: false
  profiling_enabled: false
  enable_runtime_reload: false
client:
  url: ""
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  enable_http2: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  drop_rate_limited_batches: false
  stream_lag_labels: ""
clients:
- url: http://loki-endpoint
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  enable_http2: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 1m0s
  tenant_id: tenant_1
  drop_rate_limited_batches: false
  stream_lag_labels: ""
positions:
  sync_period: 10s
  filename: /etc/promtail/positions.yaml
  ignore_invalid_yaml: false
scrape_configs:
- job_name: job_a
  static_configs:
  - targets: []
    labels:
      __path__: /var/log/application.log
      host: 127.0.0.1
      job: job_a

- job_name: job_b
  static_configs:
  - targets: []
    labels:
      __path__: /var/log/access.log
      host: 127.0.0.1
      job: job_b

target_config:
  sync_period: 10s
  stdin: false
limits_config:
  readline_rate: 10000
  readline_burst: 10000
  readline_rate_drop: true
  max_streams: 0
  max_line_size: 0
  max_line_size_truncate: false
tracing:
  enabled: true
wal:
  enabled: false
  dir: ""
  cleanSegmentsOlderThan: 0s
2

@tonyswumac Kindly suggest if you have faced this issue with heavy application log files.

3

How are you rotating your log files?

4

Hi @tonyswumac , My application logs file is rotated once it reaches 250 MB size and generally 250 MB volume is reached in 5 mins.
So 250MB file is getting rotated after every ~5 mins.

5

Do you have configuration on the tools you use for log rotation? Asking because if you rotate log with copy-and-truncate then it is expected that some logs would be lost during that process.

6

@tonyswumac these logs are rotated by log4j,
can you suggest any way promtail’s frequency of log reading can be increased.
Or else please suggest what should be done in such cases.

7

Looks like it’s configurable with min_poll_frequency and max_poll_frequency (see Configure Promtail | Grafana Loki documentation).

8

yes @tonyswumac i have changes min_poll_frequency from 250ms to 50ms. Seems its working well now.

9

@tonyswumac here is the log4j config which handles log rotation

{
    "configuration":{
        "name":"ppb",
        "properties":{
            "property":[
                {
                    "name":"log-path",
                    "value":"/var/log/xxx-validate"
                },
                {
                    "name":"archive",
                    "value":"/var/log/xxx-validate"
                },
                {
                    "name":"pattern",
                    "value":"%X{requestId} - [%-5level] %d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'} [%t] %c{1} - %msg%n"
                },
                {
                    "name":"jsonPattern",
                    "value":"{\"requestId\" : \"%X{requestId}\",\"level\" : \"%-5level\",\"time\" : \"%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}\",\"thread\" : \"%t\",\"class\" : \"%c{1}\",\"message\" : \"%msg\"}%n"
                }
            ]
        },
        "appenders":{
            "Console":{
                "name":"Console-Appender",
                "target":"SYSTEM_OUT",
                "PatternLayout":{
                    "pattern":"${pattern}"
                }
            },
            "RollingFile":[
                {
                    "name":"RollingLogFile-Appender",
                    "fileName":"${log-path}/xxx-validate_application.log",
                    "filePattern":"${archive}/xxx-validate_application.log.%d{yyyy-MM-dd-HH}-%i.gz",
                    "PatternLayout":{
                        "pattern":"${pattern}"
                    },
                    "Policies":{
                        "OnStartupTriggeringPolicy":{
                            "minSize":1
                        },
                        "TimeBasedTriggeringPolicy":{
                            "interval":1,
                            "modulate":true
                        },
                        "SizeBasedTriggeringPolicy":{
                            "size":"250 MB"
                        }
                    },
                    "DefaultRolloverStrategy":{
                        "fileIndex": "nomax",
                        "Delete": [
                            {
                                "basePath": "${archive}",
                                "IfFileName": {
                                    "glob": "xxx-validate_application.log*.gz"
                                },
                                "IfAccumulatedFileCount": {
                                    "exceeds": "100"
                                }
                            }
                        ]
                    }
                    },
                {
                    "name":"CSTRollingLogFile-Appender",
                    "fileName":"${log-path}/cst-xxx-validate_application.log",
                    "filePattern":"${archive}/cst-xxx-validate_application.log.%d{yyyy-MM-dd-HH}-%i.gz",
                    "PatternLayout":{
                        "pattern":"${pattern}"
                    },
                    "Policies":{
                        "OnStartupTriggeringPolicy":{
                            "minSize":1
                        },
                        "TimeBasedTriggeringPolicy":{
                            "interval":1,
                            "modulate":true
                        },
                        "SizeBasedTriggeringPolicy":{
                            "size":"250 MB"
                        }
                    },
                    "DefaultRolloverStrategy":{
                        "fileIndex": "nomax",
                        "Delete": [
                            {
                                "basePath": "${archive}",
                                "IfFileName": {
                                    "glob": "cst-xxx-validate_application.log*.gz"
                                },
                                "IfAccumulatedFileCount": {
                                    "exceeds": "100"
                                }
                            }
                        ]
                    }
                }
            ]
        },
        "loggers":{
            "logger":[
                {
                    "name":"com.xxx.yyy",
                    "level":"info",
                    "additivity":"false",
                    "appender-ref":[
                        {
                            "ref":"RollingLogFile-Appender",
                            "level":"debug"
                        }
                    ]
                },
                {
                    "name":"com.xxx.elasticsearch",
                    "level":"info",
                    "additivity":"false",
                    "appender-ref":[
                        {
                            "ref":"RollingLogFile-Appender",
                            "level":"debug"
                        }
                    ]
                },
                {
                    "name":"com.jcraft.jsch",
                    "level":"error",
                    "additivity":"false",
                    "appender-ref":[
                        {
                            "ref":"RollingLogFile-Appender",
                            "level":"debug"
                        }
                    ]
                },
                {
                    "name":"com.xxx.pgyyy.cst",
                    "level":"info",
                    "additivity":"false",
                    "appender-ref":[
                        {
                            "ref":"RollingLogFile-Appender",
                            "level":"debug"
                        }
                    ]
                }
            ],
            "root":{
                "level":"info",
                "appender-ref":[
                    {
                        "ref":"Console-Appender",
                        "level":"info"
                    },
                    {
                        "ref":"RollingLogFile-Appender",
                        "level":"info"
                    }
                ]
            }
        }
    }
}
10

here is the promtail v2.9.2 config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /etc/promtail/positions.yaml
  sync_period: 250ms

clients:
  - url: http://loki.distributor
    tenant_id: A
    timeout: 60s
    batchsize: 20971520 #2MB

target_config:
  sync_period: 250ms

options:
  stream_lag_labels: filename

global:
  file_watch_config:
    min_poll_frequency: 50ms
    max_poll_frequency: 50ms

limits_config:
  readline_rate_drop: false
  readline_rate_enabled: false
  readline_rate:  100000
  readline_burst:  100000

scrape_configs:
-   job_name: applicationLogs
    pipeline_stages: null
    static_configs:
    -   labels:
            __path__: /var/log/application.log
            host: 127.0.0.1
            job: applicationLogs
            stack: prod
-   job_name: tomcatLog
    pipeline_stages: null
    static_configs:
    -   labels:
            __path__: /var/log/access.log
            host: 127.0.0.1
            job: tomcatLog
            stack: prod
11

Not sure if this is related: Promtail: Use `inotify` based file watcher instead of `polling` in Linux · Issue #7967 · grafana/loki · GitHub

Also, you might consider trying some other log agents and see if they have similar problems.