I used this command to export logs
/export/loki-script/logcli --ca-cert /etc/nginx/conf/cert/ca.crt --cert /etc/nginx/conf/cert/loki.crt --key /etc/nginx/conf/cert/loki.key --timezone=UTC -q --addr=https://<LOKI_HOST>:3100 query '{app="APPNAME"}' --from="2025-03-21T16:00:00Z" --to="2025-03-22T16:00:00Z" --part-path-prefix=/export/loki-script/app/<APPNAME> --parallel-duration=10m --parallel-max-workers=20 --merge-parts -o raw --forward
but an error occurred:
2025/03/26 15:03:17 Invalid batch size 1000, the next query will have 1000 overlapping entries (there will always be 1 overlapping entry but Loki allows multiple entries to have the same timestamp, so when a batch ends in this scenario the next query will include all the overlapping entries again). Please increase your batch size to at least 1001 to account for overlapping entryes
I adjusted it up or down --parallel-duration=10m --parallel-max-workers --limit=1000, both are the same error。
Some configurations of loki.yml are as follows:
limits_config:
ingestion_rate_strategy: global
ingestion_rate_mb: 1024
ingestion_burst_size_mb: 1024
max_label_name_length: 1024
max_label_value_length: 2048
max_label_names_per_series: 15
reject_old_samples: false
reject_old_samples_max_age: 1w
creation_grace_period: 10m
max_line_size: 10MB
max_line_size_truncate: false
increment_duplicate_timestamp: true
discover_log_levels: true
use_owned_stream_count: false
max_streams_per_user: 100000
max_global_streams_per_user: 20000
unordered_writes: true
per_stream_rate_limit: 1GB
per_stream_rate_limit_burst: 1GB
max_chunks_per_query: 2000000
max_query_series: 5000
max_query_lookback: 0s
max_query_length: 100d
max_query_range: 0s
max_query_parallelism: 500
tsdb_max_query_parallelism: 1280
tsdb_max_bytes_per_shard: 600MB
tsdb_sharding_strategy: power_of_two
tsdb_precompute_chunks: false
cardinality_limit: 100000
max_streams_matchers_per_query: 1000
max_concurrent_tail_requests: 200
max_entries_limit_per_query: 0
max_cache_freshness_per_query: 10m
max_metadata_cache_freshness: 1d
max_stats_cache_freshness: 10m
max_queriers_per_tenant: 0
max_query_capacity: 0
query_ready_index_num_days: 0
query_timeout: 1h
split_queries_by_interval: 1d
split_metadata_queries_by_interval: 10m
split_recent_metadata_queries_by_interval: 1h
recent_metadata_query_window: 0s
split_instant_metric_queries_by_interval: 1h
split_ingester_queries_by_interval: 0s
min_sharding_lookback: 0s
max_query_bytes_read: 0B
max_querier_bytes_read: 150GB
volume_enabled: true
volume_max_series: 1000
Some log contents have duplicate timestamps.
how to solve the problem？
Try changing parallel duration to be bigger something like --parallel-duration=30m or even 1h.
Also unless you are in a hurry setting max worker smaller might be a better idea just so you don’t overrun your loki cluster.
I have adjusted all three parameters --parallel-duration=10m, --parallel-max-workers=20, --limit=1000 (set them to be larger or smaller), but the result is the same error
Maybe try without the limit. This is what I usually do when I need to output something with logcli:
/usr/bin/logcli \
query \
--timezone=UTC \
--from={start_string} \
--to={end_string} \
--output=default \
--parallel-duration=30m \
--parallel-max-workers=20 \
--part-path-prefix={logcli_output_prefix} \
$query
Sorry, I didn’t explain it clearly. When executing the query, the --limit parameter was not set. It was only after the error occurred and there was a hint of “1000” that I adjusted it to be larger and smaller, but the error still occurred.
Not sure, I’ve not had this problem. Do you see any error message from Loki?
I forgot to look at it at the time, but now it’s been so long and the logs are gone
Just curious, how many logs are there for the time period you are trying to export? I can see if i can replicate the same log volume and try it on our cluster.
We have 6 machines, 2 nginx+4 loki, exporting application logs every day, with a log size range of approximately 1M-30G.
The application that reported the error had approximately 13M logs on that day, with around 40000 logs. In the past few months, there has only been one occurrence of this error。
The configuration of loki.yml is as follows：
auth_enabled: false
memberlist:
bind_port: 7946
join_members:
- <IP1>:7946
- <IP2>:7946
- <IP3>:7946
- <IP4>:7946
server:
http_listen_port: 3100
http_tls_config:
cert_file: /<DIR>/loki.crt
key_file: /<DIR>/loki.key
client_ca_file: /<DIR>/ssl/ca.crt
client_auth_type: RequireAndVerifyClientCert
grpc_listen_port: 3110
grpc_server_max_recv_msg_size: 109951162777600 # 100G
grpc_server_max_send_msg_size: 109951162777600 # 100G
log_level: error # info debug error
common:
replication_factor: 1
ring:
replication_factor: 1
heartbeat_period: 2s
heartbeat_timeout: 5s
kvstore:
store: memberlist
path_prefix: /<DATA>/loki-data
ingester:
chunk_retain_period: 30m
flush_check_period: 1m
max_chunk_age: 2h
chunk_idle_period: 1h
chunk_target_size: 10485760
sync_min_utilization: 1
autoforget_unhealthy: true
wal:
enabled: true
lifecycler:
final_sleep: 10s
min_ready_duration: 3s
ring:
replication_factor: 1
schema_config:
configs:
- from: 2020-05-15
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_
period: 24h
storage_config:
tsdb_shipper:
resync_interval: 1m
aws:
s3: s3://<ACCOUNT>:<SECRET>@<URL>/loki
querier:
max_concurrent: 30
query_range:
max_retries: 2
frontend_worker:
query_frontend_grpc_client:
max_recv_msg_size: 107374182400
max_send_msg_size: 107374182400
query_scheduler_grpc_client:
max_recv_msg_size: 107374182400
max_send_msg_size: 107374182400
compactor:
compaction_interval: 60m
retention_delete_delay: 10m
retention_delete_worker_count: 150
retention_enabled: true
delete_request_store: aws
analytics:
reporting_enabled: false
limits_config:
max_global_streams_per_user: 200000
max_streams_per_user: 200000
ingestion_rate_strategy: global
ingestion_rate_mb: 1024
ingestion_burst_size_mb: 1024
reject_old_samples: false
max_query_length: 100d
max_query_parallelism: 500
max_entries_limit_per_query: 0
max_concurrent_tail_requests: 200
increment_duplicate_timestamp: true
max_line_size: 10240KB
tsdb_max_query_parallelism: 1280
split_queries_by_interval: 24h
query_timeout: 60m
split_metadata_queries_by_interval: 10m
per_stream_rate_limit: 1G
per_stream_rate_limit_burst: 1G
max_query_series: 5000
shard_streams:
enabled: false
retention_period: 30d
retention_stream:
- selector: '{app="<APP>"}'
period: 90d
