Grafana loki for log reading

Grafana Loki
1

I have set up Loki server on Ec2 and aggregating the logs of ECS services running on my AWS. I am storing logs on Loki server and pushing the logs to S3 bucket for back up. Is it possible to read logs from Loki server directly? If yes can you please help me with that. Also I have set the retention period to 0s for storing logs for indefinite period of time but still not able to see it under retention directory on Loki-local-storage. I also shared my Loki config file with you.

Kindly help me in this as I am doing R&D for Loki log aggregation.

2

You can read logs from loki directly by using the Loki HTTP API. For example here’s an endpoint that will let you run any logql query and get results

I don’t see your loki configuration file in the question though

1 Like
3

I want to store my logs on my Loki server too for indefinite period. Following is my configuration. I also set retention period as 0s but still not able to do it. I want to push the logs from Loki server to my s3 bucket at 1 hr interval. Kindly help me correcting my configuration file.

auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_format: json
log_level: info

limits_config:
enforce_metric_name: false
reject_old_samples: false
max_cache_freshness_per_query: 1m
split_queries_by_interval: 15m
max_label_name_length: 1024
max_label_value_length: 2048
max_label_names_per_series: 30
unordered_writes: false
deletion_mode: disabled
retention_period: 0s

ingester:
wal:
enabled: true
dir: /mnt/loki/wal
checkpoint_duration: 5m
flush_on_shutdown: false
replay_memory_ceiling: 4GB
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h
max_chunk_age: 1h
chunk_target_size: 1572864
chunk_retain_period: 0s
max_transfer_retries: 0
max_returned_stream_errors: 10
chunk_encoding: none

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: grafana_loki
period: 24h
chunks:
prefix: loki_chunk
period: 24h

query_range:
align_queries_with_step: true
max_retries: 5
parallelise_shardable_queries: true
cache_results: true

storage_config:
boltdb_shipper:
active_index_directory: /mnt/loki/boltdb-shipper-active
cache_location: /mnt/loki/boltdb-shipper-cache
cache_ttl: 24h
resync_interval: 5m
shared_store: s3
aws:
s3: s3://us-east-1/loki-ecs
s3forcepathstyle: true
region: us-east-1
bucketnames: loki-ecs

query_scheduler:
max_outstanding_requests_per_tenant: 100

index_gateway:
mode: simple
ring:
kvstore:
store: inmemory
prefix: collector/

analytics:
reporting_enabled: true

tracing:
enabled: true

compactor:
working_directory: /mnt/loki/boltdb-shipper-compactor
shared_store: s3
compaction_interval: 1h
retention_enabled: true

chunk_store_config:
max_look_back_period: 0s

ruler:
storage:
type: local
local:
directory: /mnt/loki/rules
rule_path: /mnt/loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
enable_alertmanager_v2: true
4

Oh my. You have retention_period: 0s in your config file. That’s not going to work. :slight_smile:

Check the Loki storage retention documentation here:

It will give you a sample config that shows you how to set up any retention period. Note it requires an index interval of 24h but it looks like you already have that.

1 Like
5

Thanks @davidallen5 Let me check with that.

1 Like
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.