Hello!
I am connecting Alloy with Grafana + Loki host in our AWS account. On a separate host is the Alloy service that i run among others in ECS. All connections light green. Logs are being tailed but no logs arrive in Grafana. I noticed that when i parse a different log file (that i do not need) additionally, i can get the logs almost instantly. Today i tested the same and when i copied few lines from the other file into the already populated log file that i monitor, the new logs arrived but old ones that are actual application log did not arrive. Is there any problem with formatting? Sharing my configurations:
logging {
level = "debug"
format = "logfmt"
}
local.file_match "local_files" {
path_targets = [{"__path__" = "/var/log/**/catalina*.log"},{"__path__" = "/var/log/**/addon*.log"}]
sync_period = "5s"
}
loki.source.file "log_scrape" {
targets = local.file_match.local_files.targets
forward_to = [loki.write.grafana_loki.receiver]
tail_from_end = true
}
loki.write "grafana_loki" {
endpoint {
url = "http://<SomeIPAddress>:3100/loki/api/v1/push"
}
}
and my docker compose:
networks:
loki:
name: grafana-stack
services:
loki:
image: grafana/loki:2.9.2
container_name: loki
ports:
- "3100:3100"
volumes:
- ./loki-data:/loki:rw
- ./loki-config:/etc/loki
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
volumes:
- ./grafana-data:/var/lib/grafana:rw
- ./grafana-config/grafana.ini:/etc/grafana/grafana.ini
- ./grafana-config/provisioning:/etc/grafana/provisioning
environment:
- GF_SECURITY_ADMIN_USER=********
- GF_SECURITY_ADMIN_PASSWORD=********
depends_on:
- loki
networks:
- loki
and finally Loki config file:
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
and some examples for the logs :
this does NOT WORK:
19-Dec-2024 14:36:37.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
19-Dec-2024 14:36:37.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
19-Dec-2024 14:36:37.239 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [2.0.6] using APR version [1.7.0].
19-Dec-2024 14:36:37.244 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 3.0.2 15 Mar 2022]
and this DOES WORK but logs as a single message:
10.0.1.220 - - [19/Dec/2024:15:32:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:32:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:33:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:33:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:33:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:33:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:34:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:34:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:34:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:34:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
this results in only the new lines being seen in grafana as a single message and missing the older lines:
I am visualizing this only using the filename as filter:
Can you please help me understand this and make it work?