Logs are not being sent to Loki in current format, adding different logs to same file uploads new lines only

Hello!
I am connecting Alloy with Grafana + Loki host in our AWS account. On a separate host is the Alloy service that i run among others in ECS. All connections light green. Logs are being tailed but no logs arrive in Grafana. I noticed that when i parse a different log file (that i do not need) additionally, i can get the logs almost instantly. Today i tested the same and when i copied few lines from the other file into the already populated log file that i monitor, the new logs arrived but old ones that are actual application log did not arrive. Is there any problem with formatting? Sharing my configurations:

logging {
  level = "debug"
  format = "logfmt"
}
local.file_match "local_files" {
  path_targets = [{"__path__" = "/var/log/**/catalina*.log"},{"__path__" = "/var/log/**/addon*.log"}]
  sync_period = "5s"
}

loki.source.file "log_scrape" {
  targets    = local.file_match.local_files.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}



loki.write "grafana_loki" {
  endpoint {
    url = "http://<SomeIPAddress>:3100/loki/api/v1/push"
  }
}

and my docker compose:

networks:
  loki:
    name: grafana-stack

services:
  loki:
    image: grafana/loki:2.9.2
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-data:/loki:rw
      - ./loki-config:/etc/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-data:/var/lib/grafana:rw
      - ./grafana-config/grafana.ini:/etc/grafana/grafana.ini
      - ./grafana-config/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=********
      - GF_SECURITY_ADMIN_PASSWORD=********
    depends_on:
      - loki
    networks:
      - loki

and finally Loki config file:

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

and some examples for the logs :
this does NOT WORK:

19-Dec-2024 14:36:37.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
19-Dec-2024 14:36:37.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
19-Dec-2024 14:36:37.239 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [2.0.6] using APR version [1.7.0].
19-Dec-2024 14:36:37.244 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 3.0.2 15 Mar 2022]

and this DOES WORK but logs as a single message:

10.0.1.220 - - [19/Dec/2024:15:32:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:32:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:33:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:33:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:33:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:33:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:34:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:34:25 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.1.220 - - [19/Dec/2024:15:34:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -
10.0.2.195 - - [19/Dec/2024:15:34:55 +0000] "GET /advanced-reports HTTP/1.1" 302 -

this results in only the new lines being seen in grafana as a single message and missing the older lines:


I am visualizing this only using the filename as filter:

Can you please help me understand this and make it work?

Couple of things:

  1. Loki will not accept older logs for a log stream if there are already newer entries. Log stream is defined as logs with the same labels.
  2. Grafana Alloy, like all logging agents, keep a state of the files it scrapped from and where, and it will not reprocess the portion of the files that it already processed.

Also, because of #1, you’ll want to make sure all your alloy instances have some sort of identifying label such as hostname.

1 Like

The thing is, when i set it to read the whole folder, that contains 3 files, it only parses the TXT one and ignores the .log ones. If i copy lines from the txt and put them in .log it works and i can see the .log path in Grafana. If i do not do that, the logs i need (tomcat logs and app log) are ignored.