Logs sending to Loki

Hi,

I am trying to understand the default behaviour for sending logs from Alloy to Loki. I have already created config.alloy configuration, which works because I checked it with Alloy UI.

loki.write "loki" {
  endpoint {
    url = "https://coin_mon:3100/loki/api/v1/push"
  }
}

local.file_match "coin_application" {
  path_targets = [
    {"__path__" = "/dev_share/usr/coin/logs/application/*.log"},
  ]
  sync_period = "5s"
}

loki.source.file "log_scrape" {
  targets       = local.file_match.coin_application.targets
  forward_to    = [loki.process.filter_logs.receiver]
}

loki.process "filter_logs" {
  stage.regex {
        expression = "^(?P<ts>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) (?P<level>[A-Z]+) (?P<msg>.*)$"
  }

  stage.labels {
    values = {
      level = "{{ .level }}",
    }
  }

  stage.limit {
    rate  = 100
    burst = 50
  }

  forward_to = [loki.write.loki.receiver]
}

logging {
  level = "debug"
}

Now, I am testing how the logs are sending to Loki, so I type in /dev_share/usr/coin/logs/application/:
echo "2025-04-27 01:01:00 ERROR test" >> test.log, and waiting for logs to see in Loki. I do not see nothing after waiting several minutes.

On the other hand, when I type echo "2025-04-27 01:01:00 ERROR test" >> USDCoinApplication.log I see after several seconds logs in Loki, so the only difference is the USDCoinApplication.log was already there (path: /dev_share/usr/coin/logs/application/), but the test.log I created during my test when I type the above command.

  1. I do not understand that behavior, why do I see stream logs from the existing .log file but not from my newly created for testing purposes?

Second thing is about the way of how Alloy is filtering logs, my above config does not say anything about matches only ERROR level, however when I type echo "2025-04-27 01:01:00 ERROR test" >> USDCoinApplication.log after while I am able to see in Loki:

2025-04-27 10:55:38.805 2025-04-27 01:01:00 ERROR test

But when I type the same replacing ERROR to INFO I am not able to see any new log in Loki, so that does not work:
echo "2025-04-27 01:01:00 INFO test" >> USDCoinApplication.log

  1. Why does Alloy filter only ERROR, not INFO as well? Why? What should I change to see both INFO and ERROR (also WARN etc.)?

The last thing is about the fields which I can see in Loki itself, when I type echo "2025-04-27 01:01:00 ERROR test" >> USDCoinApplication.log, and check Loki I see:

  1. From where those Fields are populated (filename, job, log_level etc.)? I do not set them in my above config, so those are pre-populated fields by Loki itself? Can I change them, add more, how?

Thanks!

My Alloy learning summary, please correct me @tonyswumac if I am wrong at any point :slight_smile:

  1. Log ingestion didn’t work initially, but now:
  • I created a new log file and updated the Alloy config to point to it.
  • I injected an error manually using echo.
  • Then, I restarted the Alloy service.
  • In a second terminal, I had the Alloy config open and saw that the config was reloaded:
ts=2025-04-30T15:47:47.333Z level=info msg="tail routine: started"
ts=2025-04-30T15:47:47.333Z level=info msg="Seeked"
  • I saw the ERROR log hit the correct processing stage in the Alloy GUI, and then it appeared in Loki as expected.

What I learned Alloy tracks read positions in the positions.yaml file.
So if a log file was already read (even partially), restarting Alloy won’t re-read it unless:

  • the position is reset, or
  • the log file gets a new line appended
  1. Regex issue
  • My regular expression wasn’t working initially because the timestamp format/order in the logs didn’t match the expected pattern in the stage.regex, or rather the injected sample log which I put to the file earlier.
  1. How labels are created
  • I misunderstood how labels are attached.
  • Now I see that stage.labels is the place labels can be defined which can then be used for querying.