Grafana Alloy Memory issue

Grafana Alloy
1

Hello,

I’m using Grafana Alloy on a Windows server to send log files to Loki. I frequently encounter the following event log/error messages:

ts=YYYY-MM-DDT00:00:00.0000000Z level=info msg=“tail routine: tail channel closed, stopping tailer” component_path=/ component_id=loki.source.file.[MASKED_COMPONENT] component=tailer path=[MASKED_PATH][MASKED_FILENAME].log reason=“seek [MASKED_PATH]\[MASKED_FILENAME].log: file already closed”

When I restart the Alloy service, it also gives me this error:

ts=YYYY-MM-DDT00:00:00.0000000Z level=error source=[MASKED_SOURCE_PATH] msg=“failed to set GOMEMLIMIT” package=[MASKED_PACKAGE] error=“failed to set GOMEMLIMIT: cgroups is not supported on this system”

I’ve observed that when the “file already closed” error occurs, the log file was writing approximately 54,000 lines of logs within a few seconds.

Can any experts advise on how to resolve these issues? Specifically:

  1. How can I prevent the “file already closed” error, especially during high-volume logging?
  2. Is the GOMEMLIMIT error significant, and if so, how can it be addressed on a Windows system?

Thank you for your assistance!

1 Like
2

This is from an exercise of migrating Grafana Agent to Grafana Alloy

In Grafana Agent we have this

  # temporary limiting to avoid OOMs from promtail
  limits_config:
      readline_rate: 100
      readline_burst: 200
      max_streams: 10

I wondered if we have the equivalent one in Alloy ???

3

There are some tuning configurations in loki.write under the WAL section (see loki.write | Grafana Alloy documentation), but I am not sure if you can configure readline_rate in Alloy.

4

Tony, thank you so much for answering!
Do you have any best practice configuration as an example? The problem here is that the flow is testing environment is not the same as in production environment, I can’t reproduce it.
I don’t mind updating the alloy version to 1.7.1 in prod and give it a try.

5

I think your problem is more likely to be memory related than anything else. I don’t use Windows, but from my experience on Linux 54000 lines of logs in a few seconds shouldn’t be particularly taxing for log agents.

I don’t really tune alloy agents, I do sometimes configure a backoff settings, but I don’t think that would help in this case.

6

I came across this Possible memory leak in Alloy · Issue #670 · grafana/alloy · GitHub in which someone has also experienced GOMEMLIMI back in April 2024. Do you happen to know in which version of Alloy has this issue been fixed?

Currently, I am using Alloy v1.5.1

7

what are the specs on this windows server: ram, windows version?
Are there other software installed on this windows server, what is this server used for: AD, DHCP, SQL server, domain controller? What logs are you pushing from it?

do you have other windows servers that also have alloy or just this one?

8

Hello Yosiasz,

I am rolling out grafana alloy onto all the windows 2019 application servers, and yes, there are software installed on the boxes. the box that I am having GOMEMLIMIT problem with has 32GB ram, and the average memory usage on the box is usually 40%. Never go beyond 50%. Plenty of diskspaces, and well below 11% CPU usage all the time. As for the log, its just plain text log file output by an in-house application running as a service.

1 Like
9

it seems like some have resolved this by upgrading, you could try that in one of your servers

maybe you have seen this one

also is it just one file per server for this application? is it somethin that can be configured to be pushed to a centralized location like a dedicated alloy server?

10

Hello Yosiasz,
I am planning to upgrade the box with the latest Alloy version v1.7.1

As for your 2nd question , the answer is no, there are other files for this application.
When you say alloy server, you mean Loki server right? I think the Loki server that we have can coupe with huge amount of data, I don’t think configuring a dedicated Loki server would help as the Grafana agent works without any issue.

11

i was thinking of a dedicated log file/alloy server. All windows servers push their files to this server and alloy pushes logs to Loki. or some other architect…

12

Over the weekend, I have upgraded the alloy version to v1.7.4
however, I am still seeing the same issue, do we have more tooling for debugging in this version?
let me closely monitoring that for a few days and see if there is any behavioral changes in the latest version.

13

The “file already closed” error likely occurs due to rapid log rotation or file locking issues. Try increasing the max_wait and min_backoff settings in your Alloy configuration to give the tailer more time to handle file changes. Also, ensure no other processes (like antivirus scans) are interfering with the log files.

The GOMEMLIMIT error isn’t critical on Windows, as cgroups aren’t natively supported. You can ignore it or try setting GOMEMLIMIT=0 as an environment variable to suppress the warning.

14

Thank, let me check that two config as well.

By the way, the same issue still persist after updating alloy to version v1.7.4 over the past weekend.

1 Like
15

Also I am trying this as I found it in the docs loki.process | Grafana Alloy documentation

loki.source.file “test_logs” {
targets = local.file_match.test_logs.targets
forward_to = [loki.process.test_logs.receiver]
}
loki.process “test_logs” {
stage.regex {
expression = “^E:\\Logs\\[^\\]+\\(?P[A-z\.]+).log$”
source = “filename”
}
stage.labels {
values = {
region = “”,
app = “”,
}
}
stage.regex {
expression = “^(?P[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{4}) \[(?P[\w]+)\] .+”
}
stage.replace {
source = “timestamp”
expression = “^(.*)”
replace = “{{ .Value }}00”
}
stage.timestamp {
location = “Europe/London”
source = “timestamp”
format = “2006-01-02 15:04:05.000000”
}
stage.limit {
rate = 100
burst = 200
drop = true
}
forward_to = [loki.write.default.receiver]
}