Grafana agent - High CPU/Memory consumption

Grafana Alloy
1

After installing grafana agent notice high CPU and memory consumption on the instance. Instance getting hung. Please suggest the fix.

Top Command output:

┌──[root@we1o203420bbj]─[/root]
└──╼ $ top -c
top - 07:08:44 up 1:22, 1 user, load average: 1.58, 1.25, 1.18
Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu(s): 65.3 us, 20.6 sy, 0.0 ni, 11.7 id, 1.2 wa, 0.8 hi, 0.3 si, 0.0 st
MiB Mem : 15473.7 total, 9438.5 free, 2125.8 used, 3909.4 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 12994.8 avail Mem


PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND


27328 root 20 0 2171544 245384 110924 S **153.2** 1.5 0:06.13 /usr/bin/grafana-agent -config.file=/etc/grafana-agent.yaml

Version numbers (grafana, prometheus, graphite, plugins, operating system, etc.):

┌──[root@we1o203420bbj]─[/root]
└──╼ $ rpm -qa | grep grafana
grafana-agent-0.38.1-1.x86_64
┌──[root@we1o203420bbj]─[/root]
└──╼ $ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)

Using below agent config:
┌──[root@we1o203420bbj]─[/root]
└──╼ $ cat /etc/grafana-agent.yaml

Sample config for Grafana Agent

server:
log_level: warn

metrics:
global:
scrape*interval: 60s remote*write:
- url: https://xxxxx/mimir/api/v1/push
basic_auth:
username: "xxxxxxxxxxxxxxxx"
password: "XXXXXXXXXXXXXXXX"

wal*directory: '/var/lib/grafana-agent' configs: # Example Prometheus scrape configuration to scrape the agent itself for metrics. # This is not needed if the agent integration is enabled. # - name: agent # host*filter: false
# scrape*configs: # - job*name: agent
# static_configs:
# - targets: ['127.0.0.1:9090']

integrations:
agent:
enabled: true
node*exporter: enabled: true include*exporter*metrics: true disable*collectors:
- "mdadm"

# logs:

# configs:

# - name: default

# positions:

# filename: /tmp/grafanaagentpositions.yaml

# scrape_configs:

# - job_name: varlogs

# static_configs:

# - targets: [localhost]

# labels:

# job: messages

# **path**: /var/log/messages

# - job_name: otherlogs

# - targets: [localhost]

# labels:

# job: otherlogs

# **path**: /var/log/*log

logs:
configs:

* name: default
positions:
filename: /tmp/positions.yaml
scrape_configs:
  * job*name: loki-logs static*configs:
    * targets: [localhost]
labels:
job: varlogs
**path**: /var/log/*logclients:
  * url: xxxxxx/loki/api/v1/push
basic_auth:
username: "XXXXXXXXX"
password: "XXXXXXXXX"
2

I get this from the server sending the logs to Loki, not the Grafana UI or syslog/Loki server.

The problem, at least for me, seems to be accessing the log files. I’m pulling IIS logs, app logs and others. I end up backing up the log files to a ZIP file, and clearing the log files. I only need future logs and not past ones. The high cpu issue goes away after that.

I hope this helps.