Am I missing some pre-requisite software here? I start a new Hetzner VPS, I have the following alloy config:
# (...prometheus rules - which do work correctly)
loki.write "default" {
endpoint {
url = env("LOGS_URL")
bearer_token = env("LOGS_TOKEN")
headers = {
"X-Scope-OrgID" = env("SERVER_ID"),
}
}
external_labels = {
"server_id" = env("SERVER_ID"),
}
}
loki.source.journal "read" {
forward_to = [loki.write.default.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {component = "loki.source.journal"}
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
rule {
source_labels = ["__journal__boot_id"]
target_label = "boot_id"
}
rule {
source_labels = ["__journal__transport"]
target_label = "transport"
}
rule {
source_labels = ["__journal_priority_keyword"]
target_label = "level"
}
rule {
source_labels = ["__journal__hostname"]
target_label = "instance"
}
}
The configuration works, alloy starts - I write to the journal - I tail the journal to make sure things are going in there - it just never sends anything. I enabled debugging metrics, no lines are processed - nothing hits my HTTP endpoint. I tried proxying loki.source.journal through a process that I could livedebug, nothing goes through there either. I have tried changing the journal path (/var/log/journal), among many other things. I must just be missing something incredibly basic here. Please advise - using latest alloy version on a fresh VPS (Ubuntu 24.04 via Hetzner).
Ok - I actually just tried the loki.source.journal example verbatim:
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "read" {
forward_to = [loki.write.endpoint.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {component = "loki.source.journal"}
}
loki.write "endpoint" {
endpoint {
url ="loki:3100/api/v1/push"
}
}
and loki_source_journal_target_lines_total remains 0. Now I’m really lost. Nothing broken, alloy service status:
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.182875994Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec node_id=loki.write.endpoint duration=505.4µs
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.183057105Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec node_id=loki.relabel.journal duration=159.39µs
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.184221294Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec node_id=loki.source.journal.read duration=1.145034ms
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.184273674Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec node_id=tracing duration=29.366µs
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.18432504Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec node_id=remotecfg duration=35.877µs
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.184334488Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=dd83a4194b3b28b59246e5fbf0d56fec duration=2.220167ms
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.184642338Z level=info msg="scheduling loaded components and services"
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.184902196Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.185087925Z level=info msg="peers changed" service=cluster peers_count=1 peers=ubuntu-2gb-hil-1
Sep 07 18:14:36 ubuntu-2gb-hil-1 alloy-linux-amd64[10853]: ts=2024-09-07T18:14:36.185475835Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
Ok - so I spent a solid day on this - and it’s permissions related. I ended up finding a comment from 2021 on grafana-agent’s github issues:
Agent cannot scrape journal for Systemd 246 · Issue #524 · grafana/agent (github.com)
usermod -a -G adm grafana-agent
usermod -a -G systemd-journal grafana-agent
Checking and warning about lack of permissions I think is probably important. I am genuinely surprised how little information there was about this, I would have thought that many people would have encountered the same issues. Perhaps Alloy just isn’t that popular right now?
I’ve opened up this PR: Add a note about group membership by clayton-cornell · Pull Request #1636 · grafana/alloy · GitHub to add a note about the group membership to the loki.source.journal documentation: loki.source.journal | Grafana Alloy documentation
Thank you for following up with the solution you found. There’s a lot that we can do to improve the Alloy documentation and every comment helps. If you see something else that’s wrong or missing in the docs, you can send feedback through the specific doc page, ask a question here on the forums, raise an Issue, or even open up your own PR with a proposed change.
1 Like
