Local Install:
Ubuntu 22.04 LTS
I’ve installed
$sudo apt install Grafana Loki Alloy
via APT after adding Grafana’s repo and gpg keys.
Goals:
I only have this one computer and want the ability to view and visualize the logs.
Auditd Logs:
This is the main purpose, review and possibly alert on events.
204 keys in my audit.rules file
Is there a way to track the ‘Key=’ names and combine related ones for quick view?
How can I set alerts on relevant activity for LOTL methods in my audit.rules file?
Purpose:
Detecting events when someone is running shell scripts, setting up reverse shells, installing to RAM things like mfsconsole, changing the firewall settings/rules, loading or unloading the firewall, disabling systemd firewall features or other systemd services, any of the GTFObins, modifying running processes (browser/login screen)…maybe other issues.
So Far:
I have Grafana running, localhost:3000.
I have Alloy.config with the primary 4 listed from the document,
“grafana.com docs tutorials/send-logs-to-loki”
There is no information showing in the Grafana page.
In Alloy my target path is __PATH__ = "/var/**/*.log"
I expect to see the local log files.
How do I get the local log files to appear in Grafana?
Should I comment out the “Prometheus” information in the alloy.conf?
please share your alloy config file.
Where are the CODE tags? I’m getting a posting error for Loki addresses in the alloy file.
‘Sorry, New users can only put 2 links per post’
just replace the loki address with mickey.mouse
also why are you scrapping all of this __PATH__ = "/var/**/*.log" instead of just
/var/log/grafana/*.log
Not possible. I have replaced all of the url’s and it is still giving me an error.
Maybe the site interprets my commenting out code with // as a url.
I don’t know.
#1
local.file_match "local_files" {
path_targets = [{"__path__" = "/var/**/*.log"}]
sync_period = "5s"
}
#2
loki.source.file "log_scrape" {
targets = local.file_match.local_files.targets
forward_to = [loki.process.filter_logs.receiver]
tail_from_end = true
}
loki.write "grafana_loki" {
endpoint {
url = "localhost:3100/loki/api/v1/push"
// basic_auth {
// username = "admin"
// password = "admin"
// }
}
}
#4
loki.process "filter_logs" {
stage.drop {
source = ""
expression = ".*Connection closed by authenticating user root"
drop_counter_reason = "noisy"
}
forward_to = [loki.write.grafana_loki.receiver]
}
#5
loki.process "filter_logs" {
stage.drop {
source = ""
expression = ".*Connection closed by authenticating user root"
drop_counter_reason = "noisy"
}
forward_to = [loki.write.grafana_loki.receiver]
}
and now please share the screen where you are querying loki in grafana, maybe go to the datasource of Loki and click explore
well, you need to select something
I can’t take a screenshot of the drop down box where it says No Options Found under Lable filters.
Lable browser
No Lables Found
that might means nothing is being scrapped. I would recommend you implement
loki.echo "debug" { }
to see what exactly is happening and keep your eye on the logs Loki is generating to see if there are any issues.
that is just health status, it does not tell you if it is scrapping or not.
implement debug and see if anything is being scrapped.
