Multi-server monitoring setup - looking for best approach

Grafana Configuration
1

Hi all,

I’m setting up a monitoring dashboard for several services I am using and could use some advice.

My setup

  • Main server: has internet but isn’t publicly reachable, where I am hosting Grafana container (Grafana v12.0.2).

  • EC2 instances: running Apache, Docker/Laravel apps

  • C++ application which produces text logs

The problem is that the remote services cannot directly push the information eventually to a Loki/Prometheus service on my server, so I need something in between to collect them.

What I want to monitor

  1. Logs from a C++ app (currently just plain text files).

  2. Apache access/error logs.

  3. Uptime checks for web services (Apache, Laravel application, etc.).

My idea

Install Grafana Alloy (or Loki+Promtail but I have seen that soon is going to be deprecated and here is where my confusion is beginning) on each server to collect logs/metrics and send them somewhere.

Questions

  1. Is Grafana Alloy the right tool here, or should I look at Promtail or anything else?

  2. Best way to ingest plain text C++ logs into Loki?

  3. Does Alloy handle Apache log parsing well, or will I need heavy regex work?

  4. Should uptime monitoring also go through Alloy, or use something like Blackbox Exporter?

  5. Any performance issues with running Agent on lots of servers?

I could use Cloudwatch Logs but I would like to avoid additional costs because of that.

Has anyone here built a similar setup? What worked well, and what would you avoid?

Thanks!

2
  1. Is Grafana Alloy the right tool here, or should I look at Promtail or anything else?

For stable result, in my experience, promtail is the best one

  1. Best way to ingest plain text C++ logs into Loki?

Just put your proper logs into specific folder, give the loki user to access it. It’s better if you could make the logs in json, it’s simple to parse in Grafana panel. Don;t forget to put the logrotate to make the promtail works in light resource since if the log file is too big, it hurts the OS relating to the open file.

  1. Does Alloy handle Apache log parsing well, or will I need heavy regex work?

If using promtail with json format log, it’s very easy, but if not, just using the regex or pattern

  1. Should uptime monitoring also go through Alloy, or use something like Blackbox Exporter?

You can use blackbox exporter and use the proper port, if just http/https, it’s easy, even you can get the response from the result

  1. Any performance issues with running Agent on lots of servers?

For the server that have the agent, it’s not too much resources if you config the log in proper manner as mentioned above. The Loki server is the key, Just read the Loki configuration documentation to adjust the ingest size, burst and tenant if needed.

Regards,

Fadjar