Grafana Alloy | How to maintain config properly

Grafana Alloy
, ,
1

Hi there,

I’m quite new to this stack and I wonder if there are some good advices and/or best practices how to manage and maintain our stack. Partially my questions are theoretical but I expect these to be practical problems in the future.

I would like to stick to use the available helm charts as much as possible, and the stack runs in Kubernetes.

  1. I found it quite interesting to maintain the alloy config inline in the values.yaml.
    1. Should I use configMaps instead?
    2. Is there a good way to create a config and validate it before deployment to avoid breaking the alloy pods? I just realised alloy cli can do this.
  2. ProductA uses JSON logs, ProductB uses syslog, ProductC requires some additional things.
    1. How should I manage different rules for different products?
    2. Should I stick to one big config with different pre-filtering conditions?
    3. Should I go with different config files (rivers, I guess the terminology?) and have them in separate config files?

If there are some good materials, don’t hesitate to share with me, I did not find answers for these questions.

Thanks in advance,

Patrik

PS: My questions can easily come from the lack of basic understanding how the stack works, in such a case please let me know that too, I won’t take it to the heart :smiley:

2

This is a big question that’s rather difficult to answer. Deploying Alloy is no different than any other CI/CD deployments. I think short answer is “whatever you are currently doing”, because keeping things simple and familiar will be your best bet.

To answer some of your questions more specifically:

I’d imagine this would be preferred on Kubernetes.

This depends. First, I’d try to minimize the number of pipelines. Then, if the products are on the same cluster then I’d recommend putting them into the same config file, so you don’t have to worry about running multiple agents. Alloy can filter and direct logs to different pipelines using stage.match, and I’d recommend considering that before anything else.

1 Like