Alloy log processing

Grafana Alloy
1

Hi all,

Migrating from elastic to Loki and have Loki running and Alloy installed on a couple of machines. Was about to start writing some log processing expressions for common things like linux system logs, apache, nginx, iptables… and thought rather than do this myself there must be some sort of central repo for this.

Is there?

Cheers

Trent

2

I don’t think there is one, but I suspect you can find some examples online pretty easily.

Grafana Loki is fundamentally different from ElasticSearch. You do not need to excessively parse your logs during ingestion. I’d argue most of the time you’ll probably want to parse just the timestamp so you can have accurate timestamp on your logs, keep the log lines intact, and leave the rest of processing to queries. Therefore most of the log pipeline should be pretty easy to write. For example this is what we have for our Nginx instances (this is using Grafana Agent, actually, the predecessor of Grafana Alloy, but the configuration is pretty close):

loki.source.file "nginx_access_logs" {
  targets    = [{
    __path__  = "/var/log/nginx/access.log",
    job       = "aws/ec2/grafana-agent/nginx/access",
  }]
  forward_to    = [loki.process.access_logs.receiver]
}

loki.process "access_logs" {
  forward_to  = [loki.write.local.receiver]
  stage.regex {
    expression  = "^(?P<remote_ip>[^ ]+) - (?:\\S+) \\[(?P<timestamp>.*)\\] (?:.*)"
  }
  
  stage.timestamp {
    source  = "timestamp"
    format  = "02/Jan/2006:15:04:05 +0000"
  }
}

loki.source.file "nginx_error_logs" {
  targets   = [{
    __path__  = "/var/log/nginx/error.log",
    job       = "aws/ec2/grafana-agent/nginx/error",
  }]
  forward_to    = [loki.process.error_logs.receiver]
}

loki.process "error_logs" {
  forward_to  = [loki.write.local.receiver]
  stage.regex {
    expression  = "^(?P<timestamp>\\S* \\S*) .+client: (?P<remote_ip>\\S+), (?:.*)"
  }

  stage.timestamp {
    source  = "timestamp"
    format  = "02/Jan/2006:15:04:05 +0000"
  }
}

loki.source.file "nginx_stream_logs" {
  targets    = [{
    __path__  = "/var/log/nginx/stream.log",
    job       = "aws/ec2/grafana-agent/nginx/stream",
  }]
  forward_to = [loki.process.stream_logs.receiver]
}

loki.process "stream_logs" {
  forward_to = [loki.write.local.receiver]
	stage.regex {
    expression = "^(?P<remote_ip>[^ ]+) \\[(?P<timestamp>.*)\\] (?:.*)"
  }

  stage.timestamp {
    source  = "timestamp"
    format  = "02/Jan/2006:15:04:05 +0000"
  }
}

loki.source.file "nginx_stream_error_logs" {
  targets    = [{
    __path__  = "/var/log/nginx/stream-error.log",
    job       = "aws/ec2/grafana-agent/nginx/stream-error",
  }]
  forward_to = [loki.process.stream_error_logs.receiver]
}

loki.process "stream_error_logs" {
  forward_to  = [loki.write.local.receiver]
	stage.regex {
    expression  = "^(?P<timestamp>\\S* \\S*) .+client: (?P<remote_ip>\\S+), (?:.*)"
  }

  stage.timestamp {
    source  = "timestamp"
    format  = "02/Jan/2006:15:04:05 +0000"
  }
}

loki.write "local" {
  endpoint {
    url = "{{ loki_url }}"
  }
  external_labels   = {
    aws_account_alias   = "{{ aws_account_alias }}",
    aws_account_id      = "{{ aws_account_id }}",
    aws_region          = "{{ aws_region }}",
    aws_ec2_instance_id = "{{ ansible_ec2_instance_id }}",
    environment         = "{{ aws_environment }}",
    app                 = "nginx/{{ nginx_type }}",
  }
}
3

Thanks very much for the response and example @tonyswumac

4

is the loki_url the push api for example
http://loki:3100/loki/api/v1/push

Thanks

5

It is, I copy-pasted from our Ansible playbook so it’s kinda hidden.

1 Like