How to get alloy to parse /var/log/syslog correctly?

[majerus]

I have alloy configured to gather all files from /var/log/*.log, as well as /var/log/syslog. It is forwarding to loki. Within loki, only certian logs have timestamp, and other fields identified. How do i get alloy or maybe loki to see the timestamps and other data as it should. Thanks

This is from /var/log/auth.log

2024-11-07T13:45:01.205837-06:00 ebpf CRON[1891924]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)

This is from /var/log/syslog
2024-11-07T13:54:06.455386-06:00 ebpf alloy[1870079]: ts=2024-11-07T19:54:06.454532494Z level=debug msg="collector succeeded" component_path=/ component_id=prometheus.exporter.unix.default name=cpu duration_seconds=0.001657023

It seems to me that only logs that show ts= have a time stamp, even though the /var/log/syslog , and /var/log/auth.log also have their own timestamps in the actual logs.

Here is my config.alloy file.

logging{
    level = "debug"
    format = "logfmt"
    write_to = [loki.process.alloy.receiver]
}

loki.process "alloy" {
    stage.labels {
        values = {
            service_name = "alloy",
        }
    }
    forward_to = [loki.write.default.receiver]
}

local.file_match "syslog" {
    path_targets = [
    {__path__ = "/var/log/*.log", 
    },
    {__path__ = "/var/log/syslog", 
    },


    ]
}

loki.process "syslog" {

    stage.static_labels {
    values = {
      instance = constants.hostname,
      service = "linux_logs",
    }
    }
    forward_to = [loki.write.default.receiver]
}

loki.source.file "syslog" {
    targets = local.file_match.syslog.targets
    forward_to = [loki.process.syslog.receiver]
}


loki.write "default" {
    endpoint {
        url = "http://loki.domain.com/loki/api/v1/push"
        tenant_id = 1
    }
}

Log missing fields I would anticipate should be there…

Logs with fields as i expect…

Another example of missing fields:

[tonyswumac]

All logs sent to Loki have timestamps. I am not sure what version of Grafana you are using, but in Grafana 11 there is an option to show or hide timestamp for Loki logs, and if toggled you’d see timestamp in front of logs:

In older version of Grafana I think ts was disabled as a label, but in epoch format (down to nanoseconds if i remember correctly). I am not sure why your display seems to be inconsistent.

[majerus]

I am running 11.1 or 11.3 have multiple versions running. If you expand that series you have listed, what options do you see?

[majerus]

Time not toggled…

Time toggled…

If i go to another log line…

[yosiasz]

hmm seeing the same thing locally also. not sure why, but maybe apply pattern on the none ts one?

[majerus]

I am new to loki, so bare with me here. It appears that by doing the pattern bit I see the fields. However it also axed the fields from the other logs that had more fields before.

Added Fields

Missing Fields now…

Query:
{service_name=linux_logs} | pattern | drop __error__, __error_details__

[yosiasz]

Instead of service name I do it by file name

So either do the labeling explicitly on ingest in alloy or do separate file name

[majerus]

Have a config you can share? Honestly this is my first go at a config . I would like it to actually be for each file is the label.

Provide recommended config.alloy files for Linux, Windows log and Metrics collection.

[majerus]

This is more like what I would want to see from linux, however not sure how to pull it off. Doesnt need all of these for sure. However Time, Instance, Level, etc would be super nice.

Here is what I have for windows, I borrowed a lot of it from sources I found

prometheus.exporter.self "integrations_alloy" { }

discovery.relabel "integrations_alloy" {
  targets = prometheus.exporter.self.integrations_alloy.targets

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }

  rule {
    target_label = "alloy_hostname"
    replacement  = constants.hostname
  }

  rule {
    target_label = "job"
    replacement  = "integrations/alloy-check"
  }
}

prometheus.scrape "integrations_alloy" {
  targets    = discovery.relabel.integrations_alloy.output
  forward_to = [prometheus.relabel.integrations_alloy.receiver]  

  scrape_interval = "60s"
}

prometheus.relabel "integrations_alloy" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "(prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|alloy_build.*|prometheus_remote_write_wal_samples_appended_total|process_start_time_seconds)"
    action        = "keep"
  }
}

prometheus.exporter.windows "integrations_windows_exporter" {
  enabled_collectors = ["cpu", "cs", "logical_disk", "net", "os", "service", "system", "time", "diskdrive"]
}
discovery.relabel "integrations_windows_exporter" {
  targets = prometheus.exporter.windows.integrations_windows_exporter.targets

  rule {
    target_label = "job"
    replacement  = "integrations/windows_exporter"
  }

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }
}
prometheus.scrape "integrations_windows_exporter" {
  targets    = discovery.relabel.integrations_windows_exporter.output
  forward_to = [prometheus.relabel.integrations_windows_exporter.receiver]
  job_name   = "integrations/windows_exporter"
}
prometheus.relabel "integrations_windows_exporter" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    action        = "keep"
    regex         = "(up|windows_cpu_interrupts_total|windows_cpu_time_total|windows_cs_hostname|windows_cs_logical_processors|windows_cs_physical_memory_bytes|windows_disk_drive_status|windows_logical_disk_avg_read_requests_queued|windows_logical_disk_avg_write_requests_queued|windows_logical_disk_free_bytes|windows_logical_disk_idle_seconds_total|windows_logical_disk_read_bytes_total|windows_logical_disk_read_seconds_total|windows_logical_disk_reads_total|windows_logical_disk_size_bytes|windows_logical_disk_write_bytes_total|windows_logical_disk_write_seconds_total|windows_logical_disk_writes_total|windows_net_bytes_received_total|windows_net_current_bandwidth_bytes|windows_net_bytes_total|windows_net_bytes_sent_total|windows_net_packets_outbound_discarded_total|windows_net_packets_outbound_errors_total|windows_net_packets_received_discarded_total|windows_net_packets_received_errors_total|windows_net_packets_received_unknown_total|windows_os_info|windows_os_paging_limit_bytes|windows_os_physical_memory_free_bytes|windows_os_timezone|windows_service_status|windows_service_state|windows_system_context_switches_total|windows_system_processor_queue_length|windows_system_system_up_time|windows_time_computed_time_offset_seconds|windows_time_ntp_round_trip_delay_seconds|windows_process_info|windows_system_threads|windows_system_exception_dispatches_total)"
    source_labels = ["__name__"]
  }
}

loki.process "logs_integrations_windows_exporter_application" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]

  stage.json {
    expressions = {
      level  = "levelText",
      source = "source",
    }
  }

  stage.labels {
    values = {
      level  = "",
      source = "",
    }
  }
}
loki.relabel "logs_integrations_windows_exporter_application" {
  forward_to = [loki.process.logs_integrations_windows_exporter_application.receiver]

  rule {
    source_labels = ["computer"]
    target_label  = "agent_hostname"
  }
}
loki.source.windowsevent "logs_integrations_windows_exporter_application" {
  locale                 = 1033
  eventlog_name          = "Application"
  bookmark_path          = "./bookmarks-app.xml"
  poll_interval          = "0s"
  use_incoming_timestamp = true
  forward_to             = [loki.relabel.logs_integrations_windows_exporter_application.receiver]
  labels                 = {
    instance = constants.hostname,
    job      = "integrations/windows_exporter",
  }
}
loki.process "logs_integrations_windows_exporter_system" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]

  stage.json {
    expressions = {
      level  = "levelText",
      source = "source",
    }
  }

  stage.labels {
    values = {
      level  = "",
      source = "",
    }
  }
}
loki.relabel "logs_integrations_windows_exporter_system" {
  forward_to = [loki.process.logs_integrations_windows_exporter_system.receiver]

  rule {
    source_labels = ["computer"]
    target_label  = "agent_hostname"
  }
}
loki.source.windowsevent "logs_integrations_windows_exporter_system" {
  locale                 = 1033
  eventlog_name          = "System"
  bookmark_path          = "./bookmarks-sys.xml"
  poll_interval          = "0s"
  use_incoming_timestamp = true
  forward_to             = [loki.relabel.logs_integrations_windows_exporter_system.receiver]
  labels                 = {
    instance = constants.hostname,
    job      = "integrations/windows_exporter",
  }
}



prometheus.remote_write "metrics_service" {
  endpoint {
    url = "http://prom.domain.com/api/v1/write"

  }
}

loki.write "grafana_cloud_loki" {
  endpoint {
    url = "http://chipper.domain.com/loki/api/v1/push"
    tenant_id = 1
  }
}

pyroscope.scrape "default" {
  targets = [
    {"__address__" = "alloy:12345", "service_name"="Alloy"},
  ]
  forward_to = [pyroscope.write.staging.receiver]
}

[tonyswumac]

I think you may be misunderstanding something, I’ll see if I can try answer your questions the best I could understand them:

1. As mentioned before, all logs sent to Loki have a timestamp. The timestamp is NOT part of the log string, it is NOT a label, and you should not want to see it as a label (because it’s not a label). As my reply above showed, if you toggle Time you’ll see timestamps in front of your logs, otherwise you don’t. Anything else you see is part of the log string, and whatever time you have in the log string may or may not match the actual timestamps of the logs.

Typically you’d parse timestamp out of logs then specifically set it as timestamp (not label). Here is an example:

loki.process "stream_logs" {
  forward_to = [loki.write.loki_receiver.receiver]
  
  stage.regex {
    // example log: 123.123.123.123 [05/Nov/2024:19:10:37 +0000] TCP 200 0 0 0.001 123.123.123.123:443 "0" "0" "0.001"
    expression = "^(?P<remote_ip>[^ ]+) \\[(?P<timestamp>.*)\\] (?:.*)"
  }

  stage.timestamp {
    source = "timestamp"
    format = "02/Jan/2006:15:04:05 +0000"
  }
}

Again, you will not see timestamp as a label, because it is not a label.

2. There are things you should not make into labels. See Label best practices | Grafana Loki documentation.

[majerus]

@tonyswumac Thanks for the explanation.

To make sure I understand correctly your saying by default loki is going to set a time stamp on the logs it ingests. Which is what I am seeing outlined in red on the screenshot below, when “time” is toggled on?

What I should be doing to make the loki time stamp more accurate is to map the logs defined time stamp as the timestamp loki will use when its processing the log?

Since I have no mapping of the log line timestamp and loki’s handling of it is that why I see a difference in time between the box in red, and the one in yellow? Is this also why i see two timestamps as well? (One is the “loki” timestamp, and the other is just part of the log string)

The last point is I dont want to have time as a field because because that will be a very high cardinality log, and generally the idea with loki is to have a small index as a best practice?

Dont want to make this to long, but with all that said. Have a few more questions about these pipelines which is where I think am most confused and am still missing the boat a little bit.

For a log such as the one below , would appropriate labels be the following? Instance = ebpf(this is the computers hostname), Service = CRON, LogFile=/var/log/auth.log and that’s about it? Just labels that define where it came from not the log data itself?

If so how would that be defined to pick up the labels correctly?
2024-11-07T21:17:01.772513-06:00 ebpf CRON[106079]: pam_unix(cron:session): session closed for user root

Thanks again!

Red / Yellow Image

[majerus]

One other point, in this example we can see a bunch of fields in grafana. However if I use the label browser i see that there is only a few labels , and most do not coorespond to the data being extracted. With that said where does that leave things?

Example with fields

Label Browser

[yosiasz]

@majerus

and most do not coorespond to the data being extracted.

that could be from some other stuff in Loki unrelated to the files you scrapped. this is why before you dive head on, best to vet things out via loki.echo

the way you are going about with this seems a bit shooting in the dark and hope it sticks approach, which is exactly what I did and dug myself into a very dark rabbit hole.

vet things out with loki.echo first.

[majerus]

Honestly unfamiliar with loki.echo ill look it up. Still think the questions are valid, and trying to grasp this.

[yosiasz]

so many questions and so many images it is very hard to parse things out. let’s take one item at a time. this should be doable

please keep things succinct and list out your questions one at a time.

[tonyswumac]

@majerus

One other point, in this example we can see a bunch of fields in grafana. However if I use the label browser i see that there is only a few labels , and most do not coorespond to the data being extracted. With that said where does that leave things?

There are two kinds of labels in Loki, the ones that came with logs when being ingester (this is the ones you see in label browser). The rest of what you show in the screenshot are processed labels, that come from filter such as json.

[clementduveau2]

That’s a long discussion, so I am just going to look at your first message.

Some basics first:

• Unless you instruct Alloy to parse content, it just takes the log line, put a timestamp when grabbed and push it to Loki with “zero” knowledge about the content
• Loki doesn’t parse stuff either on ingestion. You can store log lines with zero knowledge about what’s inside
• Content is parsed at query time (using pattern or logfmt or json, etc)
• The typical recommendation is to add labels at Alloy level about what’s AROUND and not INSIDE the logs (ex: hostname=my-server, logfile=/var/log/apache.log, env=prod, job=system)

After that said, your usecase is not “weird”: your logline embed a timestamp that can be different from the time the logline is written or grabbed.

First of all: does it matter to correct your timestamp ? Order is preserved, data is correct, only problem you have is that Loki thinks it happened 0.3s later than normally. If it doesn’t matter, just give up on that. You would add a lot of config for no real value.

If you still want to proceed, you have 2 different formats in your logs (one with ts, one without) and you expected 2 differents results. You need to create 2 pipelines in Alloy (you can check localhost:12345 to see your current graph). You can find some knowledge about timestamp replacement here (it uses promtail but that the same concept using Alloy semantic)