Transform JSON Windows Event Logs with Alloy

TL;DR

I am trying to transform windows event logs in Alloy before writing the data to Loki in Grafana Cloud. The idea being to minimize the data sent and to provide the data in a format that is easy to work with in Grafana. I’m trying to drop unused data fields and extract data fields from nested objects.

Context

We have a Windows Server tool by SanDisk called GRAID that we use to setup RAID for NVMe drives. This tool doesn’t offer much in the form of metrics / logging. It does have a CLI, so what I’ve done is written a PowerShell script that logs the output of the CLI commands to the Windows Event log. I run the PowerShell scripts every 5 mins using Windows Task Scheduler to create a log.

In Windows Event Viewer, the data looks like this:
image

Alloy Pipeline

If the Windows Event data is uploaded to Loki without any processing, it looks like this in Grafana Cloud:

So given that, to transform the “raw” data, I implemented a loki.process block as follows:

loki.process "graid_windows_logs_json_processor_1" {
  forward_to = [loki.relabel.graid_windows_log_files_label_cleaner.receiver]

  // stage.replace {
  //   expression = "<Data>|<\/Data>"
  //   replace    = ""
  // }
  stage.json {
    expressions = {
      log_source        = "source",
      graid_event_type  = "event_id",
      timestamp         = "timeCreated",
      message           = "event_data",
    }
  }
  stage.labels {
    values = {
      log_source        = "log_source",
      graid_event_type  = "graid_event_type",
    }
  }
  // stage.json{
  //   expressions = {
  //     result.message = "message",
  //     result.graid_event_type = "graid_event_type",
  //     result.log_source = "log_source",
  //     result.timestamp = "timestamp",
  //   }
  // }
  // stage.output {
  //   source = "result"
  // }
}

The parts that don’t work are commented out and are explained below:

Issue 1: Removing XML Data Tags

The first stage is there because I’m trying to remove the <Data> and </Data> tags from the "event_data" JSON field. I thought to try a stage.replace.

This doesn’t seem to be working. It could have to do with how regex doesn’t work very well with XML. It could also just be incorrect syntax / implementation on my side.

Issue 2: Transforming the Output JSON Object

You’ll note that the stages in the loki.process block that are uncommented are extracting data into labels. So, the final two stages (stage.json and stage.output) are to try transform the object that I send to Grafana Cloud into a final format like this:

{
   "timestamp": "2024-07-26T09:55:01.8698452Z",
   "errorCode": 0, //from event_data
   "errorMessage": "" //from event_data
   "result": [ {...}, {...} ] //from event_data
}

I thought to try map the data using the JMESPath syntax, but I think I’m getting it wrong. Also, if the stage.output could accept a map(string) it would make doing this a bit easier because then I’d just pass the new object fields in directly.


The lessons learned here can definitely be applied to any JSON object transformation with Alloy. Any help would be greatly appreciated!

why not keep it simple and write it to disk as log files in the exact format you want to in the first place?

Good question!

So originally, I was writing to a .json file on disk, but then I found that:

  • For production I probably needed to ensure that the file doesn’t get too big and possibly write a clean-up script that runs periodically.
  • I also encountered issues where the PowerShell script would occasionally not release the file correctly after writing, so when the script ran again the .json file would be locked for writing.

So, then I decided to just make use of the standard windows event log because then it’s “not my problem” to think of these standard logging related tasks.

1 Like

I would go with the following approach. run the task every 5 minutes but generates a new file then create another task that cleans up at the end of the day or every 4 hours. dont use same file which bloats

in fact forget about exporting to file, write to loki directly here are some options

this one should help you sort things out just implement it in powershell

Fair, but this would imply that we need to install the Python runtime on this server. We are trying to minimize the apps running on this server as its main purpose is infrastructure related (i.e. GRAID).

Is this sort of JSON object transformation not something that Alloy supports or supports efficiently at this stage?

so, the code sample is python, but as stated above implement it as powershell

I do some processing of the event data on specific windows events to extract the IIS app pool. Maybe this will inspire you. The regex came from promtail where it worked but it isn’t working right now in Alloy, trying to sort that out, but that’s not really the point.

loki.process “extract_app_pool” {
stage.regex {
source = “event_data”
expression = "<Data Name='AppPoolID'>(?P<pool>.+?)</Data>"
}
stage.labels {
values = {
pool = “”,
}
}
forward_to = [ loki.relabel.all.receiver ]
}

1 Like

Hmm. The expression regex didn’t come through, probably because of the XML. Will try to edit that but <Data Name='AppPoolID'>(?P<pool>.+?)</Data>