How to remove items from extracted json map in Grafana Alloy before forwarding to Loki

Grafana Loki
1

I’m using Grafana Alloy to send JSON logs from a Caddy webserver to Loki. I’m able to extract the timestamp with the config below. However, I would like to remove original timestamp ts from the JSON log before sending it to Loki. I couldn’t figure out what function/module was available to do this. Can the JSON log be modified before sending? Thanks.

local.file_match "caddy" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/access.log",
                job         = "caddy",
        }]
}

loki.process "caddy" {
        forward_to = [loki.write.default.receiver]

        stage.json {
                expressions = {
                        level     = "level",
                        timestamp = "ts",
                        status = "status",
                }
        }

        stage.labels {
                values = {
                        level = null,
                        status = null,
                }
        }

        stage.timestamp {
                source = "timestamp"
                format = "Unix"
        }

}

loki.source.file "caddy" {
        targets               = local.file_match.caddy.targets
        forward_to            = [loki.process.caddy.receiver]
}

loki.write "default" {
        endpoint {
                url = "..."
        }
        external_labels = {}
}
1 Like
2

I would want to tell you that this can be done in a stage.json block; those contain a set of JMESPath expressions that create the new JSON which will be sent forward, but my research is showing that JMESPath itself doesn’t yet have a way to surgically remove a key from an object, and so since alloy relies on JMESPath, I think this isn’t doable at this time.

3

Thanks. Taking a look, could it perhaps be done with filter projection and !=? Just a bit unsure how to put this in the config.

1 Like
4

I don’t think that would work because filter projections are removing/filtering entire records, not individual keys. For example here:

image
image1748×752 47.2 KB

It throws out the record where state=stopped, but state=running still occurs in the remaining records (discarding records, not keys).

If you could develop a filter expression using that kind of syntax, you’d end up throwing out everything with a date range in the filter, not removing the date (as was the original Q)