Dropping Logs Without a Specific Field

Grafana Alloy
1

Hi,

I’ve been trying for two days to get an Alloy config working to drop log lines that do not contain a specific field (duration in my case), as I don’t want to process them.
I’ve tried multiple approaches using json, drop, and match stages, but I haven’t been able to get it working as expected.

I’ve looked at multiple forum posts (can only put 2 links so no more…):

But I still don’t get how to make it work.

Example Log

{"level":"info","ts":"2025-03-20T16:19:07Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"12345678-aaaa-bbbb-cccc-9876543210ef","actual":{"state":"starting","date":"2025-03-20T16:19:02Z"}}
{"level":"info","ts":"2025-03-20T16:19:07Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"12345678-aaaa-bbbb-cccc-9876543210ef","next":"starting"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","actual":{"state":"starting","date":"2025-03-20T16:19:02Z"}}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","next":"initializing"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Updating process state \"initializing\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Previous state was \"starting\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","process_id":"node-x1a2b3c4","service":"node","service_version":"1.2.3-456-untested","graphical":"false","cloud_provider":"aws","region":"us-east-1","resource_type":"m5.large","state":"starting","duration":"6"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Updated Process.Status.LastState to \"&{initializing 2025-03-20 16:19:08 +0000 UTC}\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"55555555-cccc-bbbb-aaaa-999999999999","actual":{"state":"initializing","date":"2025-03-20T16:19:08Z"}}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"55555555-cccc-bbbb-aaaa-999999999999","next":"initializing"}
{"level":"info","ts":"2025-03-20T16:19:07Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"12345678-aaaa-bbbb-cccc-9876543210ef","actual":{"state":"starting","date":"2025-03-20T16:19:02Z"}}
{"level":"info","ts":"2025-03-20T16:19:07Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"12345678-aaaa-bbbb-cccc-9876543210ef","next":"starting"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","actual":{"state":"starting","date":"2025-03-20T16:19:02Z"}}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","next":"initializing"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Updating process state \"initializing\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Previous state was \"starting\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab","process_id":"node-x1a2b3c4","service":"node","service_version":"1.2.3-456-untested","graphical":"false","cloud_provider":"aws","region":"us-east-1","resource_type":"m5.large","state":"starting","duration":"6"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Updated Process.Status.LastState to \"&{initializing 2025-03-20 16:19:08 +0000 UTC}\"","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"87654321-ffff-eeee-dddd-0123456789ab"}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Calculating next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"55555555-cccc-bbbb-aaaa-999999999999","actual":{"state":"initializing","date":"2025-03-20T16:19:08Z"}}
{"level":"info","ts":"2025-03-20T16:19:08Z","msg":"Next process state","controller":"process","controllerGroup":"system.pipeline","controllerKind":"Process","Process":{"name":"node-x1a2b3c4","namespace":"test-env"},"namespace":"test-env","name":"node-x1a2b3c4","reconcileID":"55555555-cccc-bbbb-aaaa-999999999999","next":"initializing"}

I don’t care about lines without the duration field and don’t even want to store them, as they are handled by another loki.process pipeline.

My Current Process

After two days of troubleshooting, here’s the setup I have:

loki.process "task_analysis_json_extraction" {
    stage.json {
        expressions = {
            line = "",
        }
    }
    stage.match {
        selector = "{line!~\".*duration.*\"}"
        action = "drop"
    }

    forward_to = [loki.process.task_analysis_field_extraction.receiver]
}

loki.process "task_analysis_field_extraction" {
    stage.json {
        expressions = {
            level = "level",
            ts = "ts",
            msg = "msg",
            controller = "controller",
            controllerGroup = "controllerGroup",
            controllerKind = "controllerKind",
            Task = "Task",
            namespace = "namespace",
            name = "name",
            reconcileID = "reconcileID",
            task_id = "task_id",
            app = "app",
            app_version = "app_version",
            graphical = "graphical",
            state = "state",
            duration = "duration",
        }
    }

    stage.timestamp {
        source = "ts"
        format = "RFC3339"
    }

    stage.labels {
        values = {
            app_version = "",
            namespace = "",
            app = "",
            graphical = "",
            state = "",
        }
    }

    stage.structured_metadata {
        values = {
            name = "",
            reconcileID = "",
            task_id = "",
            duration = "",
        }
    }

    // Forward to logs service
    forward_to = [loki.write.logs_service.receiver]
}

I split it into two process blocks for debugging purposes.

As line filtering is not available in the match stage, and the drop stage requires an RE2 expression, I can’t use something like this ^(?!.*\bduration\b).*\n?.

Any help would be greatly appreciated!

2

I’ve never tried to chain two loki.process together, I don’t actually know if you can do that. But it’s irrelevant, because you don’t have to do it that way.

Your logic should be:

  1. Json filter, set all labels you want.
  2. stage.match if duration is not set, then drop.
  3. stage.match if duration is set, then set labels and structured metadata.
  4. Also, you don’t need to extract json fields that you are not going to use.

Something like this (not tested):

loki.process "task_analysis_json_extraction" {
    forward_to = [loki.write.logs_service.receiver]

    stage.json {
        expressions = {
            ts = "",
            namespace = "",
            name = "",
            reconcileID = "",
            task_id = "",
            app = "",
            app_version = "",
            graphical = "",
            state = "",
            duration = "",
        }
    }

    stage.match {
        selector = "{duration!~\".+\"}"
        action = "drop"
    }

    stage.match {
        selector = "{duration=~\".+\"}"

        stage.timestamp {
            source = "ts"
            format = "RFC3339"
        }

        stage.labels {
            values = {
                app_version = "",
                namespace = "",
                app = "",
                graphical = "",
                state = "",
            }
        }

        stage.structured_metadata {
            values = {
                name = "",
                reconcileID = "",
                task_id = "",
                duration = "",
            }
        }
    }
}
3

Thank you, @tonyswumac, for your answer. I tried it, but it didn’t work—I had no log lines.

I searched again based on your suggestion and eventually asked Grot for help, which helped me get it working.

Here’s what I ended up with:

loki.process "task_analysis_json_extraction" {

    // Parse the JSON first
    stage.json {
        expressions = {
            app = "",
            app_version = "",
            cloud_provider = "",
            duration = "",
            graphical = "",
            namespace = "",
            reconcile_id = "",
            state = "",
            task_id = "",
            region = "",
            ressource_type = "",
        }
    }

    stage.labels {
        values = {
            has_duration = "duration",
            namespace = "namespace",
            app_version = "app_version",
            graphical = "graphical",
            state = "state",
            app_app = "app",
            cloud_provider = "cloud_provider",
            ressource_type = "ressource_type",
        }
    }

    stage.structured_metadata {
        values = {
          reconcile_id = "reconcile_id",
          task_id = "task_id",
          duration = "duration",
          region = "region",
        }
    }

    // Drop logs where the duration label is empty
    stage.match {
        selector = "{has_duration=\"\"}"
        action = "drop"
        drop_counter_reason = "missing_duration_field"
    }

    stage.label_drop {
        values = ["has_duration", "duration"]
    }

    // For logs where the label is empty, add a default value
    stage.match {
        selector = "{cloud_provider=\"\"}"

        // Inside this stage, we only process logs that matched the selector
        stage.static_labels {
            values = {
                "cloud_provider" = "Not assigned",
            }
        }
    }

    stage.timestamp {
          source = "ts"
          format = "RFC3339"
    }

    forward_to = [loki.write.logs_service.receiver]
}

I had to search a bit more since it wasn’t always working, and I found out that this line was causing issues:

            app = "app",

That’s why I ended up using:

            app_app = "app",

I have no idea why, though—maybe it has something to do with internal Alloy mechanics?

4

I doubt it, it worked for me, and I am not aware of anything weird that is related to app.

One thing to note, is if you already have newer logs in a log stream (logs with the same set of labels) you can’t write older logs anymore, that could be your problem given it was fixed after you change the label thereby putting your logs into a different log stream.

5

@tonyswumac would you explain that component: stage.structured_metadata? Do I understand correctly that it might be used not as label on Loki, but still you can use that metadata to search via query? Could you give any practical example when are you using it?

6

In Loki you don’t want to use label that can potentially have unbounded values (IP is a good example), because that would create many log streams unnecessarily which impacts Loki performance. Structured metadata was created to address that, it looks like labels, but it does not contribute to cardinality in Loki. There are some caveats to using structured metadata, and it doesn’t search like labels do, but it’s still pretty useful. You can read more about it here: What is structured metadata | Grafana Loki documentation

I normally use structured metadata for things I think would be useful as labels but can’t. For example, in our nginx or WAF logs we have a geoip stage, and we might set the country code as label, but set the IP, latitude, and longitude as structured metadata.

1 Like