Dropping Logs Without a Specific Field

Grafana Alloy
3

Thank you, @tonyswumac, for your answer. I tried it, but it didn’t work—I had no log lines.

I searched again based on your suggestion and eventually asked Grot for help, which helped me get it working.

Here’s what I ended up with:

loki.process "task_analysis_json_extraction" {

    // Parse the JSON first
    stage.json {
        expressions = {
            app = "",
            app_version = "",
            cloud_provider = "",
            duration = "",
            graphical = "",
            namespace = "",
            reconcile_id = "",
            state = "",
            task_id = "",
            region = "",
            ressource_type = "",
        }
    }

    stage.labels {
        values = {
            has_duration = "duration",
            namespace = "namespace",
            app_version = "app_version",
            graphical = "graphical",
            state = "state",
            app_app = "app",
            cloud_provider = "cloud_provider",
            ressource_type = "ressource_type",
        }
    }

    stage.structured_metadata {
        values = {
          reconcile_id = "reconcile_id",
          task_id = "task_id",
          duration = "duration",
          region = "region",
        }
    }

    // Drop logs where the duration label is empty
    stage.match {
        selector = "{has_duration=\"\"}"
        action = "drop"
        drop_counter_reason = "missing_duration_field"
    }

    stage.label_drop {
        values = ["has_duration", "duration"]
    }

    // For logs where the label is empty, add a default value
    stage.match {
        selector = "{cloud_provider=\"\"}"

        // Inside this stage, we only process logs that matched the selector
        stage.static_labels {
            values = {
                "cloud_provider" = "Not assigned",
            }
        }
    }

    stage.timestamp {
          source = "ts"
          format = "RFC3339"
    }

    forward_to = [loki.write.logs_service.receiver]
}

I had to search a bit more since it wasn’t always working, and I found out that this line was causing issues:

            app = "app",

That’s why I ended up using:

            app_app = "app",

I have no idea why, though—maybe it has something to do with internal Alloy mechanics?