Stage.timestamp with Alloy

Grafana Alloy
1

Hey! I’ve been trying to parse my logs and assign the log timestamp to the Grafana timestamp. Right now, the timestamp reflects when the log is ingested into Loki, but I want to use the actual timestamp from the log itself. I’ve been working with the following Loki process and log structure, and the message extraction is working fine, but I can’t seem to get the timestamp sync to work.

Logs:

{
  "body": {
    "@timestamp": "2025-01-20T19:25:48.893Z",
    "message": "{\"\"}",
    "message_size": 1089,
    "stream": "stdout",
    "tags": [
      ""
    ]
  }
}

Config:

loki.process "process_logs" {
	
	forward_to = [loki.relabel.filter_labels.receiver]

	stage.static_labels {
		values = {
			job = "consumer_prod__paas_prdpg_apps",
		}
	}

	// Process the massive blob of Json for Elastic and 
   //take the useful metadata from it
	stage.json {
		expressions = {
			extracted_log_message    = "body.message",
			extracted_timestamp      = "'body.@timestamp'",
		}
	}

	stage.label_drop {
		values = ["filename"]
	}

	
		source = "extracted_log_message"
	}

	stage.timestamp {
		source = "extracted_timestamp"
		format = "RFC3339"
		

}
2

Try this:

loki.process "process_logs" {
    forward_to = [loki.relabel.filter_labels.receiver]

    stage.static_labels {
        values = {
            job = "consumer_prod__paas_prdpg_apps",
        }
    }

    // Process the massive blob of Json for Elastic and 
   //take the useful metadata from it
    stage.json {
        expressions = {
            extracted_body    = "body",
        }
    }

    stage.json {
        source      = "extracted_body"
        expressions = {
            extracted_log_message    = "message",
            extracted_timestamp      = "@timestamp",
        }
    }

    stage.label_drop {
        values = ["filename"]
    }

    stage.output {
        source = "extracted_log_message"
    }

    stage.timestamp {
        source = "extracted_timestamp"
        format = "RFC3339"
    }
}
3

Still doesn’t work!
The parsing of the message is working correctly and the output is correct, but I can’t seem to get the timestamp part to work.

4

Tested the config, I think the problem is with the @ symbol. I would recommend you to see if you can remove that from your source log. If not, use a replace block to get rid of it.

This is the config I used to test your log:

Sample log (/tmp/test.log):

{"body": {"@timestamp": "2025-01-20T19:25:48.893Z","message": "{\"this is test log\"}","message_size": 1089,"stream": "stdout","tags": [""]}}

Alloy config:

local.file_match "test" {
  path_targets = [{
    __path__ = "/tmp/test.log",
    job      = "test",
  }]
}

loki.source.file "logs" {
  targets    = local.file_match.test.targets
  forward_to = [loki.process.process_logs.receiver]
}

loki.process "process_logs" {
    forward_to = [loki.echo.test.receiver]

    stage.replace {
      expression = `(\@timestamp)`
      replace    = "timestamp"
    }

    stage.static_labels {
      values = {
        job = "consumer_prod__paas_prdpg_apps",
      }
    }

    stage.json {
      expressions = {
        extracted_body = "body",
      }
    }

    stage.json {
      source      = "extracted_body"
      expressions = {
        extracted_log_message = "message",
        extracted_timestamp   = "timestamp",
      }
    }

    # for testing only
    stage.labels {
      values = {
        extracted_timestamp = "",
      }
    }

    stage.label_drop {
      values = ["filename"]
    }

    stage.output {
      source = "extracted_log_message"
    }

    stage.timestamp {
      source = "extracted_timestamp"
      format = "RFC3339"
    }
}

loki.echo "test" { }

Alloy echo output:

level=info component_path=/ component_id=loki.echo.test receiver=loki.echo.test entry="{\"this is test log\"}" labels="{extracted_timestamp=\"2025-01-20T19:25:48.893Z\", job=\"consumer_prod__paas_prdpg_apps\"}"
5

It worked! Thank you for your help!!

6

Hello,
I have the same issue, and can’t get to take the time of timestamp to be the same in grafana timestamp.

Here’s my actual log format,
{“method”:“GET”,“url”:“/auth/logged_in”,“level”:“info”,“message”:“GET /auth/logged_in”,“timestamp”:“2025-02-21T07:16:48.362Z”}
{“statusCode”:200,“level”:“info”,“message”:“Response code: 200 for GET /logged_in”,“timestamp”:“2025-02-21T07:16:48.370Z”}

My config.alloy file:-
local.file_match “log_files” {
path_targets = [{“path” = “/var/log/app-logs//.log”}]
sync_period = “5s”
}

local.file_match “log_gz_files” {
path_targets = [{“path” = “/var/log/app-logs//.log.gz”}]
sync_period = “5s”
}

loki.source.file “file_source” {
targets = local.file_match.log_files.targets
tail_from_end = false
forward_to = [loki.relabel.add_service_name.receiver]
}

loki.source.file “file_gz_source” {
targets = local.file_match.log_gz_files.targets
tail_from_end = false
decompression {
enabled = true
initial_delay = “10s”
format = “gz”
}
forward_to = [loki.relabel.add_service_name.receiver]
}

loki.relabel “add_service_name” {
forward_to = [loki.process.log_processing.receiver]
rule {
source_labels = [“filename”]
regex = “/var/log/app-logs/([^/]+)/.*”
target_label = “service_name”
replacement = “$1”
}
}

loki.process “log_processing” {
stage.json {
expressions = {
extracted_timestamp = “timestamp”,
}
}

stage.timestamp {
source = “extracted_timestamp”
format = “RFC3339”
}
forward_to = [loki.write.loki_receiver.receiver]
}

loki.write “loki_receiver” {
endpoint {
url = “http://loki:3100/loki/api/v1/push”
}
}

Could anyone please check and tell, where I am going wrong…?

7
  1. Please do not hijack thread.
  2. Try bypass relabel (forward to loki.process directly) and see if that works. If that does, use stage.labels under loki.process to manipulate labels. loki.relabel is usually used to filter/enforce labels
8

Hi tony,

I have fixed it, instead I have directly sending it to loki.process instead of going through loki.relabel

Here’s my updated code--------------
local.file_match “log_files” {
path_targets = [{“path” = “/var/log/app-logs//.log”}]
sync_period = “5s”
}

local.file_match “log_gz_files” {
path_targets = [{“path” = “/var/log/app-logs//.log.gz”}]
sync_period = “5s”
}

loki.source.file “file_source” {
targets = local.file_match.log_files.targets
tail_from_end = false
forward_to = [loki.process.log_processing.receiver]
}

loki.source.file “file_gz_source” {
targets = local.file_match.log_gz_files.targets
tail_from_end = false
decompression {
enabled = true
initial_delay = “10s”
format = “gz”
}
forward_to = [loki.process.log_processing.receiver]
}

loki.process “log_processing” {
stage.json {
expressions = { “level” = “level”, “time” = “timestamp”}
}
stage.regex {
expression = “/var/log/app-logs/(?P[^/]+)/.*”
source = “filename”
}
stage.labels {
values = { “loglbl” = “level”, “service_name” = “service” }
}
stage.timestamp {
source = “time”
format = “RFC3339”
action_on_failure = “skip”
}
forward_to = [loki.write.loki_receiver.receiver]
}

loki.write “loki_receiver” {
endpoint {
url = “http://loki:3100/loki/api/v1/push”
}
}

Actually my problem is that I want the time from my log file(timestamp) to be used in loki.
Output in Grafana dashboard, when displaying the logs-----------
2025-02-25 13:20:25.631
{
“message”: “empUUID: null, currentUserMail: project@eflag.ai”,
“level”: “info”,
“timestamp”: “2025-02-21T07:16:48.641+00:00”
}
2025-02-25 13:20:25.631 → time, when I started my grafana/alloy, grafana/loki, grafana/grafana containers.
2025-02-21T07:16:48.641+00:00 → time, when the log was generated.
I have tried all the below different type of time format, yet unsuccessful in capturing the timestamp from my logs and overriding it.

  1. 2025-02-21T07:16:48Z
  2. 2025-02-21T07:16:48.377Z
  3. 2025-02-21T07:16:48.377+00:00

Could you please help me with this…?

9

I tested your configuration and found no issue. One thing to note, I am not sure if it’s because of your copy-pasting or your source files were already like that, the example log messages you posted had windows styled curly double quotes, and it’s not valid JSON.

As mentioned earlier, this topic was already resolved for the original post. If you wish to continue our discussions please create a new topic.

10

Hi, i have a problem with my stage.timestamp in grafana alloy, here is my configuration:

local.file_match "tomcat_logs" {
	path_targets = [{
		__address__ = "ibiatap01",
		__path__    = "/opt/tomcat_a/logs/webapps_blue/analytics/main.log",
		app         = "analytics",
		environment = "blue",
		job         = "tomcat",
	}, 
	{
		__address__ = "ibiatap01",
		__path__    = "/opt/tomcat_a/logs/webapps_green/analytics/main.log",
		app         = "analytics",
		environment = "green",
		job         = "tomcat",
	}]
}

loki.process "tomcat_logs" {	
	stage.regex {
		expression = "\\[(?P<tenant>[^\\]]*)\\]\\*? (?P<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}); (?P<thread>[^;]+); \\[(?P<level>[^\\]]+)\\]; (?P<component>[^;]+); (?P<message>.+)"
	}

	stage.multiline {
		firstline = "\\[(?P<tenant>[^\\]]*)\\]\\*? (?P<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}); (?P<thread>[^;]+); \\[(?P<level>[^\\]]+)\\]; (?P<component>[^;]+); (?P<message>.+)"
	}

	stage.labels {
		values = {
			tenant = "tenant",
			thread = "thread",
			level = "level",
			component = "component",
			time = "time",
		}
	}

	stage.timestamp {
		source = "time"
		format = "2006-01-02 15:04:05,999"
	}

	forward_to = [loki.write.default.receiver]
}

loki.source.file "tomcat_logs" {
	targets = local.file_match.tomcat_logs.targets
	forward_to = [loki.process.tomcat_logs.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://10.221.9.61:3100/loki/api/v1/push"
	}
	external_labels = {}
}

livedebugging {
	enabled = true
}

Everything works correctly if i don’t add the stage.timestamp setted. when i set it, is like alloy is not able to detect the format of the rows of the log. I’m sure that the regex is correct because i checked with regex101: build, test, and debug regex and i douboled all the \ like wrote in the documentation of alloy. My logs file have this kink of formati, i show following some lines:

[tenant.test]* 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page

And there are also line that should be catched as multiline that are like the log line before but after that a stack trace that don’t match the regex.

I really need and help to undestrand what is the problem of the stage.timestamp because my goal is to map the time extracted from the log line to the time of the log line in grafana e loki

11

Please open a new thread, and include a example of your logs.

12

I did that as required, you can find the thread at this link: Grafana Alloy: stage.timestamp Fails to Parse Tomcat Logs with Millisecond Comma Format
thank you so much.