Grafana Alloy: stage.timestamp Fails to Parse Tomcat Logs with Millisecond Comma Format

[atabonidev]

Hi, i have a problem with my stage.timestamp in grafana alloy, here is my configuration:

local.file_match "tomcat_logs" {
	path_targets = [{
		__address__ = "ibiatap01",
		__path__    = "/opt/tomcat_a/logs/webapps_blue/analytics/main.log",
		app         = "analytics",
		environment = "blue",
		job         = "tomcat",
	}, 
	{
		__address__ = "ibiatap01",
		__path__    = "/opt/tomcat_a/logs/webapps_green/analytics/main.log",
		app         = "analytics",
		environment = "green",
		job         = "tomcat",
	}]
}

loki.process "tomcat_logs" {	
	stage.regex {
		expression = "\\[(?P<tenant>[^\\]]*)\\]\\*? (?P<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}); (?P<thread>[^;]+); \\[(?P<level>[^\\]]+)\\]; (?P<component>[^;]+); (?P<message>.+)"
	}

	stage.multiline {
		firstline = "\\[(?P<tenant>[^\\]]*)\\]\\*? (?P<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}); (?P<thread>[^;]+); \\[(?P<level>[^\\]]+)\\]; (?P<component>[^;]+); (?P<message>.+)"
	}

	stage.labels {
		values = {
			tenant = "tenant",
			thread = "thread",
			level = "level",
			component = "component",
			time = "time",
		}
	}

	stage.timestamp {
		source = "time"
		format = "2006-01-02 15:04:05,999"
	}

	forward_to = [loki.write.default.receiver]
}

loki.source.file "tomcat_logs" {
	targets = local.file_match.tomcat_logs.targets
	forward_to = [loki.process.tomcat_logs.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://10.221.9.61:3100/loki/api/v1/push"
	}
	external_labels = {}
}

livedebugging {
	enabled = true
}

Everything works correctly if i don’t add the stage.timestamp setted. when i set it, is like alloy is not able to detect the format of the rows of the log. I’m sure that the regex is correct because i checked with regex101: build, test, and debug regex and i douboled all the \ like wrote in the documentation of alloy. Here following there are some example of logs that i want to detect:

• log with tenant defined single line

[tenant.test]* 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page

• Log with tenant not defined:

[]* 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page

• Log without the ‘*’

[tenant.test] 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page

• Log multiline with stack trace, that need to be detected as 1 single log (the first line could match one of the pattern shared before):

[]* 2025-03-19 06:00:08,767; Timer-0; [WARN]; MainLogger; Generic error occurred at 06:00:08 303ms Wednesday 19 March originating thread process_24.0xxx for user CPContext of user:5.0 guest:false company:xxx username to be determined  instance: pippo.test  last accessed:Wed Mar 19 06:00:08 CET 2025 with stacktrace:java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at iacl_fspace_statsR.Page_4(iacl_fspace_statsR.java:475)
	at iacl_fspace_statsR.Page_1(iacl_fspace_statsR.java:364)
	at iacl_fspace_statsR.Run(iacl_fspace_statsR.java:520)
	at iacl_fspace_statsR.Run(iacl_fspace_statsR.java:505)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

And there are also line that should be catched as multiline that are like the log line before but after that a stack trace that don’t match the regex.

I really need and help to undestrand what is the problem of the stage.timestamp because my goal is to map the time extracted from the log line to the time of the log line in grafana e loki

[tonyswumac]

You can use the backtick character ` to enclose your regex statement and avoid having to double escape special characters. This is the regex I have:

expression = `^\[(?P<tenant>[^\]]*)\]\**\W(?P<time>\d{4}-\d{2}-\d{2}\W\d{2}:\d{2}:\d{2},\d{3});\W(?P<thread>[^;]+);\W\[(?P<level>[^\]]+)\];\W(?P<component>[^;]+);\W(?P<message>.+)`

Give that a try, and see if it works. If not I can test your configuration later.

Also a sidenote, since your line starts with [ you can simplify your multiline regex to

firstline = `^\[.+`

[atabonidev]

But the regex like i wrote was working fine, the problem is that don’t work the stage.timestamp that don’t work correctly

[tonyswumac]

I tested your logs and configuration, this worked for me:

alloy process stage:

  stage.multiline {
    firstline = `^\[.+`
  }

  stage.regex {
    expression = `^\[(?P<tenant>[^\]]*)\]\**\W(?P<time>\d{4}-\d{2}-\d{2}\W\d{2}:\d{2}:\d{2},\d{3});\W(?P<thread>[^;]+);\W\[(?P<level>[^\]]+)\];\W(?P<component>[^;]+);\W(?P<message>.+)`
  }

  stage.labels {
    values = {
      tenant = "tenant",
      thread = "thread",
      level = "level",
      component = "component",
      time = "time",
    }
  }

  stage.timestamp {
    source = "time"
    format = "2006-01-02 15:04:05,999"
  }

test.log:

[iablue.zucchettianalyticslab.cloud]* 2025-02-26 01:37:37,612; MantenanceTimer-0; [DEBUG]; SSOManager; pulizia della mappa dei token gia' utilizzati
[tenant.test]* 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page
[]* 2025-03-18 08:44:49,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page
[tenant.test] 2025-03-18 08:44:50,192; ajp-nio-0.0.0.0-8010-exec-2; [ERROR]; HeadTag; com.agency.IA.dispatchers.actiondescriptor.internal.IASessionExpiredException: IASessionExpiredException - Instance not found, need redirect to login page
[]* 2025-03-19 06:00:08,767; Timer-0; [WARN]; MainLogger; Generic error occurred at 06:00:08 303ms Wednesday 19 March originating thread process_24.0xxx for user CPContext of user:5.0 guest:false company:xxx username to be determined  instance: pippo.test  last accessed:Wed Mar 19 06:00:08 CET 2025 with stacktrace:java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at iacl_fspace_statsR.Page_4(iacl_fspace_statsR.java:475)
    at iacl_fspace_statsR.Page_1(iacl_fspace_statsR.java:364)
    at iacl_fspace_statsR.Run(iacl_fspace_statsR.java:520)
    at iacl_fspace_statsR.Run(iacl_fspace_statsR.java:505)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

[atabonidev]

Yes but from the documentation of stage.timestamp:

The stage.timestamp inner block configures a processing stage that sets the timestamp of log entries before they’re forwarded to the next component. When no timestamp stage is set, the log entry timestamp defaults to the time when the log entry was scraped.

But in your screenshot the two dates are not the same as i show you in the following image:

[tonyswumac]

The timestamp on the left is the timestamp in Loki, and visualized to my local time (PST). The one on the right is the timestamp in the logs (part of the log).

They are the same, if you take time zone into consideration.

[atabonidev]

Perfect it work, thank you for the explanation. I can asj to you if you can help me also in this grafana alerting task?

How to put an alert rule dependent to a dashboard variable?

Hi, i have a variable in one of my Grafana’s dashboards and i have a row of panels that repeat itself with all the panel inside for every value of the dashboard variable. In every of these rows i have a panel that show the number of seconds passed from the last ping made by an agent installed in the client machine to my application. The problem is that the SQL code to retrive this information is made like that: SELECT now() AS time, EXTRACT(EPOCH FROM now() - lastsuccessfullogin AT TIME ZON…

[atabonidev]

Perfect it work, thank you very much for the explanation. I can ask to you if you can help me also in this grafana alerting task?

How to put an alert rule dependent to a dashboard variable?

Hi, i have a variable in one of my Grafana’s dashboards and i have a row of panels that repeat itself with all the panel inside for every value of the dashboard variable. In every of these rows i have a panel that show the number of seconds passed from the last ping made by an agent installed in the client machine to my application. The problem is that the SQL code to retrive this information is made like that: SELECT now() AS time, EXTRACT(EPOCH FROM now() - lastsuccessfullogin AT TIME ZON…

[yosiasz]

Probably best to start a separate thread