Hello,
I am facing an issue using promtail to parse the stout of a k8s pod. The logs are in json format and everything works fine until the the stacktrace is too large:
Scrape config:
- job_name: custom-config
pipeline_stages:
- cri: {}
- json:
expressions:
timestamp: timestamp
level: level
service: service
traceId: traceId
spanId: spanId
thread: thread
class: logger
message: message
context: context
stack_trace: stack_trace
- labels:
timestamp:
level:
service:
traceId:
spanId:
thread:
class:
message:
context:
stack_trace:
- timestamp:
format: RFC3339
source: timestamp
- output:
source: message
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_controller_name
regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
action: replace
target_label: __tmp_controller_name
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- __tmp_controller_name
- __meta_kubernetes_pod_name
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: app
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_pod_label_release
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
- __meta_kubernetes_pod_label_component
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: $1
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__
Log message
{
"timestamp": "2024-01-10T11:04:11.074",
"level": "ERROR",
"service": "profile-service",
"traceId": "659e79aacc9b494a407b55d29315e273",
"spanId": "7e54469782462843",
"thread": "http-nio-8080-exec-10",
"class": "c.b.p.exception.CustomExceptionHandler",
"message": "Internal server error response generated",
"stack_trace": "\ncom.test.profileservice.exception.FirebaseUserException: Deleting user failed\n at com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:19)\n at com.test.profileservice.service.ProfileServiceImpl.deleteUserProfile(ProfileServiceImpl.java:116)\n at com.test.profileservice.controller.ProfileController.deleteUserProfile(ProfileController.java:275)\n at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)\n at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.base/java.lang.reflect.Method.invoke(Method.java:568)\n at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)\n at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)\n at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)\n at com.test.profileservice.controller.ProfileController$$SpringCGLIB$$0.deleteUserProfile(<generated>)\n at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.base/java.lang.reflect.Method.invoke(Method.java:568)\n at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207)\n at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152)\n at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)\n at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)\n at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)\n at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:936)\n at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:596)\n at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)\n at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)\n at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)\n at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)\n at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)\n at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)\n at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)\n at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)\n at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)\n at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)\n at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)\n at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)\n at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)\n at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)\n at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)\n at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)\n at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)\n at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)\n at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)\n at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:738)\n at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)\n at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)\n at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)\n at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)\n at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)\n at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)\n at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n at java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.google.firebase.auth.FirebaseAuthException: No user record found for the given identifier (USER_NOT_FOUND).\n at com.google.firebase.auth.internal.AuthErrorHandler.createException(AuthErrorHandler.java:119)\n at com.google.firebase.auth.internal.AuthErrorHandler.createException(AuthErrorHandler.java:35)\n at com.google.firebase.internal.AbstractHttpErrorHandler.handleHttpResponseException(AbstractHttpErrorHandler.java:57)\n at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:108)\n at com.google.firebase.auth.internal.AuthHttpClient.sendRequest(AuthHttpClient.java:75)\n at com.google.firebase.auth.internal.AuthHttpClient.sendRequest(AuthHttpClient.java:69)\n at com.google.firebase.auth.FirebaseUserManager.post(FirebaseUserManager.java:351)\n at com.google.firebase.auth.FirebaseUserManager.deleteUser(FirebaseUserManager.java:179)\n at com.google.firebase.auth.AbstractFirebaseAuth$14.execute(AbstractFirebaseAuth.java:852)\n at com.google.firebase.auth.AbstractFirebaseAuth$14.execute(AbstractFirebaseAuth.java:849)\n at com.google.firebase.internal.CallableOperation.call(CallableOperation.java:36)\n at com.google.firebase.auth.AbstractFirebaseAuth.deleteUser(AbstractFirebaseAuth.java:830)\n at com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:17)\n ... 138 common frames omitted\nCaused by: com.google.api.client.http.HttpResponseException: 400 Bad Request\nPOST https://identitytoolkit.googleapis.com/v1/projects/youbastard/accounts:delete\n{\n \"error\": {\n \"code\": 400,\n \"message\": \"USER_NOT_FOUND\",\n \"errors\": [\n {\n \"message\": \"USER_NOT_FOUND\",\n \"domain\": \"global\",\n \"reason\": \"invalid\"\n }\n ]\n }\n}\n at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)\n at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)\n at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:96)\n ... 147 common frames omitted\n "
}
Error message in promtail
level=error ts=2024-01-10T11:04:11.128441781Z caller=client.go:430 component=client host=loki-stack:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): stream '{app=\"profile-service\", container=\"profile-service\", filename=\"/var/log/pods/default_profile-service-579b4765c7-4xw84_6b09d696-71da-4dd7-8e37-743933056ac7/profile-service/0.log\", instance=\"profile-service\", job=\"default/profile-service\", level=\"ERROR\", message=\"Internal server error response generated\", namespace=\"default\", node_name=\"aks-agentpool-32864626-vmss000018\", pod=\"profile-service-579b4765c7-4xw84\", service=\"profile-service\", spanId=\"7e54469782462843\", stack_trace=\"\\ncom.test.profileservice.exception.FirebaseUserException: Deleting user failed\\n\\tat com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:19)\\n\\tat com.test.profileservice.service.ProfileServiceImpl.deleteUserProfile(ProfileServiceImpl.java:116)\\n\\tat com.test.profileservice.controller.ProfileController.deleteUserProfile(ProfileController.java:275)\\n\\tat jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)\\n\\tat java.base/jdk.internal.reflect.Dele"
Loki limits:
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_entries_limit_per_query: 40000
max_line_size: 0