[Promtail][Multiline][Serilog Compact] No config for multiline seems to work

Grafana Loki
1

Hello,

I’ve been trying to configure promtail to correctly display multiline errors as a single one but i can’t seem to get it to work properly.

I am using a basic C# API project.

Here is a complete single-line log generated in my json log file that contains the \n that i would like to be considered as a single log :

{"@t":"2024-11-26T08:34:59.0688983Z","@mt":"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": An unhandled exception was thrown by the application.","@l":"Error","@x":"System.InvalidOperationException: Randomly generated exception for testing.\n   at ErpOrchestrator.Controllers.WeatherForecastController.Get() in /src/ErpOrchestrator/Controllers/WeatherForecastController.cs:line 34\n   at lambda_method4(Closure, Object)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location ---\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)","@tr":"70595badb948bbc0eee71b18f663ca06","@sp":"38e7e7cf75458783","ConnectionId":"0HN8D7UJV6FII","TraceIdentifier":"0HN8D7UJV6FII:00000003","EventId":{"Id":13,"Name":"ApplicationError"},"SourceContext":"Microsoft.AspNetCore.Server.Kestrel","RequestId":"0HN8D7UJV6FII:00000003","RequestPath":"/weatherforecast","ExceptionDetail":{"Type":"System.InvalidOperationException","HResult":-2146233079,"Message":"Randomly generated exception for testing.","Source":"ErpOrchestrator","StackTrace":"   at ErpOrchestrator.Controllers.WeatherForecastController.Get() in /src/ErpOrchestrator/Controllers/WeatherForecastController.cs:line 34\n   at lambda_method4(Closure, Object)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location ---\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)","TargetSite":"Void MoveNext()"}}

The YAML ConfigMap as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: monitoring
  labels:
    app: promtail
data:
  config.yaml: |
    server:
      http_listen_port: 3101

    positions:
      filename: /var/log/positions.yaml

    clients:
      - url: http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push
        tenant_id: erp-orchestrator

    scrape_configs:
      - job_name: erporchestrator-logs
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            target_label: job
            replacement: erporchestrator-logs
          - source_labels: [__meta_kubernetes_pod_label_app]
            target_label: pod_name
            replacement: erporchestrator
          - source_labels: [__meta_kubernetes_namespace]
            target_label: __namespace__
            replacement: prod
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: __path__
            replacement: /app/logs/application_logs*.json

        pipeline_stages:
          - multiline:
              firstline: '^{"@t"'

          - replace:
              expression: '(\n|\r|\r\n)'
              replacement: ' '

          - json:
              expressions:
                timestamp: '"@t"'
                message: '"@mt"'
                level: '"@l"'
                exception: '"@x"'
                stack_trace: '"@x"'

This config is correctly applied to my promtail pod and i can see the logs generated in grafana through loki source :

image
image1592×601 139 KB

But as you can see they are all in multiple lines.

The log file looks something like this :

image
image1634×171 17.4 KB

I would like the \n in the stack trace to be correctly handled as beeing part of a single log. I’ve tried to do a replace stage in the pipeline but didnt seem to help.

The configuration file is correctly applied to the pod :

image
image835×667 14.6 KB

Cheers for any help !

2

If I am reading your question right, I think your problem is that because your logs are JSON formatted, you can’t perform multiline filter directly, instead you’ll want to parse first with JSON, set your log line to the output part of the JSON string that you want, then do multilne afterwards. Something like this (not tested):

- json:
    expressions:
      output: @x
      timestamp: @t

<DO OTHER PROCESSING SUCH AS SETTING LABELS>

- output:
    source: output

- multiline:
    firstline: `^[^-\s]+`  # match any string that doesn't start with whitespace