Change and merge log level labels

Grafana Loki
1

Hi teams,
I have this query to group and count the log_level

sum by(severity,level) (count_over_time({source="gitlab",zone="Staging", vm_name="lum1-cicd-1-a"}  
| json 
| logfmt 
| drop __error__ , __error_details__  [$__range]
))

image
image2529×787 56.4 KB

As you can see in the image, I have 2 fields of log level: severity and level. How can I change the field name from “severity” to “level”?
And some log levels have values in lowercase such as info, debug. Some log levels have value in uppercase such as INFO, DEBUG. How can I group log levels that have same value without lowercase or uppercase?

Thanks.

2

@tonyswumac, could you have a look at this?

3

I tried to use the relabel_configs, but it still does not work, here is my promtail config file:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: https://loki.lumera-services.com/loki/api/v1/push
    basic_auth:
      username: lumsrv
      password: configured
    external_labels:
      vm_name: ${HOSTNAME}
      zone: Staging
      export: loki
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          source: sys-varlogs
          __path__: /var/log/*log
      - targets:
          - localhost
        labels:
          source: sys-messages
          __path__: /var/log/messages
      - targets:
          - localhost
        labels:
          source: sys-secure
          __path__: /var/log/secure
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels:
         - __meta_docker_container_name
        regex: /(.*)
        target_label: source
      - source_labels: [severity]
        target_label: level

    pipeline_stages:
      - match:
          selector: '{source="sys-varlogs"}'
          stages:
            - regex:
                expression: '^(?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{4})\s+(?P<log_level>\S+)\s(?P<rest>.*)$'
            - labels:
                level: log_level
      - match:
          selector: '{source="nexus"}'
          stages:
            - regex:
                expression: '^(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\+\d{4})\s+(?P<log_level>\S+)\s+(?P<rest>.*)$'
            - labels:
                level: log_level
      - match:
          selector: '{source="jenkins"}'
          stages:
            - regex:
                expression: '^(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+\+\d{4})\s*(?:\[(?:id=\d+)\])?\s*(?P<log_level>\S+)\s+(?P<rest>.*)$'
            - labels:
                level: log_level
      - match:
          selector: '{source="gitlab"}'
          stages:
            - json:
                expressions:
                  timestamp: timestamp
                  message: message
                  level: level
            - labels:
                level:
4

1.) Use label_replace to rename label severity to level:

2.) Use lower to “normalize” value:

This is just advice, not a copy&paste solution. Make a few tries until you reach desired result on the LogQL level.

5

Many thanks @jangaraj ,
I tried to use label_replace but seem it not a good solution for me.
I used replace and it work for me.
For lower I tried to modify it from promtail before the logs will be sent to Loki. I used template, but it still not work. Here is my config:

  - match:
      selector: '{source="gitlab"}'
      stages:
        - replace:
            expression: "(severity)"
            replace: "level"
        - json:
            expressions:
              severity: severity
              timestamp: time
              message: message
              level: level
        - logfmt:
            mapping:
              timestamp: ts
              caller: caller
              level: level
              unknown:
        - labels:
            severity:
            message:
            timestamp:
            caller:
            level:
        - template:
            source: level
            template: '{{ ToLower .Value }}'

Could you have a look on my config?
Is there anything wrong?

Thanks

6

I gave you idea on LogQL level. Promtail is not my cup of tea.