Kubernetes Vanilla Parse Ingress Logs on Loki from Promtail

Grafana Configuration
1

Hello dear friends, I will tell you what my issue is.

I installed loki and promtail, via helm. The logs are arriving, but I would like to make a match of the logs of the ingress-nginx.

I show you how they are arriving to my loki:

loki
loki3394×704 196 KB

I really don’t know what I am doing wrong, I have tried to configure this regular expression without success.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki-stack.monitoring.svc.cluster.local:3100/loki/api/v1/push
    external_labels:
      job: promtail

scrape_configs:
  - job_name: nginx-ingress-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx-ingress-logs
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>[^ ]+) - - \[[^\]]+\] "(?P<request>[A-Z]+)'
          labels:
            app: nginx-ingress-microk8s-controller
            container: nginx-ingress-microk8s
            job: ingress/nginx-ingress-microk8s-controller
            namespace: ingress
            node_name: ubuntu-master
            pod: nginx-ingress-microk8s-controller-ns44v
            stream: stdout

Then

helm upgrade loki-stack grafana/loki-stack -n monitoring --set promtail.config.promtailYaml="$(cat promtail-config.yaml)"

here the crude line:

192.168.0.10 - - [02/Oct/2023:23:45:49 +0000] "GET /demo?1696290349286&_=1696289145128 HTTP/2.0" 200 142 "https://mi-dominio.xxxx.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" 481 0.003 [default-app-dominio-80] [] 10.1.172.217:8080 142 0.003 200 fbadb1d63c0d5d322bcf182e18180062

Any idea? Thanks! Guys! I appreciate! How can i have parsed, with source_ip, http_status, etc.

Regards,

Santi

2

Update: I add the labels on a Query.

{app="nginx-ingress-microk8s-controller",namespace="ingress"}
| pattern `<remote_addr> - - [<time_local>] "<method> <request> HTTP/<http_version>" <status> <body_bytes_sent> "<referer>" "<user_agent>" <request_time> <_> <_> <_> <status_ingress> [<resource_name>] <_> <_> <_> <log_id>`
|  method="GET" | referer=`https://DOMAIN.com/`

But I want to have it by defult. I Try with this but something its wrong.

I create promtail-labels-configmap.yaml

#promtail-labels-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-labels
  namespace: monitoring
data:
  labels.yml: |
    - source_labels: ['message']
      target_label: 'method'
      regex: '"([A-Z]+) /'
    - source_labels: ['message']
      target_label: 'status'
      regex: '" (\d{3}) '
    - source_labels: ['message']
      target_label: 'domain'
      regex: ' "GET /api/[^/]+/uid/([^/]+)/'
    - source_labels: ['message']
      target_label: 'user_agent'
      regex: '" "([^"]+)" '
    - source_labels: ['message']
      target_label: 'source_ip'
      regex: ' (\d+\.\d+\.\d+\.\d+) - - '

And promtail-config.yaml:

  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki-stack.monitoring.svc.cluster.local:3100/loki/api/v1/push
    external_labels:
      job: promtail

scrape_configs:
  - job_name: demo_logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: demo_logs
    pipeline_stages:
      - json:
          expressions:
            remote_addr: '([^ ]+)'
            dash1: '-'
            dash2: '-'
            timestamp: '\[([^ ]+ [^ ]+)\]'
            request: '"([^"]+)"'
            status_code: '(\d{3})'
            response_size: '(\d+)'
            referer: '"([^"]+)"'
            user_agent: '"([^"]+)"'
            dash3: '-'
            dash4: '-'
            dash5: '-'
            dash6: '-'
            dash7: '-'
            dash8: '-'
            dash9: '-'
            dash10: '-'
            dash11: '-'
            dash12: '-'
            dash13: '-'
            dash14: '-'
            dash15: '-'
            remote_addr2: '([^ ]+)'
            dash16: '-'
            response_time: '([\d.]+)'
            response_size2: '(\d+)'
            status_code2: '(\d{3})'
            hash: '([0-9a-f]+)'
  - job_name: ingress/nginx-ingress-microk8s-controller
    static_configs:
      - targets:
          - localhost
        labels:
          app: nginx-ingress-microk8s-controller
          namespace: ingress
    pipeline_stages:
      - match:
          selector: '{job="nginx-ingress-microk8s-controller"}'
          patterns:
            - '<remote_addr> - - [<time_local>] "<method> <request> HTTP/<http_version>" <status> <body_bytes_sent> "<referer>" "<user_agent>" <request_time> <_> <_> <_> <status_ingress> [<resource_name>] <_> <_> <_> <log_id>'
            
  - job_name: ingress/nginx-ingress-microk8s-controller
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        separator: ;
        regex: (.+)
        target_label: container
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        separator: ;
        regex: (.+)
        target_label: pod
        replacement: $1
        action: replace
    pipeline_stages:
      - match:
          selector: '{job="ingress/nginx-ingress-microk8s-controller"}'
          stages:
            - json:
                expressions:
                  - '^(?<remote_addr>\S+) - - \[(?<time_local>[^\]]+)\] "(?<method>\S+) (?<request>[^\s]+) (?<http_version>\S+)" (?<status>\d+) (?<body_bytes_sent>\d+) "(?<referer>[^\"]+)" "(?<user_agent>[^\"]+)" (?<request_time>\d+(\.\d+)?) \[(?<status_ingress>[^\]]+)\] \[\] (?<server_ip_port>[^ ]+) (?<server_response_time>\d+(\.\d+)?) (?<response_status>\d+) (?<log_id>\S+)'
                source: log

Then apply on the helm chart:

helm upgrade loki-stack grafana/loki-stack -n monitoring --set promtail.config.promtailYaml="$(cat promtail-config.yaml)"

Some can help me?

Regards