Access Logs By HTTP codes

Grafana Cloud
, , , ,
1

Hello,
I am using Grafana agent and build in Dashboard to get apache metrics and logs. Everything is working except the part of the dashboard for showing numbers of each http code groups. What do I need to adjust to get line graph showing codes?
Thank you.

hostname.example.com
hostname.example.com1572×506 53.4 KB

2

Hey there!
Sorry you are having issues. :frowning:

You may need to adjust the time range for the dashboard to see the metrics in question.

The metric for that panel is generated by the logs section of the Grafana Agent Apache integration.

You can see the logs snippet below.

    - job_name: integrations/apache_access
      static_configs:
      - targets:
        - localhost
        labels:
          __path__: /var/log/apache2/access.log
          job: integrations/apache_http
          instance: '<your-instance-name>'
      pipeline_stages:
        - regex:
            expression: '^(?P<ip>[^ ]*) [^ ]* (?P<user>[^ ]*) \[(?P<timestamp>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^ ]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>.*)")?$'
        - metrics:
            response_http_codes:
              type: Histogram
              description: "Apache responses by HTTP codes"
              prefix: apache_
              source: code
              config:
                buckets: [199,299,399,499,599]

The resulting apache_response_http_codes_bucket metric is what is behind that panel.

You can check your Grafana Agent logs to see if the logs component is failing in some way.

A line like the following can confirm you are tailing the logs.

ts=2024-02-23T18:50:50.795154Z caller=tailer.go:147 level=info component=logs logs_config=integrations component=tailer msg="tail routine: started" path=/opt/homebrew/var/log/httpd/access_log
ts=2024-02-23T18:50:50.808989Z caller=tailer.go:147 level=info component=logs logs_config=integrations component=tailer msg="tail routine: started" path=/opt/homebrew/var/log/httpd/error_log

A promql query for the bucket metrics can be used to verify that log entries are being counted.
apache_response_http_codes_bucket

image
image1707×830 96.1 KB

1 Like
3

Thanks @peterolivo,
I managed to get metrics by adjusting regex expression.

But now Grafana panel doesn’t look right, it shows almost the same amount of all status codes when it should be mostly 200s:

image
image861×276 21.1 KB

There is an open issue also: Apache HTTP codes histogram is not displayed correctly · Issue #985 · grafana/jsonnet-libs · GitHub

EDIT:
I added code label to config to be able to filter loki logs and it looks it’s showing correctly the numbers.

        - labels:
            method:
            code:

image
image919×387 21.6 KB

And when I query actual metrics, it’s not summed correctly.

image
image563×557 28 KB

Thank you for your help!