simtcr
March 4, 2025, 7:02am
1
I am trying to get started with dashboard using loki as datasource.
I have nginx access.log file being scarped by loki. And loki is added as a datasource in grafana.
A field named ‘status’ is extracted in grafana with all the http status codes.
Can any one please help with simple query to display all ‘status’ codes using a bar graph. Something like below,
eg access.log entry is,
{"geoip_country_code": "United States","remote_addr": "xx.xx.xx.xx", "status": "404", "request_uri": "/ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application", "msec": "1741071697.969", "connection": "6339", "connection_requests": "1", "pid": "481871", "request_id": "1f97bde40607559a67675edbdf0d2107", "request_length": "185", "remote_user": "", "remote_port": "44560", "time_local": "04/Mar/2025:12:31:37 +0530", "time_iso8601": "2025-03-04T12:31:37+05:30", "request": "GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.1", "args": "", "body_bytes_sent": "19599", "bytes_sent": "20005", "http_referer": "", "http_user_agent": "Mozilla/5.0 zgrab/0.x", "http_x_forwarded_for": "", "http_host": "152.70.67.140", "server_name": "_", "request_time": "0.196", "upstream": "[2406:7400:51:4754::10]:443", "upstream_connect_time": "0.024", "upstream_header_time": "0.184", "upstream_response_time": "0.196", "upstream_response_length": "87306", "upstream_cache_status": "", "ssl_protocol": "TLSv1.2", "ssl_cipher": "ECDHE-ECDSA-AES128-GCM-SHA256", "scheme": "https", "request_method": "GET", "server_protocol": "HTTP/1.1", "pipe": ".", "gzip_ratio": "4.46", "http_cf_ray": ""}
Thanks.
please share a sample log file
simtcr
March 4, 2025, 6:51pm
4
Is one entry from access.log file enough?
All entries follow same structure.
{"geoip_country_code": "United States","remote_addr": "xx.xx.xx.xx", "status": "404", "request_uri": "/ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application", "msec": "1741071697.969", "connection": "6339", "connection_requests": "1", "pid": "481871", "request_id": "1f97bde40607559a67675edbdf0d2107", "request_length": "185", "remote_user": "", "remote_port": "44560", "time_local": "04/Mar/2025:12:31:37 +0530", "time_iso8601": "2025-03-04T12:31:37+05:30", "request": "GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.1", "args": "", "body_bytes_sent": "19599", "bytes_sent": "20005", "http_referer": "", "http_user_agent": "Mozilla/5.0 zgrab/0.x", "http_x_forwarded_for": "", "http_host": "152.70.67.140", "server_name": "_", "request_time": "0.196", "upstream": "[2406:7400:51:4754::10]:443", "upstream_connect_time": "0.024", "upstream_header_time": "0.184", "upstream_response_time": "0.196", "upstream_response_length": "87306", "upstream_cache_status": "", "ssl_protocol": "TLSv1.2", "ssl_cipher": "ECDHE-ECDSA-AES128-GCM-SHA256", "scheme": "https", "request_method": "GET", "server_protocol": "HTTP/1.1", "pipe": ".", "gzip_ratio": "4.46", "http_cf_ray": ""}
1 Like
looks good. how are you scraping this using Loki?
simtcr
March 4, 2025, 7:04pm
6
ubuntu@dual-stack:~$ cat /etc/promtail/config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: https://xx.xx.xx.xx:3100/loki/api/v1/push
tls_config:
insecure_skip_verify: true
scrape_configs:
- job_name: nginx-relay
static_configs:
- targets:
- nginx-relay
labels:
job: nginx-relay
host: dual-stack
__path__: /var/log/nginx/access.log
pipeline_stages:
- json:
expressions:
geoip_country_code: geoip_country_code
remote_addr: remote_addr
status: status
request_uri: request_uri
msec: msec
connection: connection
connection_requests: connection_requests
pid: pid
request_id: request_id
request_length: request_length
remote_user: remote_user
remote_port: remote_port
time_local: time_local
time_iso8601: time_iso8601
request: request
args: args
body_bytes_sent: body_bytes_sent
bytes_sent: bytes_sent
http_referer: http_referer
http_user_agent: http_user_agent
http_x_forwarded_for: http_x_forwarded_for
http_host: http_host
server_name: server_name
request_time: request_time
upstream: upstream
upstream_connect_time: upstream_connect_time
upstream_header_time: upstream_header_time
upstream_response_time: upstream_response_time
upstream_response_length: upstream_response_length
upstream_cache_status: upstream_cache_status
ssl_protocol: ssl_protocol
ssl_cipher: ssl_cipher
scheme: scheme
request_method: request_method
server_protocol: server_protocol
pipe: pipe
gzip_ratio: gzip_ratio
http_cf_ray: http_cf_ray
- labels:
geoip_country_code: geoip_country_code
remote_addr: remote_addr
status: status
http_user_agent: http_user_agent
request_time: request_time
ubuntu@dual-stack:~$
1 Like
here is one way, using Extract transformation
simtcr
March 5, 2025, 3:39am
8
When I go to transformations, I see it empty and have an option to add transformations. I clicked add transformations. Added “Extract Fields”
However Source field dropdown is empty and also its not allowing to enter {} labels
Well what does your Loki query look like and show us the Data view
simtcr
March 5, 2025, 4:05am
10
simtcr:
{} labels
So far I have the query as below,
sum by (status) (count_over_time({job="nginx-relay"} | json | status=~".*" [1h]))
and below got generated without transformations.
by data view I believe you mean the table view,?
1 Like
simtcr
March 6, 2025, 3:59am
11
Any help please?
Basically I want total count of each status types, for eg,
{status = “200”} = 52
{status = “404”} = 89
{status = “301”} = 102
Then create bar graph out of it.
simtcr
March 6, 2025, 4:11am
12
I finally got it working as below,
1 Like