NGINX logs are not taken as JSON?

Grafana Alloy
, ,
1

Hi, need your help. I want to collect logs from web application befing the NGINX.
I’ve setup NGINX to save logs in JSON format:

    log_format json_analytics escape=json '{
      "msec": "$msec",
      "connection": "$connection",
      "connection_requests": "$connection_requests",
      ............   
      "ssl_cipher": "$ssl_cipher",
      "scheme": "$scheme",
      "request_method": "$request_method",
      "server_protocol": "$server_protocol",
      "pipe": "$pipe",
      "gzip_ratio": "$gzip_ratio"
    }';
    
    access_log  /var/logs/my_site.com_json.log json_analytics;

Alloy config is:

// -------------- Loki logs -------------------------
loki.write "grafana_cloud_loki" {
    endpoint {
        url = "https://logs-prod-xxx.grafana.net/loki/api/v1/push"

        basic_auth {
            username = "username"
            password = "pass"
        }
    }
}

local.file_match "logs_node_exporter_direct_scrape" {
    path_targets = [{
        __address__ = "localhost",
        __path__    = "/var/logs/my_site.com_json.log",
        instance    = "my_site",
        host        = "my_site",
        job         = "integrations/nginx",
  }]
}

loki.source.file "logs_node_exporter_direct_scrape" {
    targets    = local.file_match.logs_node_exporter_direct_scrape.targets
    forward_to = [loki.write.grafana_cloud_loki.receiver]
}

But in Grafana Cloud I see logs which are not accepted as JSON but as lines:

image
image1038×672 43.8 KB

Please help me to fix that. Thnx

2

wonder if it is reading it as line feeds as it seems like each “row” is scrapped separately. might want to look at stage.json

3

I’m following this doc: Nginx integration | Grafana Cloud documentation
And there is no mentioning stage.json…

4

there is mention of it in alloy docu

1 Like
5

What does it look like from the log file /var/logs/my_site.com_json.log?

6

I’ve fixed it!!!

It turned out that AI Grot gave me wrong syntax for NGINX logs (quotes). And in logs JSON was splited into separate lines.

log_format json_analytics escape=json '{
  "msec": "$msec",
  "connection": "$connection",
  "connection_requests": "$connection_requests",
  "pid": "$pid",
  "request_id": "$request_id",
  "request_length": "$request_length",
  "remote_addr": "$remote_addr",
  "remote_user": "$remote_user",
  "remote_port": "$remote_port",
  "time_local": "$time_local",
  "time_iso8601": "$time_iso8601",

From official doc is:

log_format json_analytics escape=json '{'
        '"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
        '"connection": "$connection", ' # connection serial number
        '"connection_requests": "$connection_requests", ' # number of requests made in connection
        '"pid": "$pid", ' # process pid
        '"request_id": "$request_id", ' # the unique request id
        '"request_length": "$request_length", ' # request length (including headers and body)
        '"remote_addr": "$remote_addr", ' # client IP
        '"remote_user": "$remote_user", ' # client HTTP username
        '"remote_port": "$remote_port", ' # client port
        '"time_local": "$time_local", '
        '"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
        '"request": "$request", ' # full path no arguments if the request
        '"request_uri": "$request_uri", ' # full path and arguments if the request
        '"args": "$args", ' # args

That helped.

Thank you everyone!

1 Like
7

so the format is json but the file extension is .log? all those single quotes look oogly and a maintenance issue and not clean to me.

something just does not jive to me imho

@tonyswumac how would you approach this with the following json format log file

{
	"msec": "foo_mec",
	"connection": "vader",
	"connection_requests": "sith",
	"ssl_cipher": "mandalorian",
	"scheme": "dark",
	"request_method": "force",
	"server_protocol": "bazinga",
	"pipe": "flute",
	"gzip_ratio": "light speed"
}

and a config

logging {
  level  = "info"
  format = "logfmt"
}


local.file_match "json_logs" {
    path_targets = [{
        __path__ = "/tmp/ngnix.log",
        job      = "json_logs",
    }]
}

loki.source.file "json_logs" {
    targets    = local.file_match.json_logs.targets
    forward_to = [loki.echo.debug.receiver,loki.write.local_loki.receiver]
}

loki.process "ngnix" {
    stage.json {
        expressions = {msec = "", connection = ""}
    }

    stage.labels {
      values = {
        msec = "",
        connection = "",
      }
    }
    forward_to = [loki.echo.debug.receiver,loki.write.local_loki.receiver]
}

loki.echo "debug" { }

loki.write "local_loki" {
    endpoint {
        url = "http://loki:3100/loki/api/v1/push"
    }
}
8

I would try to make it so the json string is on one single line (as the OP mentioned above it was his nginx log configuration that needed fixed).

1 Like
9

It feels like there is a feature missing for Json files you would think that we can add file type Json and it would read the whole file instead of individual lines

11

Perhaps.

Personally I always consider each log line to be an individual unit (with the exception of stack traces). While it’s probably a relatively easy feature to implement and I can’t really think of any hard reason against such feature, I do think that having logs be spanned across multiple lines without good reason is a bad practice, and by implementing this feature some users may potentially feel encouraged (or rather not discouraged) to do so.

1 Like
12

Agreed. The challenge is that many system generated JSON data is not collapsed. In fact when you think of JSON, you don’t generally think of it as collapsed (see grafana JSON model)

Maybe adding a data/file type parameter feature of json would inform the process to read not as individual lines but as a whole

Or a feature that collapses the JSON data when reading it