Grafana Loki escape character

I send docker logs from fluent bit to grafana
tcpdump shows that fluent bit send log with escape character:
i need to escape double quotes by \

example of tcpdump output:
[["1701677792000000000","message=\"Query error: ERROR: invalid input syntax for type integer: \"713dsa\"\nLINE 4: AND \"curs_id\" = '713dsa'\n ^ - Invalid query: SELECT *\nFROM \"courses\"\nWHERE \"archive\" = 0\nAND \"curs_id\" = '713dsa'\" level_name=\"ERROR\" channel=\"production\" datetime=\"2023-12-04 11:16:32\" server_name=\"_\" host_name=\"main-ry.com\""],["1701677792000000000","message=\"Uncaught Exception Error: \"Call to a member function row_array() on bool\" at /var/www/html/application/models/Teacher_model.php line 1722\" level_name=\"ERROR\" channel=\"production\" datetime=\"2023-12-04 11:16:32\" server_name=\"_\" host_name=\"main-ry.com\""]]}]}

but in loki i see that data comes without escaping double quotes and not in a one string:
message=“Query error: ERROR: invalid input syntax for type integer: “713dsa”
LINE 4: AND “curs_id” = ‘713dsa’
^ - Invalid query: SELECT *
FROM “courses”
WHERE “archive” = 0
AND “curs_id” = ‘713dsa’” level_name=“ERROR” channel=“production” datetime=“2023-12-04 11:16:32” server_name=“" host_name=“main-ry.com"�message=“Uncaught Exception Error: “Call to a member function row_array() on bool” at /var/www/html/application/models/Teacher_model.php line 1722” level_name=“ERROR” channel=“production” datetime=“2023-12-04 11:16:32” server_name=”” host_name=“main-ry.com

do someone know a reason?
or maybe someone saw similar problem?

The tcpdump output is likely escaped by whatever you are using to export the dump. I would recommend you to do this:

  1. Configure your fluentbit to output to a local file. This way you can actually see what the output looks like from fluentbit, and I suspect you’ll see exactly what you see in Grafana Loki.

  2. Use multiline parsing in fluentbit to properly group your loglines.

unfortunately i had with output to file the same as with tcpdump(

root@app:/etc/fluent-bit# cat app.log app.log: [1701730869.000000000, {"message":"Query error: ERROR: invalid input syntax for type integer: \"dsadas\"\nLINE 4: AND \"curs_id\" = 'dsadas'\n ^ - Invalid query: SELECT *\nFROM \"courses\"\nWHERE \"archive\" = 0\nAND \"curs_id\" = 'dsadas'","context":{},"level":400,"level_name":"ERROR","channel":"production","datetime":"2023-12-05 02:01:09","extra":{},"server_name":"_","host_name":"main-ry.com"}] app.log: [1701730869.000000000, {"message":"Uncaught Exception Error: \"Call to a member function row_array() on bool\" at /var/www/html/application/models/Teacher_model.php line 1722","context":{"exception":{"class":"Error","message":"Call to a member function row_array() on bool","code":0,"file":"/var/www/html/application/models/Teacher_model.php:1722"}},"level":400,"level_name":"ERROR","channel":"production","datetime":"2023-12-05 02:01:09","extra":{},"server_name":"_","host_name":"main-ry.com"}]

Couple of things to try:

  1. First of all, your app is not writing proper JSON (unless it’s not supposed to). If we look at first line here:
“Query error: ERROR: invalid input syntax for type integer: "dsadas"\nLINE 4

You can see it’s already not escaped properly. This is probably not that big of a deal during ingestion, but you will have a hard time parsing the log as JSON string.

  1. If your logs come in multiple lines, then you need to configure fluentbit with multiline configuration. If your logs come in as one single line with newline character in them, then you can try to replace the newline character with a space.

I apologize for my previous answer
I edited it (escape characters were not displayed)
and as you can see i have escape character near integer: “dsadas”

so, i think, fluent-bit send to loki right log
but loki cant read it(

sorry, didnt answer for you first reply about multi parsing
in docker log i have:
{"message":"Query error: ERROR: invalid input syntax for type integer: \"713afssfasfa\"\nLINE 4: AND \"curs_id\" = '713afssfasfa'\n ^ - Invalid query: SELECT *\nFROM \"courses\"\nWHERE \"archive\" = 0\nAND \"curs_id\" = '713afssfasfa'","context":{},"level":400,"level_name":"ERROR","channel":"production","datetime":"2023-12-05 17:14:25","extra":{},"server_name":"_","host_name":"main-ry.com"}

its a 1 json string

right now im searching info for other storage parameters
my loki config is
`/etc/loki $ cat local-config.yaml
auth_enabled: false

server:
http_listen_port: 3100

common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
`

I’d recommend you to revisit your fluentbit configuration and see if perhaps additional processing is messing up the format. If the loglines are sent to Loki intact with proper quotes it should be fine.

As a test, I forwarded your example log line using promtail to my Loki server, and this is what it looks like:

1 Like

thank you for you time and testing
i found problem with my custom fluent-bit parser
i chose another parser and loki and grafana got right logs with escape character

i think i can choose your answer as a solution because it works at some point

i thought tcpdump shows package that will be received by host side(