We have been using Graylog for some time. We are now trying out Loki and using Grafana Agent/Promtail to recieve logs in GELF format and push them to Loki.
The application sending the logs in this case is always sending the “full_message” field. And if there is no data for the field the data is set to NULL.
This works well in Graylog. In Graylog it simply shows the data as Nul for the f"ull_message" field.
Grafana Agent/Promtail does however give this message:
ts=2023-03-16T07:44:22.5273352Z caller=gelftarget.go:100 level=error component=logs logs_config=windows msg="error while reading gelf message" listen_address=0.0.0.0:12201 err="json.Unmarshal: invalid type for field full_message"
We believe that Promtail is expecting a value for the field “full_message” to be of type string and nothing else and this is why the error occurs. According to the GELF format the “full_message” field is optional. So the application could have opted to not send the field if there is no data to send.
But, Graylog have opted to treate the field “full_message” with the null data type as if it was simply an empty string.
My question is, is this the way Promtail deals with a “full_message” field with the wrong data type? And what are your thoughts on Promtail dealing with it the similar to Graylog? Meaning, treat it basically as an empty string.
One could of course argue that the application really should send the “full_message” field when there is something in that field to send.
I hope I made my question and problem clear.
Thanks in advance for any insights.