Promtail recieving GELF "full_message" with NULL data type gives error

Grafana Loki
1

We have been using Graylog for some time. We are now trying out Loki and using Grafana Agent/Promtail to recieve logs in GELF format and push them to Loki.

The application sending the logs in this case is always sending the “full_message” field. And if there is no data for the field the data is set to NULL.

This works well in Graylog. In Graylog it simply shows the data as Nul for the f"ull_message" field.

Grafana Agent/Promtail does however give this message:
ts=2023-03-16T07:44:22.5273352Z caller=gelftarget.go:100 level=error component=logs logs_config=windows msg="error while reading gelf message" listen_address=0.0.0.0:12201 err="json.Unmarshal: invalid type for field full_message"

We believe that Promtail is expecting a value for the field “full_message” to be of type string and nothing else and this is why the error occurs. According to the GELF format the “full_message” field is optional. So the application could have opted to not send the field if there is no data to send.

But, Graylog have opted to treate the field “full_message” with the null data type as if it was simply an empty string.

My question is, is this the way Promtail deals with a “full_message” field with the wrong data type? And what are your thoughts on Promtail dealing with it the similar to Graylog? Meaning, treat it basically as an empty string.

One could of course argue that the application really should send the “full_message” field when there is something in that field to send.

I hope I made my question and problem clear.

Thanks in advance for any insights.

2

I dug a bit into the code of the Go Gelf library and this error we see originates from the JSON standard library for Go.

And when a message is recieved with Null it will simply throw an error. So, this answers my question.

3

This problem occurs when using Serilog (a popular .NET library for logging) together with Serilog.Sinks.Graylog.

For short messages (smaller than 500 characters), the library will set short_message to the message and full_message explicitly to null. For longer messages short_message is truncated and full_message is set to the full message.

This is an example of how the GELF-message can look for a short message:

{“facility”:“GELF”,“full_message”:null,“host”:“myhost”,“level”:7,“short_message”:“My short message”,“timestamp”:1679473505.141,“version”:“1.1”,“_stringLevel”:“Debug”,“_MethodName”:“MyMethodName”,“_Duration”:30,“_SourceContext”:“My.Namespace”}

I think that the most reasonable behavior would be for Promtail to treat a full_message being set to null the same way as if full_message was missing altogether from the message.

It is of course possible to change the behavior in Serilog.Sinks.Graylog (it is probably enough to set NullValueHandling to Ignore for the NewtonSoft.Json library). But there might be other logging libraries out there that has a similar behavior, so it makes sense to be less strict regarding this in Promtail.