Loki Parsing for Log that doesn't have fix pattern

[sasmilyshweta]

Hi Team,

I am having my log setup from grafana telegraf and loki. and all the logs are coming in the grafana panel but I am not able to extract the information from that log.

I am using [[inputs.tail]] parser to get the log and pattern is grok with greedy data. as my logs are not in the same format.

Sharing some sample log here:

jsonstring="{“channel”:“crm_process_booking_log”,“log_type”:“DEBUG”,“index_name”:“crm_process_booking_log”,“index_type”:“crm_process_booking_log”,“error_type”:0,“message”:{“data”:{“STEP”:“Booking Successful”,“action”:“Booking Created”,“message”:“Booking (123) placed successfully.”,“amzn_trace_id”:“Root=1-61a76286-4e9cd6ffb47157340d”},“calling_trace”:{“file”:"/var/www/newcrm/application/abc.php",“line”:998,“function”:“addProcessBookingLog”,“class”:“Enterprise_logger”}},“amzn_trace_id”:“Root=1-61a76286-4e9cd6ffb47157340d”,“nginx_request_id”:“de0bfcae0ad79e38195ebaf6”,“timestamp”:“2021-12-01 17:24:46”}"

jsonstring="{“channel”:“crm_process_booking_log”,“log_type”:“DEBUG”,“index_name”:“crm_process_booking_log”,“index_type”:“crm_process_booking_log”,“error_type”:0,“message”:{“data”:{“STEP”:“booking_updated”,“slot_data”:{“samplecollector_id”:0,“isavailable”:0,“booking_id”:“123”,“user_id”:false},“amzn_trace_id”:“Root=1-61a76286-4157340d”},“calling_trace”:{“file”:"/var/www/newcrm/application/abc.php",“line”:1652,“function”:“addProcessBookingLog”,“class”:“Enterprise_logger”}},“amzn_trace_id”:“Root=1-61a76286-4157340d”,“nginx_request_id”:“de0bf80a3595ebaf6”,“timestamp”:“2021-12-01 17:24:46”}"

and in most of the cases I want my log to be extracted from booking_id i.e. 123

this Booking_id can be come inside message object as string or as a key value like “booking_id”
so i want to apply some generic pattern that will bring all instances for a perticular booking_id or a perticular word.

Any help would be appreciated.

Thanks,
Shweta

[b0b]

Hello @sasmilyshweta ,

I’m by no means a Loki and LogQL pro yet but here are some suggestions.

You should be able to search for e.g. booking_id anywhere in a log using a line filter.

After that, you have to get creative. I think you could then use the logfmt parser to break out the JSON part. Then you might be able to put that through the json parser. This will not be very efficient though.

Hope that helps