Send custom log file to grafana

toug31 · December 2, 2022, 11:27am

What Grafana version and what operating system are you using?
The data would come from a raspberry pi device armv7
What are you trying to achieve?
I would like grafana to parse a custom log file the device generate. Log file is updated from python scripts.

Here is a log sample:

2022-12-01_15-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.906
2022-12-01_15-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.875
2022-12-01_15-45-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.781
2022-12-01_16-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.781
2022-12-01_16-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.7495
2022-12-01_16-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.687
2022-12-01_16-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.6245
2022-12-01_17-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-01_17-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-01_17-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-01_17-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.5
2022-12-01_18-00-02;Allumage Circu ECS;GPIO16=1;PIN36=1
2022-12-01_18-00-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.5
2022-12-01_18-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-01_18-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5935
2022-12-01_18-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5935
2022-12-01_19-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-01_19-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.6245
2022-12-01_19-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.6245
2022-12-01_19-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-01_20-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-01_20-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-01_20-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.625
2022-12-01_20-45-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-01_21-00-01;Extinction Circu ECS;GPIO16=0;PIN36=0
2022-12-01_21-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.7185
2022-12-01_21-15-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.7185
2022-12-01_21-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.687
2022-12-01_21-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.687
2022-12-01_22-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.687
2022-12-01_22-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.625
2022-12-01_22-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.625
2022-12-01_22-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-01_23-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-01_23-15-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.406
2022-12-01_23-30-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.406
2022-12-01_23-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.312
2022-12-02_00-00-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3125
2022-12-02_00-15-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_00-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_00-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_01-00-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_01-15-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_01-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.281
2022-12-02_01-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.375
2022-12-02_02-00-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3125
2022-12-02_02-15-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.312
2022-12-02_02-30-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3125
2022-12-02_02-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3435
2022-12-02_03-00-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3435
2022-12-02_03-15-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.375
2022-12-02_03-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.3435
2022-12-02_03-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.437
2022-12-02_04-00-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.406
2022-12-02_04-15-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.4685
2022-12-02_04-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.437
2022-12-02_04-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-02_05-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_05-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_05-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-02_05-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.156
2022-12-02_06-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_06-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5935
2022-12-02_06-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5935
2022-12-02_06-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5935
2022-12-02_07-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-02_07-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-02_07-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.5
2022-12-02_07-40-01;Allumage Circu ECS;GPIO16=1;PIN36=1
2022-12-02_07-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.5625
2022-12-02_08-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-02_08-10-01;Extinction Circu ECS;GPIO16=0;PIN36=0
2022-12-02_08-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.7185
2022-12-02_08-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.75
2022-12-02_08-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.25
2022-12-02_09-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.6875
2022-12-02_09-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-02_09-30-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.656
2022-12-02_09-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.687
2022-12-02_10-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.625
2022-12-02_10-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.625
2022-12-02_10-30-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.125
2022-12-02_10-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_11-00-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.562
2022-12-02_11-15-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_11-30-02;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_11-45-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_12-00-01;Plancher_OFF;GPIO20=0;PIN38=0;LED_ON;CONSIGNE:19.5;READ_TEMPERATURE:19.531
2022-12-02_12-15-02;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.437

The file format can be easily changed if required.

I read some topics but I was not able to understand where I say to grafana agent it has to read this file which is located in /home/pi/Logs-chaudiere.log

I aleady configured grafana with the predefined “Linux Server” configuration . So I’m able to see the raspbery pi device system metrics. But now I would like to parse this custom log file.

Thanks.

yosiasz · December 2, 2022, 1:13pm

Welcome

You can install the csv plugin and configure it to read this file using the ; delimiter

toug31 · December 2, 2022, 5:47pm

Ok thanks, I made a search and found this webpage:

But how I make my device to send the log file to grafana?

yosiasz · December 2, 2022, 5:57pm

what standard are you using for this log file? I see what may seem like discrepancies to the naked eye

2022-12-01_17-45-01;Plancher_ON;GPIO20=1;PIN38=1;LED_OFF;CONSIGNE:19.5;READ_TEMPERATURE:19.5
2022-12-01_18-00-02;Allumage Circu ECS;GPIO16=1;PIN36=1

Top row has more data.

You get this error because of that

Some data is

GPIO20=1

while others are

CONSIGNE:19.5

toug31 · December 2, 2022, 6:09pm

I can work the file format. The 2 lines are different because they are different type of action of the heat system of my house. The first one is for the heating of the house, and the second one is about the hot water for faucets.
First one is about heating floor and Plancher_ON is triggered when read temperature is below consigne temperature. Check is done every 15 minutes. 2nd one is only triggered on specific hours.
I will create different log file for each actions.
But I want the raspeberry pi device automatically send the data to grafana to make it to draw the temperatures and if Plancher_ON (floor heating) or Plancher_OFF (floor not heating).

All these scripts are started with cron. Maybe I could link cron logs to grafana.

yosiasz · December 2, 2022, 6:11pm

You cant send data directly to grafana. grafana is a visualization tool that consumes data sources

Your other options would be to send the data to

Loki
mysql
sql server
sqlite
influxdb

instead of sending it to a log file.

toug31 · December 2, 2022, 7:43pm

Ok, but how to I send the data to grafana?

yosiasz · December 2, 2022, 7:48pm

In grafana You would

install data source plugin
create a datasource connection

But for now you can start out with a clean log file instead of learning too many moving pieces.

You need to resolve the issues I mentioned. Once you do poat back a clean log file.

toug31 · December 3, 2022, 8:15am

Hi,
How did you tried to manually insert the log file?
I did the change on the log file. Now each line has same parameter number. I created 3 different log file.
The one I’m the more interested in is this one:

Timestamp;PLANCHER_STATE;GPIO20_STATE;PIN38_STATE;LED2_STATE;CONSIGNE_TEMPERATURE;READ_TEMPERATURE
2022-11-08_21-00-01;Plancher_ON;1;1;OFF;25.0;23.375
2022-11-08_21-15-01;Plancher_ON;1;1;OFF;25.0;23.156
2022-11-08_21-30-01;Plancher_ON;1;1;OFF;25.0;23.156
2022-11-08_21-45-02;Plancher_ON;1;1;OFF;25.0;23.2185
2022-11-08_22-00-01;Plancher_ON;1;1;OFF;25.0;23.1875
2022-11-08_22-15-01;Plancher_ON;1;1;OFF;25.0;22.937
2022-11-08_22-30-01;Plancher_ON;1;1;OFF;25.0;22.937
2022-11-19_16-15-01;Plancher_ON;1;1;OFF;23.0;22.937
2022-11-19_16-30-01;Plancher_ON;1;1;OFF;23.0;22.281
2022-11-19_16-45-01;Plancher_ON;1;1;OFF;23.0;22.437
2022-11-19_17-00-01;Plancher_OFF;0;0;ON;23.0;23.031
2022-11-19_17-15-02;Plancher_OFF;0;0;ON;23.0;23.125
2022-11-19_17-30-01;Plancher_ON;1;1;OFF;23.0;22.406
2022-11-19_17-45-01;Plancher_ON;1;1;OFF;23.0;22.9375
2022-11-19_18-00-02;Plancher_ON;1;1;OFF;23.0;22.812
2022-11-19_18-15-01;Plancher_ON;1;1;OFF;23.0;22.531
2022-11-19_18-30-01;Plancher_ON;1;1;OFF;23.0;22.25
2022-11-19_18-45-02;Plancher_ON;1;1;OFF;23.0;22.937
2022-11-19_19-00-01;Plancher_ON;1;1;OFF;23.0;22.312
2022-11-19_19-15-01;Plancher_ON;1;1;OFF;23.0;22.406
2022-11-19_19-30-02;Plancher_ON;1;1;OFF;23.0;22.625
2022-11-19_19-45-01;Plancher_ON;1;1;OFF;23.0;22.937
2022-11-19_20-00-01;Plancher_ON;1;1;OFF;23.0;22.906
2022-11-19_20-15-02;Plancher_OFF;0;0;ON;23.0;23.2495

I would like to try it.

yosiasz · December 3, 2022, 2:27pm

Looks great. What kind of graph do you want to plot this as?

toug31 · December 3, 2022, 8:44pm

Like this graph I created with excel:

Here there is CONSIGNE_TEMPERATURE and READ_TEMPERATURE COLUMN, but I would like to add maybe the PLANCHER_STATE which is text format.

yosiasz · December 5, 2022, 8:29pm

toug31 · December 9, 2022, 5:53pm

Thanks, But I want to do it by myself, and then make the Pi to send the data to grafana.

yosiasz · December 9, 2022, 5:57pm

Not sure what you mean pi to send the data to grafana

You dont “send” data to grafana
Grafana reads data from data sources

fadjar340 · December 16, 2022, 11:04am

First of all, Grafana just rely on the data source, second, you need to send the log to the Grafana data source.
I suggest you install loki in the grafana server and it will need some space to store the log, then you need to install promtail into the Pi and config the loki server as a target in promtail config and the log folder source in the Pi.

Then you can use Loki data source in Grafana and create the dashboard using PromQL syntax.

Regards,
Fadjar T