Parse CSV to metric

Grafana Loki
#1

Hi,
I would like to parse a csv and plot its content in Grafana.

I have a csv like the following:

date,key0,key1
20220610t133612,5260,3.05
20220610t133712,5260,78.4
...

Converted it in a list of json:

{"date": "20220610t133612", "key0": "5260", "key1": "3.05"}
{"date": "20220610t133712", "key0": "5260", "key1": "78.4"}
...

Use the following config.yaml:

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://<redacted>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
  - json:
      expressions:
        output: data
        stream: stream
        timestamp: date
        key0:
        key1:
  - timestamp:
      source: timestamp
      format: "20060102t150405"
  static_configs:
  - targets:
      - localhost
    labels:
      job: data
      __path__: /var/log/*.log

Send data using promtail:

podman run \
    --name promtail \                                                                                                                                              
    --volume "$PWD/config.yaml:/etc/promtail/config.yaml" \                                                                                                        
    --volume "$PWD:/var/log" \                                                                                                                                                                                                                                                                 
    grafana/promtail:master \                                                                                                                                      
       -config.file=/etc/promtail/config.yaml \                                                                                                                   
       -log.level=debug

If I try to query the data for a new Panel:

{job="data"} | json | line_format "{{.ping}}"

I can see the data only in a table format.
It does not allow me to plot the timeseries.

Any ideas?

Thanks

#2

Welcome

You dont want grafana to read the csv directly and plot it?

#3

Thanks for the reply.

That would be great.
I have already tried the CSV data sources but it is a bit clunky and slow, that’s why I preferred to ingest the file in Loki.

Do you have a better suggestion?

1 Like
#4

this is a custom time format so what data type is it being saved in Loki? is it being save as a real date time value?

#5

pushed your data to influxdb, definitely an issue with that timestamp I had to do some manipulation on it.

image
image1157×697 42.2 KB

#6

I can confirm that the data is correctly read from Loki.

I have already successfully used influxdb + grafana locally, but I would like to try grafana cloud that’s why I was trying Loki

1 Like