Parse CSV to metric

Hi,
I would like to parse a csv and plot its content in Grafana.

I have a csv like the following:

date,key0,key1
20220610t133612,5260,3.05
20220610t133712,5260,78.4
...

Converted it in a list of json:

{"date": "20220610t133612", "key0": "5260", "key1": "3.05"}
{"date": "20220610t133712", "key0": "5260", "key1": "78.4"}
...

Use the following config.yaml:

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://<redacted>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
  - json:
      expressions:
        output: data
        stream: stream
        timestamp: date
        key0:
        key1:
  - timestamp:
      source: timestamp
      format: "20060102t150405"
  static_configs:
  - targets:
      - localhost
    labels:
      job: data
      __path__: /var/log/*.log

Send data using promtail:

podman run \
    --name promtail \                                                                                                                                              
    --volume "$PWD/config.yaml:/etc/promtail/config.yaml" \                                                                                                        
    --volume "$PWD:/var/log" \                                                                                                                                                                                                                                                                 
    grafana/promtail:master \                                                                                                                                      
       -config.file=/etc/promtail/config.yaml \                                                                                                                   
       -log.level=debug    

If I try to query the data for a new Panel:

{job="data"} | json | line_format "{{.ping}}"

I can see the data only in a table format.
It does not allow me to plot the timeseries.

Any ideas?

Thanks

Welcome

You dont want grafana to read the csv directly and plot it?

Thanks for the reply.

That would be great.
I have already tried the CSV data sources but it is a bit clunky and slow, that’s why I preferred to ingest the file in Loki.

Do you have a better suggestion?

1 Like

this is a custom time format so what data type is it being saved in Loki? is it being save as a real date time value?

pushed your data to influxdb, definitely an issue with that timestamp I had to do some manipulation on it.

I can confirm that the data is correctly read from Loki.

I have already successfully used influxdb + grafana locally, but I would like to try grafana cloud that’s why I was trying Loki

1 Like