Help needed with Grafana + Loki + Promtail for CSV log graphs

Grafana Loki
1

Hello,

After a few days of poor progress, I thought I better check here if I should keep going or if the approach is altogether wrong.

Goal: Visualization of some hardware monitoring metrics over time, in Grafana. For now, I’m trying to get working a “hello world” with only four metrics. In total, there are 400+ metrics, but one step at a time.
Logfile: CSV log, mounted from a network share, output is from HWInfo64

From what I read, there are two main options:

  1. Promtail to use a regex pipeline stage to parse log lines to metrics, + Loki + Grafana
  2. Infinity plugin (for the CSV parsing) + web server to serve local log files to Infinity + Loki + Grafana

I am working with 1. I don’t know how to configure promtail to take column data as metrics.

Thank you for your help.

promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: hwinfo
    static_configs:
      - targets:
          - localhost
        labels:
          job: hwinfo
          __path__: /var/log/hwinfo/*.csv
    pipeline_stages:
      - regex:
          expression: "^(?P<DateTime>.*?),(?P<VirtualMemoryCommitted>.*?),(?P<VirtualMemoryAvailable>.*?),(?P<VirtualMemoryLoad>.*?),(?P<PhysicalMemoryUsed>.*?)"
      - labels:
          DateTime:
          VirtualMemoryCommitted:
          VirtualMemoryAvailable:
          VirtualMemoryLoad:
          PhysicalMemoryUsed:

docker-compose.yaml

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /mnt/logs:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki
    volumes:
      - grafana_data:/var/lib/grafana

networks:
  loki:
    driver: bridge

volumes:
  grafana_data:

station1.csv
Date/Time,"Virtual Memory Committed [MB]","Virtual Memory Available [MB]","Virtual Memory Load [%]","Physical Memory Used [MB]","Physical Memory Available [MB]","Physical Memory Load [%]","Page File Usage [%]","Core VIDs (avg) [V]","P-core 0 VID [V]","P-core 1 VID [V]","P-core 2 VID [V]","P-core 3 VID [V]","P-core 4 VID [V]","P-core 5 VID [V]","E-core 6 VID [V]","E-core 7 VID [V]","E-core 8 VID [V]","E-core 9 VID [V]","E-core 10 VID [V]","E-core 11 VID [V]","E-core 12 VID [V]","E-core 13 VID [V]","SA VID [V]","Core Clocks (avg) [MHz]","P-core 0 Clock [MHz]","P-core 1 Clock [MHz]","P-core 2 Clock [MHz]","P-core 3 Clock [MHz]","P-core 4 Clock [MHz]","P-core 5 Clock [MHz]","E-core 6 Clock [MHz]","E-core 7 Clock [MHz]","E-core 8 Clock [MHz]","E-core 9 Clock [MHz]","E-core 10 Clock [MHz]","E-core 11 Clock [MHz]","E-core 12 Clock [MHz]","E-core 13 Clock [MHz]","Bus Clock [MHz]","Ring/LLC Clock [MHz]","Core Effective Clocks (avg) [MHz]","P-core 0 T0 Effective Clock [MHz]","P-core 0 T1 Effective Clock [MHz]","P-core 1 T0 Effective Clock [MHz]","P-core 1 T1 Effective Clock [MHz]","P-core 2 T0 Effective Clock [MHz]","P-core 2 T1 Effective Clock [MHz]","P-core 3 T0 Effective Clock [MHz]","P-core 3 T1 Effective Clock [MHz]","P-core 4 T0 Effective Clock [MHz]","P-core 4 T1 Effective Clock [MHz]","P-core 5 T0 Effective Clock [MHz]","P-core 5 T1 Effective Clock [MHz]","E-core 6 T0 Effective Clock [MHz]","E-core 7 T0 Effective Clock [MHz]","E-core 8 T0 Effective Clock [MHz]","E-core 9 T0 Effective Clock [MHz]","E-core 10 T0 Effective Clock [MHz]","E-core 11 T0 Effective Clock [MHz]","E-core 12 T0 Effective Clock [MHz]","E-core 13 T0 Effective Clock [MHz]","Average Effective Clock [MHz]","Core Usage (avg) [%]","P-core 0 T0 Usage [%]","P-core 0 T1 Usage [%]","P-core 1 T0 Usage [%]","P-core 1 T1 Usage [%]","P-core 2 T0 Usage [%]","P-core 2 T1 Usage [%]","P-core 3 T0 Usage [%]","P-core 3 T1 Usage [%]","P-core 4 T0 Usage [%]","P-core 4 T1 Usage [%]","P-core 5 T0 Usage [%]","P-core 5 T1 Usage [%]","E-core 6 T0 Usage [%]","E-core 7 T0 Usage [%]","E-core 8 T0 Usage [%]","E-core 9 T0 Usage [%]","E-core 10 T0 Usage [%]","E-core 11 T0 Usage [%]","E-core 12 T0 Usage [%]","E-core 13 T0 Usage [%]","Max CPU/Thread Usage [%]","Total CPU Usage [%]","On-Demand Clock Modulation [%]","Core Utility (avg) [%]","P-core 0 T0 Utility [%]","P-core 0 T1 Utility [%]","P-core 1 T0 Utility [%]","P-core 1 T1 Utility [%]","P-core 2 T0 Utility [%]","P-core 2 T1 Utility [%]","P-core 3 T0 Utility [%]","P-core 3 T1 Utility [%]","P-core 4 T0 Utility [%]","P-core 4 T1 Utility [%]","P-core 5 T0 Utility [%]","P-core 5 T1 Utility [%]","E-core 6 T0 Utility [%]","E-core 7 T0 Utility [%]","E-core 8 T0 Utility [%]","E-core 9 T0 Utility [%]","E-core 10 T0 Utility [%]","E-core 11 T0 Utility [%]","E-core 12 T0 Utility [%]","E-core 13 T0 Utility [%]","Total CPU Utility [%]","Core Ratios (avg) [x]","P-core 0 Ratio [x]","P-core 1 Ratio [x]","P-core 2 Ratio [x]","P-core 3 Ratio [x]","P-core 4 Ratio [x]","P-core 5 Ratio [x]","E-core 6 Ratio [x]","E-core 7 Ratio [x]","E-core 8 Ratio [x]","E-core 9 Ratio [x]","E-core 10 Ratio [x]","E-core 11 Ratio [x]","E-core 12 Ratio [x]","E-core 13 Ratio [x]","Uncore Ratio [x]","Core Temperatures (avg) [�C]","P-core 0 [�C]","P-core 1 [�C]","P-core 2 [�C]","P-core 3 [�C]","P-core 4 [�C]","P-core 5 [�C]","E-core 6 [�C]","E-core 7 [�C]","E-core 8 [�C]","E-core 9 [�C]","E-core 10 [�C]","E-core 11 [�C]","E-core 12 [�C]","E-core 13 [�C]","P-core 0 Distance to TjMAX [�C]","P-core 1 Distance to TjMAX [�C]","P-core 2 Distance to TjMAX [�C]","P-core 3 Distance to TjMAX [�C]","P-core 4 Distance to TjMAX [�C]","P-core 5 Distance to TjMAX [�C]","E-core 6 Distance to TjMAX [�C]","E-core 7 Distance to TjMAX [�C]","E-core 8 Distance to TjMAX [�C]","E-core 9 Distance to TjMAX [�C]","E-core 10 Distance to TjMAX [�C]","E-core 11 Distance to TjMAX [�C]","E-core 12 Distance to TjMAX [�C]","E-core 13 Distance to TjMAX [�C]","CPU Package [�C]","Core Max [�C]","Core Thermal Throttling (avg) [Yes/No]","P-core 0 Thermal Throttling [Yes/No]","P-core 1 Thermal Throttling [Yes/No]","P-core 2 Thermal Throttling [Yes/No]","P-core 3 Thermal Throttling [Yes/No]","P-core 4 Thermal Throttling [Yes/No]","P-core 5 Thermal Throttling [Yes/No]","E-core 6 Thermal Throttling [Yes/No]","E-core 7 Thermal Throttling [Yes/No]","E-core 8 Thermal Throttling [Yes/No]","E-core 9 Thermal Throttling [Yes/No]","E-core 10 Thermal Throttling [Yes/No]","E-core 11 Thermal Throttling [Yes/No]","E-core 12 Thermal Throttling [Yes/No]","E-core 13 Thermal Throttling [Yes/No]","Core Critical Temperature (avg) [Yes/No]","P-core 0 Critical Temperature [Yes/No]","P-core 1 Critical Temperature [Yes/No]","P-core 2 Critical Temperature [Yes/No]","P-core 3 Critical Temperature [Yes/No]","P-core 4 Critical Temperature [Yes/No]","P-core 5 Critical Temperature [Yes/No]","E-core 6 Critical Temperature [Yes/No]","E-core 7 Critical Temperature [Yes/No]","E-core 8 Critical Temperature [Yes/No]","E-core 9 Critical Temperature [Yes/No]","E-core 10 Critical Temperature [Yes/No]","E-core 11 Critical Temperature [Yes/No]","E-core 12 Critical Temperature [Yes/No]","E-core 13 Critical Temperature [Yes/No]","Core Power Limit Exceeded (avg) [Yes/No]","P-core 0 Power Limit Exceeded [Yes/No]","P-core 1 Power Limit Exceeded [Yes/No]","P-core 2 Power Limit Exceeded [Yes/No]","P-core 3 Power Limit Exceeded [Yes/No]","P-core 4 Power Limit Exceeded [Yes/No]","P-core 5 Power Limit Exceeded [Yes/No]","E-core 6 Power Limit Exceeded [Yes/No]","E-core 7 Power Limit Exceeded [Yes/No]","E-core 8 Power Limit Exceeded [Yes/No]","E-core 9 Power Limit Exceeded [Yes/No]","E-core 10 Power Limit Exceeded [Yes/No]","E-core 11 Power Limit Exceeded [Yes/No]","E-core 12 Power Limit Exceeded [Yes/No]","E-core 13 Power Limit Exceeded [Yes/No]","Package/Ring Thermal Throttling [Yes/No]","Package/Ring Critical Temperature [Yes/No]","Package/Ring Power Limit Exceeded [Yes/No]","CPU Package [�C]","CPU IA Cores [�C]","CPU GT Cores (Graphics) [�C]","iGPU VID [V]","VDDQ TX Voltage [V]","VR VCC Current (SVID IOUT) [A]","CPU Package Power [W]","IA Cores Power [W]","GT Cores Power [W]","System Agent Power [W]","Rest-of-Chip Power [W]","PL1 Power Limit (Static) [W]","PL1 Power Limit (Dynamic) [W]","PL2 Power Limit (Static) [W]","PL2 Power Limit (Dynamic) [W]","GPU Clock [MHz]","GPU D3D Usage [%]","GPU D3D Utilizations (avg) [%]","GPU Video Decode 0 Usage [%]","GPU Video Decode 1 Usage [%]","GPU Video Processing 0 Usage [%]","GPU D3D Memory Dynamic [MB]","Current cTDP Level []","Core C0 Residency (avg) [%]","P-core 0 T0 C0 Residency [%]","P-core 0 T1 C0 Residency [%]","P-core 1 T0 C0 Residency [%]","P-core 1 T1 C0 Residency [%]","P-core 2 T0 C0 Residency [%]","P-core 2 T1 C0 Residency [%]","P-core 3 T0 C0 Residency [%]","P-core 3 T1 C0 Residency [%]","P-core 4 T0 C0 Residency [%]","P-core 4 T1 C0 Residency [%]","P-core 5 T0 C0 Residency [%]","P-core 5 T1 C0 Residency [%]","E-core 6 T0 C0 Residency [%]","E-core 7 T0 C0 Residency [%]","E-core 8 T0 C0 Residency [%]","E-core 9 T0 C0 Residency [%]","E-core 10 T0 C0 Residency [%]","E-core 11 T0 C0 Residency [%]","E-core 12 T0 C0 Residency [%]","E-core 13 T0 C0 Residency [%]","Core C1 Residency (avg) [%]","P-core 0 C1 Residency [%]","P-core 1 C1 Residency [%]","P-core 2 C1 Residency [%]","P-core 3 C1 Residency [%]","P-core 4 C1 Residency [%]","P-core 5 C1 Residency [%]","E-core 6 C1 Residency [%]","E-core 7 C1 Residency [%]","E-core 8 C1 Residency [%]","E-core 9 C1 Residency [%]","E-core 10 C1 Residency [%]","E-core 11 C1 Residency [%]","E-core 12 C1 Residency [%]","E-core 13 C1 Residency [%]","Core C6 Residency (avg) [%]","P-core 0 C6 Residency [%]","P-core 1 C6 Residency [%]","P-core 2 C6 Residency [%]","P-core 3 C6 Residency [%]","P-core 4 C6 Residency [%]","P-core 5 C6 Residency [%]","E-core 6 C6 Residency [%]","E-core 7 C6 Residency [%]","E-core 8 C6 Residency [%]","E-core 9 C6 Residency [%]","E-core 10 C6 Residency [%]","E-core 11 C6 Residency [%]","E-core 12 C6 Residency [%]","E-core 13 C6 Residency [%]","Core C7 Residency (avg) [%]","P-core 0 C7 Residency [%]","P-core 1 C7 Residency [%]","P-core 2 C7 Residency [%]","P-core 3 C7 Residency [%]","P-core 4 C7 Residency [%]","P-core 5 C7 Residency [%]","Memory Clock [MHz]","Memory Clock Ratio [x]","Tcas [T]","Trcd [T]","Trp [T]","Tras [T]","Trc [T]","Trfc [T]","Command Rate [T]","Gear Mode []","IA Limit Reasons (avg) [Yes/No]","IA: PROCHOT [Yes/No]","IA: Thermal Event [Yes/No]","IA: Residency State Regulation [Yes/No]","IA: Running Average Thermal Limit [Yes/No]","IA: VR Thermal Alert [Yes/No]","IA: VR TDC [Yes/No]","IA: Electrical Design Point/Other (ICCmax,PL4,SVID,DDR RAPL) [Yes/No]","IA: Package-Level RAPL/PBM PL1 [Yes/No]","IA: Package-Level RAPL/PBM PL2,PL3 [Yes/No]","IA: Max Turbo Limit [Yes/No]","IA: Turbo Attenuation (MCT) [Yes/No]","IA: Thermal Velocity Boost [Yes/No]","GT Limit Reasons (avg) [Yes/No]","GT: PROCHOT [Yes/No]","GT: Thermal Event [Yes/No]","GT: DDR RAPL [Yes/No]","GT: Residency State Regulation [Yes/No]","GT: Running Average Thermal Limit [Yes/No]","GT: VR Thermal Alert [Yes/No]","GT: VR TDC [Yes/No]","GT: Max VR Voltage, ICCmax, PL4 [Yes/No]","GT: Domain-Level PBM PLGT [Yes/No]","GT: Package-Level RAPL/PBM PL1 [Yes/No]","GT: Package-Level RAPL/PBM PL2,PL3 [Yes/No]","GT: Inefficient Operation [Yes/No]","GT: Fuses limit [Yes/No]","Ring Limit Reasons (avg) [Yes/No]","RING: PROCHOT [Yes/No]","RING: Thermal Event [Yes/No]","RING: DDR RAPL [Yes/No]","RING: Residency State Regulation [Yes/No]","RING: Running Average Thermal Limit [Yes/No]","RING: VR Thermal Alert [Yes/No]","RING: VR TDC [Yes/No]","RING: Max VR Voltage, ICCmax, PL4 [Yes/No]","RING: Package-Level RAPL/PBM PL1 [Yes/No]","RING: Package-Level RAPL/PBM PL2,PL3 [Yes/No]","CPU [�C]","AUXTIN3 [�C]","CPU (PECI) [�C]","Vcore [V]","VIN1 [V]","+3.3V (AVCC) [V]","+3.3V (3VCC) [V]","VIN0 [V]","3VSB [V]","VBAT [V]","VTT [V]","VIN2 [V]","VIN3 [V]","VIN7 [V]","VIN9 [V]","CPUFANIN0 [RPM]","AUXFANIN4 [RPM]","Chassis Intrusion [Yes/No]","TZ.TZ00 [�C]","Drive Temperature [�C]","Drive Temperature [�C]","Drive Temperature [�C]","Drive Temperature [�C]","Drive Temperature [�C]","Drive Temperature 2 [�C]","Drive Temperature 3 [�C]","Drive Failure [Yes/No]","Drive Warning [Yes/No]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Read Activity [%]","Write Activity [%]","Total Activity [%]","Read Rate [MB/s]","Write Rate [MB/s]","Read Total [MB]","Write Total [MB]","Framerate (Presented) [FPS]","Framerate (Displayed) [FPS]","Frame Time [ms]","GPU Busy [ms]","GPU Wait [ms]","CPU Busy [ms]","CPU Wait [ms]","Total DL [MB]","Total UP [MB]","Current DL rate [KB/s]","Current UP rate [KB/s]","Total DL [MB]","Total UP [MB]","Current DL rate [KB/s]","Current UP rate [KB/s]","Total DL [MB]","Total UP [MB]","Current DL rate [KB/s]","Current UP rate [KB/s]","Total Errors []",
2024-10-27 18:33:08.94803Z,22463,12090,65.0,20148,12357,61.9,2.9,0.711,0.719,0.715,0.704,0.704,0.711,0.711,0.706,0.710,0.710,0.712,0.712,0.712,0.711,0.711,1.124,833.7,798.0,798.0,798.0,897.8,997.6,997.6,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,99.8,997.6,27.4,36.2,0.9,13.3,0.3,124.1,0.7,164.2,15.3,9.1,0.1,36.2,2.8,36.2,36.1,2.4,7.9,2.3,0.2,20.0,40.6,30.4,2.0,2.1,0.0,0.7,0.0,11.5,0.1,12.8,2.7,0.4,0.0,2.4,0.8,2.3,1.6,0.0,0.0,0.2,0.0,0.5,2.0,12.8,2.0,100.0,1.4,1.8,0.1,0.7,0.0,7.1,0.0,9.0,1.1,0.4,0.0,2.0,0.2,1.6,1.5,0.1,0.3,0.1,0.0,1.0,1.7,1.4,8.4,8.0,8.0,8.0,9.0,10.0,10.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,10.0,31,30,29,31,26,28,30,32,34,34,34,32,32,31,31,70,71,69,74,72,70,68,66,66,66,68,68,69,69,32,34,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,No,No,Yes,35,33,32,0.255,1.200,9.375,8.806,1.921,0.165,6.337,0.064,65.0,65.0,95.0,95.0,300.0,1.2,0.0,0.0,0.0,0.0,1866,0,1.5,1.7,0.1,0.8,0.0,7.9,0.1,8.6,1.4,0.3,0.0,1.6,0.3,2.2,1.5,0.2,0.3,0.2,0.0,1.1,1.9,0.8,1.7,1.2,2.8,1.7,0.2,0.9,0.6,0.3,0.0,0.0,0.0,0.0,1.2,0.6,56.2,0.0,0.0,0.0,0.0,0.0,0.0,96.5,97.8,99.7,99.7,99.7,99.9,97.0,96.9,90.1,94.0,97.1,78.9,77.4,99.0,94.3,1596.1,16.00,19,19,19,43,62,560,2,2,Yes,No,No,No,No,No,No,No,Yes,Yes,Yes,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,No,No,No,No,No,No,No,Yes,No,No,35.0,39.0,33.0,0.720,1.376,3.408,3.344,1.000,3.408,3.184,0.512,1.816,0.096,0.760,0.096,744,450,No,27.9,28,52,40,42,37,37,48,No,No,0.0,1.5,1.5,0.000,0.015,16170,7970,0.0,0.0,0.0,0.000,0.006,16169,7582,0.0,0.0,0.0,0.000,0.000,1435162,538094,0.0,0.1,0.1,0.000,0.012,0,58,0.0,0.4,0.4,0.000,0.047,35245,15832,0.0,1.5,1.5,0.000,0.022,32340,15552,0.0,0.0,0.00,0.00,0.00,0.00,0.00,0,0,0.000,0.000,564613,65771,1.893,1.128,0,0,0.000,0.000,0,
2024-10-27 18:33:13.94203Z,22461,12092,65.0,20147,12358,61.9,2.9,0.736,0.719,0.711,0.710,0.710,0.710,0.703,0.678,1.173,0.691,0.691,0.703,0.703,0.703,0.707,1.124,961.9,698.3,698.3,698.3,698.3,698.3,698.3,798.0,3691.0,798.0,798.0,798.0,798.0,798.0,798.0,99.8,997.6,14.6,9.8,1.6,3.9,0.2,91.8,0.4,0.5,104.7,3.8,1.1,0.1,22.0,15.1,7.4,1.0,0.3,16.9,0.7,1.6,9.6,16.5,1.7,1.4,0.0,1.0,0.1,10.7,0.2,0.0,14.1,0.0,0.0,0.1,3.0,1.0,0.2,0.2,0.2,1.6,0.2,0.3,0.7,14.1,1.7,100.0,0.9,0.7,0.1,0.3,0.0,5.9,0.0,0.0,7.5,0.2,0.0,0.0,1.4,0.7,0.3,0.1,0.0,1.1,0.0,0.1,0.5,0.9,9.6,7.0,7.0,7.0,7.0,7.0,7.0,8.0,37.0,8.0,8.0,8.0,8.0,8.0,8.0,10.0,31,29,32,31,29,29,33,32,32,32,32,31,32,32,32,71,68,69,71,71,67,68,68,68,68,69,68,68,68,34,33,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,No,No,Yes,38,38,31,0.255,1.200,13.125,8.055,1.173,0.164,6.337,0.062,65.0,65.0,95.0,95.0,300.0,0.9,0.0,0.0,0.0,0.0,1844,0,1.3,1.0,0.2,0.4,0.0,6.9,0.1,0.0,8.4,0.4,0.1,0.0,2.1,1.5,0.7,0.1,0.0,2.1,0.1,0.2,1.2,0.3,0.0,0.0,1.1,0.9,0.0,0.0,0.0,0.0,0.0,0.0,1.6,0.0,0.0,0.0,56.1,0.0,0.0,0.0,0.0,0.0,0.0,97.9,99.0,99.8,99.9,91.9,99.9,99.7,98.1,91.9,96.8,98.8,82.7,78.7,99.2,95.2,1596.1,16.00,19,19,19,43,62,560,2,2,Yes,No,No,No,No,No,No,No,Yes,Yes,Yes,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,No,No,No,No,No,No,No,Yes,No,No,35.0,39.0,33.0,0.760,1.384,3.408,3.344,1.000,3.408,3.184,0.512,1.816,0.096,0.760,0.096,749,451,No,27.9,28,52,40,42,37,37,45,No,No,0.0,1.2,1.2,0.000,0.013,16170,7970,0.0,0.0,0.0,0.000,0.000,16169,7582,0.0,0.0,0.0,0.000,0.000,1435162,538094,0.0,0.1,0.1,0.000,0.002,0,58,0.0,0.2,0.2,0.000,0.011,35245,15832,0.0,1.2,1.2,0.000,0.014,32340,15552,0.0,0.0,0.00,0.00,0.00,0.00,0.00,0,0,0.000,0.000,564613,65771,5.045,3.563,0,0,0.000,0.000,0,
2024-10-27 18:33:18.94103Z,22463,12091,65.0,20147,12358,61.9,2.9,0.738,0.709,0.713,0.708,0.706,0.713,0.708,0.705,1.183,0.688,0.688,0.701,0.706,0.701,0.707,1.124,1090.2,897.8,897.8,897.8,897.8,997.6,997.6,897.8,3691.0,897.8,798.0,798.0,798.0,897.8,897.8,99.8,997.6,16.9,16.1,4.4,6.5,0.3,63.6,0.3,2.0,102.5,7.2,0.3,1.9,41.0,20.8,10.0,5.1,2.1,24.2,0.9,2.2,25.9,17.8,2.0,2.0,0.6,0.7,0.1,9.0,0.1,0.2,13.9,1.2,0.1,0.0,3.5,1.6,0.9,0.4,0.2,2.5,0.1,0.2,2.1,13.9,2.0,100.0,1.0,0.9,0.2,0.3,0.0,4.4,0.0,0.1,6.9,0.3,0.0,0.1,2.3,0.9,0.4,0.2,0.1,1.4,0.0,0.1,1.1,1.0,10.9,9.0,9.0,9.0,9.0,10.0,10.0,9.0,37.0,9.0,8.0,8.0,8.0,9.0,9.0,10.0,31,30,29,31,30,26,33,34,34,32,32,32,31,31,30,70,71,69,70,74,67,66,66,68,68,68,69,69,70,33,34,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,No,No,Yes,34,34,31,0.255,1.200,16.875,8.042,1.169,0.165,6.324,0.066,65.0,65.0,95.0,95.0,300.0,1.5,0.0,0.0,0.0,0.0,1844,0,1.4,1.5,0.3,0.6,0.0,5.2,0.0,0.2,8.1,0.5,0.0,0.1,2.4,1.5,0.8,0.3,0.2,2.8,0.1,0.2,2.1,0.2,0.3,0.0,0.7,0.6,0.0,0.2,0.0,0.0,0.0,0.0,0.9,0.0,0.0,0.0,56.0,0.0,0.0,0.0,0.0,0.0,0.0,97.8,98.7,99.5,99.8,91.0,99.9,99.7,97.4,92.3,96.1,98.7,85.8,79.5,99.2,94.5,1596.1,16.00,19,19,19,43,62,560,2,2,Yes,No,No,No,No,No,No,No,Yes,Yes,Yes,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,No,No,No,No,No,No,No,Yes,No,No,35.0,39.0,33.0,0.720,1.384,3.408,3.344,1.000,3.408,3.184,0.512,1.816,0.096,0.760,0.096,749,451,No,27.9,28,52,40,42,37,37,45,No,No,0.0,0.8,0.8,0.000,0.017,16170,7970,0.0,0.0,0.0,0.000,0.000,16169,7582,0.0,0.0,0.0,0.000,0.000,1435162,538094,0.0,0.1,0.1,0.000,0.003,0,58,0.0,0.9,0.9,0.000,0.026,35245,15832,0.0,0.8,0.8,0.000,0.018,32340,15552,0.0,0.0,0.00,0.00,0.00,0.00,0.00,0,0,0.000,0.000,564613,65771,2.058,5.337,0,0,0.000,0.000,0,
2024-10-27 18:33:23.92903Z,22464,12089,65.0,20148,12357,61.9,2.9,0.709,0.713,0.712,0.710,0.710,0.711,0.711,0.706,0.706,0.708,0.708,0.712,0.707,0.707,0.707,1.124,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,798.0,99.8,997.6,21.3,20.7,2.4,11.2,1.2,119.5,8.8,0.8,136.7,5.1,0.3,0.9,65.6,13.0,15.6,5.7,1.7,5.6,1.0,1.2,8.6,25.6,2.1,1.6,0.0,0.6,0.0,11.6,0.1,0.3,15.8,0.0,0.0,0.3,5.3,1.5,1.4,0.8,0.1,0.5,0.0,0.0,1.2,15.8,2.1,100.0,1.2,1.1,0.2,0.6,0.1,6.8,0.4,0.1,8.4,0.2,0.0,0.0,3.4,0.6,0.7,0.2,0.1,0.3,0.1,0.1,0.4,1.2,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,8.0,10.0,31,29,34,30,26,27,33,32,32,32,32,31,31,31,31,71,66,70,74,73,67,68,68,68,68,69,69,69,69,33,34,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,No,No,Yes,36,36,32,0.255,1.200,15.000,8.297,1.431,0.166,6.312,0.070,65.0,65.0,95.0,95.0,300.0,2.1,0.0,0.0,0.0,0.0,1855,0,1.6,1.4,0.2,0.9,0.1,8.6,0.2,0.1,10.5,0.4,0.0,0.1,4.2,1.3,1.2,0.5,0.2,0.6,0.1,0.1,1.0,0.4,0.3,0.0,2.0,2.8,0.0,0.0,0.0,0.0,0.0,0.0,0.3,0.0,0.0,0.0,56.5,0.0,0.0,0.0,0.0,0.0,0.0,98.1,98.3,99.3,99.8,98.1,99.8,99.8,98.3,89.9,95.9,97.8,80.0,74.4,99.2,92.1,1596.1,16.00,19,19,19,43,62,560,2,2,Yes,No,No,No,No,No,No,No,Yes,Yes,Yes,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,No,Yes,No,No,No,No,No,No,No,Yes,No,No,35.0,39.0,33.0,0.736,1.376,3.408,3.328,1.000,3.408,3.184,0.512,1.816,0.096,0.760,0.096,749,449,No,27.9,28,52,40,42,37,37,45,No,No,0.0,1.6,1.6,0.000,0.016,16170,7971,0.0,0.0,0.0,0.000,0.000,16169,7582,0.0,0.0,0.0,0.000,0.000,1435162,538094,0.0,0.1,0.1,0.000,0.005,0,58,0.0,0.1,0.1,0.000,0.003,35245,15832,0.0,1.6,1.6,0.000,0.017,32340,15552,0.0,0.0,0.00,0.00,0.00,0.00,0.00,0,0,0.000,0.000,564613,65771,7.196,19.372,0,0,0.000,0.000,0,
2

Looking at your sample data I think it’s better suited for a metrics platform than logging platform. Imagine following logline:

2024-10-27 18:33:08.94803Z,22463,12090,65.0,20148,12357

How do you envision this in Loki? If you are sending the entire line then it’s not very usable because you don’t have the header metadata. If you are sending it parsed with each column as a label then it’s literally becoming a metric, no reason to use Loki because you likely will get very poor performance due to label cardinality. You can still do it if you really want, but my recommendation would be to forward these to a metrics platform such as mimir or prometheus.

If I were to attempt this I would probably try one of the following solution:

  1. Pick a metrics agent that is capable of parsing CSV file (i believe telegraf can), get rid of the header of the CSV file, then forward metrics to mimir.
  2. Write a script (python, for example) and transform the CSV file into JSON file, with each key/value corresponding to a column. Probably need to cleanse the column header a bit by removing space for easier parsing, then send to Loki via Alloy. This allows you to work around cardinality issue (each column, instead of being turned into alabel, is now a json key/value pair).
3

Thank you for the feedback, I will try that route with Mimir or Prometheus.