Nvidia GPU State Timeline

This finished panel should look something like this

  • How are you trying to achieve it?

So far i have tried to create the state time line with the following flux query

from(bucket: "${bucket}")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["host"] == "${hostname}")
  |> filter(fn: (r) => r["name"] == "${GPU}")
  |> filter(fn: (r) => r["_measurement"] == "nvidia_smi")
  |> filter(fn: (r) => r["_field"] == "fan_speed")
  |> group(columns: ["host", "name", "uuid"])
  • What happened?

This is the result

  • What did you expect to happen?

That the state would be shown as a single row and look like the first image in the post.

Please provide a good amount of sample data?

nvidia_smi,compute_mode=Default,host=8218cf,index=0,name=GeForce\ GTX\ 1070,pstate=P2,uuid=GPU-823bc202-6279-6f2c-d729-868a30f14d96 fan_speed=100i,memory_free=7563i,memory_total=8112i,memory_used=549i,temperature_gpu=53i,utilization_gpu=100i,utilization_memory=90i 1523991122000000000

Sure

#group,false,false,true,true,false,false,true,true,true,true,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,long,string,string,string,string,string,string,string,string
#default,_result,,,,,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,compute_mode,host,index,name,pstate,uuid
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:06:30Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:06:40Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:06:50Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:00Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:10Z,32,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:20Z,39,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:30Z,44,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:40Z,47,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:07:50Z,50,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:00Z,52,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:10Z,54,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:20Z,56,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:30Z,58,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:40Z,59,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:08:50Z,60,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:00Z,61,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:10Z,62,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:20Z,63,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:30Z,64,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,0,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:40Z,64,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P0,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,1,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T14:09:50Z,65,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P3,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:32:10Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:32:20Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:32:30Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:32:40Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:32:50Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:33:00Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:33:10Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:33:20Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:33:30Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
,,2,2022-06-22T13:01:22.119642866Z,2022-06-22T16:01:22.119642866Z,2022-06-22T13:33:40Z,30,fan_speed,nvidia_smi,Default,dlap-4000,0,Quadro RTX 4000,P8,GPU-66f485fd-e844-1e36-f14f-bcf0602bb604
1 Like

Something like this
You will have to play with the time grouping on the Query Options

Check this one out also.
https://play.grafana.org/d/qD-rVv6Mz/6-state-timeline-and-status-history?orgId=1

from(bucket: "nvidia_smi")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "nvidia_smi")
  |> filter(fn: (r) => r["_field"] == "fan_speed" )
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> group()  
  |> drop(columns: ["_start", "_stop", "_measurement","type"])

Use value mapping to color changes.

1 Like

Thanks for your help, i have not used pivot before.

I managed to create the panel using the following query.

from(bucket: "nvidia_smi")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "nvidia_smi")
  |> filter(fn: (r) => r["_field"] == "utilization_gpu" )
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> group()  
  |> drop(columns: ["_start", "_stop", "_measurement", "type", "index", "host", "name", "uuid", "compute_mode", "utilization_gpu"])

1 Like