I am trying to plot plot transient metrics such as Job execution time or queue time that are calculated based on the difference between two timestamps (e.g. hpc_job_execution_time=hpc_job_execution_completion_time-hpc_job_execution_start_time)
Say I record a job_execution_time of 70 second. If we have length of the job in seconds on the y axis and calendar time on x axis, Grafana plots this metric as a straight line that starts from the the calendar time the job has completed and continues going on forever. See screenshot below.
Is it possible to show the value of this metric as a single point (e.g. as if we were plotting it in a Cartesian coordinate system, with unique x and y coordinate values)?
The data source for Grafana is a Prometheus Server. Since this metric is transient and cannot be scraped by Prormetheus (using for example an exporter), I am using the Prometheus Pushgateway in order to push the job timestamps (later used to calculate the job execution time) to Prometheus.