Basically, we have networking devices which will stream every 1 second to our collector. We process it and convert it into prometheus metric format and send it via push gateway. We are scraping the metrics every 250ms on prometheus job… When we look at records on prometheus console we have the messages with value along with the right timestamp.
But, when we plot a graph on Grafana: we are not seeing the actual timestamp of message on each datapoint. Data points are getting replicated in because of step interval and resolution. Though we set 1 sec as scraping interval on grafana and 1/1 as resolution with 1sec as step interval. But when we look at data points, its showing every second through there are few dropped or missed collection from networking device(which is streaming the data). Basically, it is happening for all kinds of data which we store it on prometheus.
The no.of data points on prometheus console and grafana are different for the same query it is because of step interval and resolution.
Can you please help us to get only the actual timestamped data points on grafana.