Abnormal peaks in time series - how to avoid them?

Grafana InfluxDB
1

When displaying time series from influxdb i often get abnormal peaks of values exceeding the regular ones by thousand and more times. This is caused by some issues on the network or systems.
How do you usually avoid this? The image below displays an example.

image
image1755Ă—383 36 KB

2

Depending on what your querry is, perhaps you could include a and _value < Treshold_value.

3

That probably makes sense.
Below is sample with query, but i do not quite understand how to proper reference a result of the query. Putting just … AND _value < 10000000000000 … gives no data.

image
image1172Ă—659 32.5 KB

4

Not sure how to do that in influxql when using derivative functions. Tried some sub queries, but failed in my attempts. But if you have the opportunity to switch to flux, this here works:

from(bucket:"fmbb/autogen")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "wwan0" and
 r._field == "rx_bytes" and
 r.serial_number =~ /^$serial$/ 
 )
|> aggregateWindow(every: v.windowPeriod, fn: last)
|> derivative(nonNegative: true)
|> filter(fn: (r) => r._value < 1024*1024)
|> map(fn:(r) => ({ _time:r._time,rx:r._value * 8.0}))

The key here is the |> filter(fn: (r) => r._value < 1024*1024) line. Since the data in my example is bytes I here cap at one megabyte/s.

Without cap

image
image1267Ă—1047 141 KB

With cap

image
image1287Ă—1034 115 KB

5

And as I posted the flux solution I remembered how to do it in influxql as well. Had the order wrong in my initial attempts.

SELECT max("rxbytes")  FROM 
	(SELECT non_negative_derivative(mean("rx_bytes"), 1s) *8 as rxbytes FROM "wwan0" WHERE ("serial_number" =~ /^$serial$/) AND $timeFilter GROUP BY time(10s) fill(null))
where "rxbytes" < 10000000 GROUP BY time($__interval)

The inner select is what you have already, but you perhaps need to select different time intervals in the inner GROUP BY.

1 Like
6

Yup, that works with influxql.
Many thanks for your help, my friend))

1 Like
7

Also (in case you did not know about it), Grafana introduced soft min and max to address this exact type of issue. Change axis display | Grafana Labs

1 Like
8

It is good to also know about this way to solve the issue. Kudos for recommending it.

That being said, personally I’m not so found of “hybrid solutions”. Meaning doing part of the limits/maths/joins in the graphing tool and parts in the query tool. The reason is that I find it hard to debug later on. Meaning when I inherit someone else’s graphs or come back to mine a year later. Remembering to not only look at the query, but also hunt around in the gui throws me off every time.

9

BTW, is it possible to sum two or more time series with influxql (or may be flux which i have not upgraded to yet) rather than stacking graphs or executing an expressions on them ?
For example two metrics - for uplink and downlink traffic from the examples above?

10

Suggest you open a new topic so its easier to search questions/answers for other forum users. :slight_smile: