Hide metrics in past days

Hello,

I have installed Collectd + Riemann + Graphite + Grafana some months ago. I have configured some basic metrics, such as cpu, mem and disk. It works fine. Recently, I have added some other metrics, such as curl invocations and tcp connection track. The problem is when I try to show the historical curl and tcp connections (yesterday, 2 days ago, and so on). The only manner to show collected metrics is picking up “Today” and “Last 24 hours”. Weirdly if I select “Last 24 hours” the metrics are showed, meanwhile “yesterday” no show any metric, even if two time ranges share time points, so I conjeture that all metrics are available but Grafana can not graph them for unkown causes.

“Yesterday”

“Las 24 hours”

Greetings!

Make sure your Graphite storage schema is correct and your storing the data.

Checkout this guide:
How to troubleshoot metric query issues

Hi, Torkel

Thanks for your suggestions. I am looking “Yesterday” pletoric of null values, meanwhile, “Last 5 minutes” has nulls and non-zeros values, this is ok, because sometimes URLs has latency and timeout comes.

“Yesterday”.

[{“target”: “TCP Connections”, “datapoints”: [[null, 1491368490], [null, 1491368580], [null, 1491368670], [null, 1491368760], [null, 1491368850], [null, 1491368940], [null, 1491369030], [null, 1491369120], [null, 1491369210], [null, 1491369300], [null, 1491369390], [null, 1491369480], [null, 1491369570], [null, 1491369660], [null, 1491369750], [null, 1491369840], [null, 1491369930], [null, 1491370020], [null, 1491370110], [null, 1491370200], [null, 1491370290], [null, 1491370380], [null, 1491370470], [null, 1491370560], [null, 1491370650], [null, 1491370740], [null, 1491370830], [null, 1491370920], [null, 1491371010], [null, 1491371100], [null, 1491371190], [null, 1491371280], [null, 1491371370], [null, 1491371460], [null, 1491371550], [null, 1491371640], [null, 1491371730], [null, 1491371820], [null, 1491371910], [null, 1491372000], [null, 1491372090], [null, 1491372180], [null, 1491372270], [null, 1491372360], [null, 1491372450], [null, 1491372540], [null, 1491372630], [null, 1491372720], [null, 1491372810], [null, 1491372900], [null, 1491372990], [null, 1491373080], [null, 1491373170], [null, 1491373260], [null, 1491373350], [null, 1491373440], [null, 1491373530], [null, 1491373620], [null, 1491373710], [null, 1491373800], [null, 1491373890], [null, 1491373980], [null, 1491374070], [null,
[trunkated]

“Last 5 minutes”

[{“target”: “TCP Connections”, “datapoints”: [[null, 1491500574], [4.0, 1491500575], [null, 1491500576], [null, 1491500577], [null, 1491500578], [null, 1491500579], [null, 1491500580], [null, 1491500581], [null, 1491500582], [null, 1491500583], [null, 1491500584], [4.0, 1491500585], [null, 1491500586], [null, 1491500587], [null, 1491500588], [null, 1491500589], [null, 1491500590], [null, 1491500591], [null, 1491500592], [null, 1491500593], [null, 1491500594], [4.0, 1491500595], [null, 1491500596], [null, 1491500597], [null, 1491500598], [null, 1491500599], [null, 1491500600], [null, 1491500601], [null, 1491500602], [null, 1491500603], [null, 1491500604], [4.0, 1491500605], [null, 1491500606], [null, 1491500607], [null, 1491500608], [null, 1491500609], [null, 1491500610], [null, 1491500611], [null, 1491500612], [null, 1491500613], [null, 1491500614], [4.0, 1491500615], [null, 1491500616], [null, 1491500617], [null, 1491500618], [null, 1491500619], [null, 1491500620], [null, 1491500621], [null, 1491500622], [null, 1491500623], [null, 1491500624], [4.0, 1491500625], [null, 1491500626], [null, 1491500627], [null, 1491500628], [null, 1491500629], [null, 1491500630], [null, 1491500631], [null, 1491500632], [null, 1491500633], [null, 1491500634], [4.0, 1491500635], [null, 1491500636], [null, 1491500637], [null, 1491500638], [null, 1491500639], [null, 1491500640], [null, 1491500641], [null, 1491500642], [null, 1491500643], [null, 1491500644], [4.0, 1491500645], [null, 1491500646], [null, 1491500647], [null, 1491500648], [null, 1491500649], [null, 1491500650], [null, 1491500651], [null, 1491500652], [null, 1491500653], [null, 1491500654], [6.0, 1491500655],
[Trunkated]

Into graphite whisper folder I have 38Mb of metrics.

/var/lib/graphite/whisper/xxx/hosts/tclmon010-1
[root@tclmon010-1 tclmon010-1]# du -sh * | grep tcp
38M tcpconns-3000-local
38M tcpconns-3001-local
[root@tclmon010-1 tclmon010-1]#

My storages-schema.conf:

Carbon’s internal metrics. This entry should match what is specified in

CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings

[carbon]
pattern = ^carbon.
retentions = 60:90d

#[default_1min_for_1day]
#pattern = .*
#retentions = 60s:1d

[default]
pattern = .*
retentions = 1s:24h, 10s:7d, 1m:30d, 10m:2y

My collectd.conf has configured 10s, in order to match with my storages-schema.conf

Interval 10
CheckThresholds true
WriteQueueLimitHigh 9000
WriteQueueLimitLow 9000

LoadPlugin logfile

<Plugin “logfile”>
LogLevel “debug”
File “/opt/collectd/log/collectd.log”
Timestamp true

LoadPlugin threshold

Include “/opt/collectd/etc/collect.d/*.conf”

“Yesterday”

“Last 24 hours”

I have queried to my graphite installation. The problem is there, because I have tested some date parameters and Graphite have had the same behavior than Grafana. So, I appreciate a lot your comments.

One day ago - OK

http://127.0.0.1:8889/render/?width=586&height=308&_salt=1355992522.674&target=alias(dc.hosts.amxre01-1.tcpconns-*.tcp_connections-ESTABLISHED,%20'TCP%20Connections')&from=-1days

Two days ago - NULL

http://127.0.0.1:8889/render/?width=586&height=308&_salt=1355992522.674&target=alias(dc.hosts.amxre01-1.tcpconns-*.tcp_connections-ESTABLISHED,%20'TCP%20Connections')&from=-2days

Your storage schema looks wrong, you have one rule that matches all metrics that only specifes 1 day retention, remember if you change storage schema your need to rebuild your wishper files

There was a problem with “#” interpretation in my entry. These lines are commented in my storage-schemes.config