HI,
We have configured Graphite with Grafana. We are taking metrics from more than 10 running services. But sometimes it throws NO Data Alerts and stops collecting the metrics. And once we restart graphite service we will be able to get the data.
Kindly suggest the solution for it.
Investigate why graphite crashes? Check graphite logs
We just have :
03/04/2018 06:37:03 :: 127.0.0.1:36536 connection lost: Connection to the other side was lost in a non-clean fashion.
03/04/2018 06:37:03 :: 127.0.0.1:36537 connection lost: Connection to the other side was lost in a non-clean fashion.
everywhere in the logs.
The above log is from query.log
From which log file does that come from (is it from the Carbon query.log)? Is it due to the carbon-cache daemon restarting?
Yes its from query.log. So, whenever there is No data alert we see this error on the query.log file.
And we are using graphite = 0.9.12 version
Are you seeing errors in any of the other log files (like console.log for example)? Or seeing any unscheduled restarts for any of the Carbon daemons?
hi,
In console.log we can see this error,
22/04/2018 09:19:25 :: Error writing to /opt/graphite/storage/whisper/metrics/prod/io/dropwizard/jetty/MutableServletContextHandler/post-requests/p95.wsp
22/04/2018 09:19:25 :: Unhandled Error
Traceback (most recent call last):
File “/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py”, line 262, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File “/usr/local/lib/python2.7/dist-packages/twisted/python/context.py”, line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File “/usr/local/lib/python2.7/dist-packages/twisted/python/context.py”, line 81, in callWithContext
return func(*args,**kw)
File “/opt/graphite/lib/carbon/writer.py”, line 158, in writeForever
writeCachedDataPoints()
— —
File “/opt/graphite/lib/carbon/writer.py”, line 137, in writeCachedDataPoints
whisper.update_many(dbFilePath, datapoints)
File “/usr/local/lib/python2.7/dist-packages/whisper.py”, line 583, in update_many
points = [ (int(t),float(v)) for (t,v) in points]
exceptions.RuntimeError: deque mutated during iteration
Think you will have to open an issue either in the Whisper or Carbon repo.
Looks like there is something strange going on with the /opt/graphite/storage/whisper/metrics/prod/io/dropwizard/jetty/MutableServletContextHandler/post-requests/p95.wsp whisper file. Does it look ok? If you examine the file, do you get any info:
whisper-info.py /opt/graphite/storage/whisper/metrics/prod/io/dropwizard/jetty/MutableServletContextHandler/post-requests/p95.wsp
Are the retention rates the same as you have configured in your storage-schemas.conf?