Failed CarbonLink query

Grafana Graphite
1

Hi,

I cant find out what’s going on with my setup, but after creating a whole set of metrics, the graph rendering is quite slow on grafana, and the exception.log is spammed with the following :

Tue Mar 21 18:12:14 2017 :: Failed CarbonLink query 'storage.performance.cient.serial.sp.spb.storage.lun.lun129.responseTime'
    Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/graphite/render/datalib.py", line 385, in fetchData
        cachedResults = CarbonLink.query(dbFile.real_metric)
      File "/usr/lib/python2.6/site-packages/graphite/render/datalib.py", line 146, in query
        results = self.send_request(request)
      File "/usr/lib/python2.6/site-packages/graphite/render/datalib.py", line 207, in send_request
        conn = self.get_connection(host)
      File "/usr/lib/python2.6/site-packages/graphite/render/datalib.py", line 134, in get_connection
        connection.connect( (server, port) )
      File "<string>", line 1, in connect
    error: [Errno 13] Permission denied

For 5 minutes logging :

 # egrep -c "Failed CarbonLink query "  exception.log
211733

I’m using grafana 4.1.1 with graphite

 # graphite-manage --version
1.4.21

 # rpm -qa | grep carbon
python-carbon-0.9.15-1.el6.noarch

I have for now 22065 metrics, 90% of them updated every minute.

Any ideas where i should look ?

2

First question is, is this a new installation? If so what instructions did you follow to set it up? I think that [quote=“cesar, post:1, topic:160”]

graphite-manage --version

1.4.21
[/quote]

refers to the graphite-manage version and not graphite.

If it is not a new installation:

  • Have you investigated the permissions error?
  • Does graphite-web have access to the carbon agent?
  • Any problems with file permissions for the whisper files?
3

Hi,

It’s not a new installation, it worked fine before. It looks like it started after creating a bunch of new metrics.

I cant point any permission problem :
all wsp files are carbon:carbon 644
all directories are carbon:carbon 755

How can i check that graphite-web have access to the carbon agent ?

4

Maybe you are hitting the limit of having one Carbon daemon?

https://answers.launchpad.net/graphite/+question/252963

5

Well.
On the end, i reinstalled the OS CentOS 6.8 to 7.3.
And configured a new setup with one relay and 4 cache daemons.
I restored the wsp files, and everything is running fine now.

Didnt find the source of the carbon link failure :frowning: