I just installed grafana, and am running loki with command: “loki-windows-amd64.exe --config.file=loki-config.yaml”
loki-config.yaml: #LOKIauth_enabled: falseserver: http_listen_port: 3100 #log_level: - Pastebin.com
When i try to add loki in grafana, i get the following error when pressing “save & test”: “Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details”
Error messages in the loki window: ts=2022-08-22T18:04:12.0587155Z caller=spanlogger.go:80 user=fake method=query.L - Pastebin.com
Screenshot of grafana error message and input: https://usercontent.irccloud-cdn.com/file/99fYsytK/screenshot-grafana-error-message.png
Also is promtail needs to be running and feeding Loki.
here is my promtail config, I have a few txt files that promtail reads to push to Loki
- url: http://localhost:3100/loki/api/v1/push
- job_name: system
Then you get this
Few leads in following url to sort out errors.
Still have the same problem.
Error messages when trying to “save & test” loki as a datasource:
If i go to http://localhost:3100/, i get “404 page not found”, which means that the loki server is running.
When i try to press “save & test”, i get the following log entries:
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-07T15:30:46.2934775+01:00 level=error msg="Failed to call resource" error="Get \"http://localhost:3100/loki/api/v1/labels?start=1670422816283000000&end=1670423416283000000\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" traceID=00000000000000000000000000000000
logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-07T15:30:46.296726+01:00 level=error msg="Request Completed" method=GET path=/api/datasources/9/resources/labels status=500 remote_addr=127.0.0.1 time_ms=30012 duration=30.0127575s size=83 referer=http://localhost:3000/datasources/edit/YJF87QK4k traceID=00000000000000000000000000000000
ts=2022-12-07T14:55:00.0223554Z caller=spanlogger.go:80 user=fake method=query.Label level=info org_id=fake latency=fast query_type=labels length=10m0s duration=566.4µs status=200 label= throughput=0B total_bytes=0B total_entries=2
level=warn ts=2022-12-07T14:55:05.7937448Z caller=pool.go:184 msg="removing frontend failing healthcheck" addr=172.29.176.1:9096 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=error ts=2022-12-07T14:55:05.7937448Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = grpc: the client connection is closing" frontend=172.29.176.1:9096
level=error ts=2022-12-07T14:55:30.0319769Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-07T14:55:30.0319769Z caller=stats.go:57 component=frontend msg="failed to record query metrics" err="expected one of the *LokiRequest, *LokiInstantRequest, *LokiSeriesRequest, *LokiLabelNamesRequest, got "
My conclusion so far: Grafana and loki is running. Grafana can connect to loki.
How are you running loki? Straight from the exe or in docker?
Straight from exe. I don’t use docker at all.
C:\Program Files\GrafanaLabs!Loki\loki-windows-amd64.exe --config.file=loki-config.yaml
What does it show when you run it in commamd line
Please post screen shot?
Content of the tmp folder:
screenshot of metrics:
Btw, the content of the following file is actually from the logfile on a remote server via promtail, so its actually connected …wow… C:\Program Files\GrafanaLabs!Loki\tmp\loki\wal\00000000
Also, right before i created this post, i deleted the whole tmp folder, and it populated it.
and following section of you loki config file, what does yours show?
My loki conf:
# enabled: true
# distributed: false
# max_size_mb: 100
- from: 2020-10-24
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# Refer to the buildReport method to see what goes into a report.
# If you would like to disable reporting, uncomment the following lines:
# reporting_enabled: false
Btw, dont have any alertmanager setup yet.
so in grafana you entered https here we see it works on http
You have good eyes! i tried both HTTP and HTTPS, but i found out that i got the 404 on HTTP, so i understood that i had to use HTTP.
Sorry for the missleading screenshot above.
I got the same error in grafana using HTTP.
no lets move on to possibly networking issues
is grafana on same server as loki?
Grafana is running on my pc, same as loki. Promtail is installed on an remote server.
Regarding my conclusion: when i press “save & test”, something happens in the loki logs. The 5 lines of loki log i pasted above comes from grafana since i pressed that button. The five lines comes each time i press the button. → just installed loki locally on my windows computer(same as grafana runs on), and i cannot "save and test". - #3 by patchie
I also know that loki connects to promtail, since it fetches data from the remote server. → just installed loki locally on my windows computer(same as grafana runs on), and i cannot "save and test". - #9 by patchie
I can also add that on the same remote server running promtail, i have also “windows_exporter” installed on that server. windows_exporter sends data to prometheus(local pc, same as loki and grafana).
Just because you see folders in loki does not necessarily mean things are working. You have to do actual tests.
I think the above is where your issue is. Notice what doco says about Promtail
Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.
It is a push system. Cause I do not see anything in your Loki config, I have know to be blind often, that points to Promtail.
Let’s see your promtail config please.
I think this is how it is supposed to work. I might be wrong
in the following folder, i found files with actual log data from the server, so thats why i know it fetches data: C:\Program Files\GrafanaLabs!Loki\tmp\loki\wal\00000000
yep, the graphics you posted seems fair. I have grafana and loki on same pc, and promtail on another server.
- url: http://*********:3100/loki/api/v1/push
# The URL where Loki is listening, denoted in Loki as http_listen_address and
# http_listen_port. If Loki is running in microservices mode, this is the HTTP
# URL for the Distributor. Path to the push API needs to be included.
# Example: http://example.com:3100/loki/api/v1/push
- job_name: *******-Logs
# Configures the discovery to look on the current machine.
# This is required by the prometheus service discovery code but doesn't
# really apply to Promtail which can ONLY look at files on the local machine
# As such it should only have the value of localhost, OR it can be excluded
# entirely and a default value of localhost will be applied by Promtail.
# The path to load logs from. Can use glob patterns (e.g., /var/log/*.log).
# Additional labels to assign to the logs
# [ <labelname>: <labelvalue> ... ]