-
Grafana version: latest (8.3.x)
-
Operating system: Docker (with docker-compose) or Ubuntu server 20.04 LTS
-
What are you trying to achieve?
I want to get my server, services and docker logs and metrics on a webpage. -
How are you trying to achieve it?
By using Loki to get logs, Grafana to display them and Traefik as a reverse-proxy -
What happened?
I get a weird error message:
-
What did you expect to happen?
I expect to get to the Grafana login page -
My configuration
docker-compose.yaml
version: "3.8"
volumes:
loki-data:
services:
traefik:
container_name: traefik
restart: unless-stopped
image: traefik
command
- "--log.level=INFO"
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/home/louison/turbonas/traefik/acme.json:/acme.json"
- "/home/louison/turbonas/traefik/.htpasswd:/.htpasswd"
- "/home/louison/turbonas/traefik/traefik.yaml:/traefik.yaml"
- "/home/louison/turbonas/traefik/traefik_dynamic.yaml:/traefik_dynamic.yaml"
- "/home/louison/turbonas/logs:/logs"
loki:
container_name: loki
restart: unless-stopped
image: grafana/loki:2.0.0
ports:
- 23100:3100
command:
- "-config.file=/loki-config.yml"
- "-log.level=info"
volumes:
- /home/louison/turbonas/loki/loki-config.yml:/loki-config.yml
- loki-data:/loki
labels:
- traefik.http.routers.loki.rule=Host(`stats.mydomain.fr/loki`)
- traefik.http.routers.loki.tls=true
- traefik.http.routers.loki.tls.certresolver=myresolver
- traefik.http.routers.loki.middlewares=simpleAuth@file
grafana:
container_name: grafana
restart: unless-stopped
image: grafana/grafana
expose:
- 3000
ports:
- 3000:3000
volumes:
- /home/louison/turbonas/grafana/grafana.ini:/etc/grafana/grafana.ini
- /home/louison/turbonas/grafana/dashboards:/var/lib/grafana/dashboards
- /home/louison/turbonas/grafana/datasources:/etc/grafana/datasources
- /home/louison/turbonas/grafana/plugins:/var/lib/grafana/plugins
- /home/louison/turbonas/grafana:/var/lib/grafana
- /home/louison/turbonas/logs:/logs
labels:
- traefik.http.routers.grafana.rule=Host(`stats.mydomain.fr`)
- traefik.http.routers.grafana.tls=true
- traefik.http.routers.grafana.tls.certresolver=myresolver
traefik.yaml
entryPoints:
web:
address: ':80'
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ':443'
api:
dashboard: true
certificatesResolvers:
myresolver:
acme:
httpChallenge:
entryPoint: web
email: myadresse@mail.com
storage: "acme.json"
providers:
docker:
watch: true
file:
filename: traefik_dynamic.yaml
log:
filePath: "/logs/traefik.log"
format: common
level: INFO
accessLog:
filePath: "/logs/access.log"
format: common
pilot:
token: <my-token>
experimental:
plugins:
fail2ban:
moduleName: "github.com/tommoulard/fail2ban"
version: "v0.6.2"
traefik_dynamic.yaml
http:
routers:
api:
rule: Host(`monitor.mydomaine.fr`)
entrypoints:
- websecure
middlewares:
- simpleAuth
service: api@internal
tls:
certResolver: myresolver
middlewares:
simpleAuth:
basicAuth:
usersfile: .htpasswd
# users:
my-fail2ban:
plugin:
fail2ban:
logLevel: "INFO"
bantime: "3h"
findtime: "3m"
maxretry: 3
enabled: true
loki-config.yml
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /loki/chunks
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
grafana.ini
[paths]
logs = /logs/grafana
[server]
domain = stats.mydomain.fr
[users]
allow_sign_up = false
auto_assign_org = true
auto_assign_org_role = Admin
[date_formats]
full_date = DD-MM-YYYY HH:mm:ss
I’ve configure /etc/docker/daemon.json as following to change the default log driver
{
"log-driver": "loki",
"log-opts": {
"loki-url": "http://localhost:3100/loki/api/v1/push"
}
}
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
I can’t get to login page, instead i got json error message in the browser without interacting at all.
In the log i get:
grafana | t=2022-01-07T21:44:58+0100 lvl=eror msg="invalid username or password" logger=context error="invalid username or password"
Complete log extract
louison@server:~/server$ docker-compose logs grafana
Attaching to grafana
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings
grafana | t=2022-01-07T21:44:27+0100 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="App mode production" logger=settings
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
grafana | t=2022-01-07T21:44:27+0100 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rwxrwxrwx expected=-rw-r-----
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Starting DB migrations" logger=migrator
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=708.347µs
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Initialising plugins" logger=plugin.manager
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-clock-panel
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="warming cache for startup" logger=ngalert
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager
grafana | t=2022-01-07T21:44:27+0100 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
grafana | t=2022-01-07T21:44:58+0100 lvl=eror msg="invalid username or password" logger=context error="invalid username or password"
grafana | t=2022-01-07T21:44:58+0100 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=401 remote_addr=192.168.1.254 time_ms=2 size=48 referer=
-
Did you follow any online instructions? If so, what is the URL?
I’ve partially followed this one : Collecting Docker logs with Grafana Loki | LinuxBlog.xyz -
By curiosity I’ve successfully tried to disable auth by enabling anonymous auth:
[auth.anonymous]
enabled = true
[users]
allow_sign_up = false
auto_assign_org = true
auto_assign_org_role = Admin
I’m now certain that my problem is a missinteraction between traefik and grafana. I’ve consider disabling grafana auth and use Traefik basicAuth. But this option lead me to another problem, so I’ve surrendered this path.
I’m open to every proposition to change this stack if:
- I can watch my logs and metrics via an UI
- I can get auth by any way
- I can use Traefik
Thanks a lot if you read upon this !