[SOLVED] Grafana empty: "No Data". And no error logs

siment · August 22, 2023, 6:37am

Hi, guys. Sorry to bother you with this, but I just cannot figure this one out. I am testing Grafana Loki in a Docker Compose setup, but not seeing any data ingested into Grafana dashboard.

I am trying my hardest to find any hints as to where ingestion is blocked, but with no luck.

I will post complete Grafana, Loki and Promtail logs below.

Config

Docker Compose config

Here is my Docker Compose file:

# docker-compose.yaml
version: "3"

networks:
  loki:

services:
  loki:
    logging:
      driver: journald
      options:
        tag: loki
    image: grafana/loki:2.8.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /etc/loki:/etc/loki
      - /var/lib/loki:/loki
    networks:
      - loki

  promtail:
    logging:
      driver: journald
      options:
        tag: promtail
    image: grafana/promtail:2.8.0
    volumes:
      - /var/log:/var/log
      - /etc/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    logging:
      driver: journald
      options:
        tag: grafana
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy 
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    networks:
      - loki

Loki config

Here is my Loki config:

# /etc/loki/local-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  # You can omit this:
  log_level: warn

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Promtail config

Here is my Promtail config:

# config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
  # You can omit this:
  log_level: debug

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Grafana config

The Grafana config file is too big to post, but it is the default one copied from /usr/share/grafana/conf/defaults.ini inside the docker container.

Steps taken

In Grafana web UI

Grafana is successfully connecting to the data source “Loki” on http://loki:3100: Data source successfully connected.

In Promtail container

I can send logs from Promtail container to Loki:

cat /var/log/dummy.log | promtail --stdin --dry-run --client.url http://loki:3100/loki/api/v1/push

Output:

Clients configured:
----------------------
url: http://loki:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
enable_http2: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
drop_rate_limited_batches: false
stream_lag_labels: ""

2023-08-22T06:18:00.60657654+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:35 2023: this
2023-08-22T06:18:00.606577712+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:36 2023: is
2023-08-22T06:18:00.606579966+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:37 2023: a
2023-08-22T06:18:00.606582932+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:38 2023: dummy
2023-08-22T06:18:00.606585066+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:39 2023: log

I can also send without --dry-run:

cat /var/log/dummy.log | promtail --stdin --client.url http://loki:3100/loki/api/v1/push

No output, which indicates success.

I have also deleted /tmp/positions.yaml to ensure logs are re ingested:

rm /tmp/positions.yaml

I can see that /tmp/positions.yaml re populates shortly after being deleted:

# /tmp/positions.yaml:
positions:
  /var/log/alternatives.log: "23471"
  /var/log/apport.log: "4917"
  /var/log/bootstrap.log: "64549"
  /var/log/cloud-init-output.log: "87040"
  /var/log/cloud-init.log: "1880701"
  /var/log/dpkg.log: "909892"
  /var/log/dummy.log: "150"
  /var/log/fail2ban.log: "495573"
  /var/log/faillog: "0"
  /var/log/fontconfig.log: "484"
  /var/log/lastlog: "0"
  /var/log/openvpn-protected.log: "94773"
  /var/log/ubuntu-advantage-timer.log: "14025"
  /var/log/ubuntu-advantage.log: "168393"

Summary and questions

After all this, dashboards are still empty in Grafana

What kind of obvious (or not) rookie mistake am I doing here? I would greatly appreciate any tips.

Full logs

Docker Compose logs

[+] Running 3/0
 ✔ Container loki-grafana-1   Created                                                                                                                                                                                                                                                        0.0s 
 ✔ Container loki-loki-1      Created                                                                                                                                                                                                                                                        0.0s 
 ✔ Container loki-promtail-1  Created                                                                                                                                                                                                                                                        0.0s 
Attaching to loki-grafana-1, loki-loki-1, loki-promtail-1
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.78770515Z level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787837199Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787845886Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787848961Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787851917Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787854732Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787857157Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787859782Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787862507Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787865623Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787868599Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787872205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787875421Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787878527Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ENABLED=true"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787881232Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787884128Z level=info msg=Target target=[all]
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787888646Z level=info msg="Path Home" path=/usr/share/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787891522Z level=info msg="Path Data" path=/var/lib/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787896471Z level=info msg="Path Logs" path=/var/log/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787899246Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787902042Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787904627Z level=info msg="App mode production"
loki-grafana-1   | logger=sqlstore t=2023-08-22T06:23:09.788035754Z level=info msg="Connecting to DB" dbtype=sqlite3
loki-grafana-1   | logger=migrator t=2023-08-22T06:23:09.814328341Z level=info msg="Starting DB migrations"
loki-grafana-1   | logger=migrator t=2023-08-22T06:23:09.826870944Z level=info msg="migrations completed" performed=0 skipped=485 duration=1.066916ms
loki-grafana-1   | logger=secrets t=2023-08-22T06:23:09.827522275Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
loki-grafana-1   | logger=local.finder t=2023-08-22T06:23:09.877037417Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
loki-grafana-1   | logger=query_data t=2023-08-22T06:23:09.879240009Z level=info msg="Query Service initialization"
loki-grafana-1   | logger=live.push_http t=2023-08-22T06:23:09.8816113Z level=info msg="Live Push Gateway initialization"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.876198031Z caller=promtail.go:125 msg="Reloading configuration file"
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.876678348Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=ddc18cca7f5fcf4eb48e967dd00065aa
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.876901029Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[system:{}]
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.877024342Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.879091148Z caller=server.go:323 http=0.0.0.0:9080 grpc=0.0.0.0:45349 msg="server listening on addresses"
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.879261119Z caller=main.go:174 msg="Starting Promtail" version="(version=2.8.0, branch=HEAD, revision=90888a0cc)"
loki-promtail-1  | level=warn ts=2023-08-22T06:23:09.879317656Z caller=promtail.go:265 msg="enable watchConfig"
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.879382368Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.884346981Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.884370595Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
loki-grafana-1   | logger=infra.usagestats.collector t=2023-08-22T06:23:11.33850898Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
loki-grafana-1   | logger=provisioning.alerting t=2023-08-22T06:23:11.358026002Z level=info msg="starting to provision alerting"
loki-grafana-1   | logger=provisioning.alerting t=2023-08-22T06:23:11.358035971Z level=info msg="finished to provision alerting"
loki-grafana-1   | logger=ngalert.state.manager t=2023-08-22T06:23:11.358196054Z level=info msg="Warming state cache for startup"
loki-grafana-1   | logger=grafanaStorageLogger t=2023-08-22T06:23:11.358270274Z level=info msg="storage starting"
loki-grafana-1   | logger=modules t=2023-08-22T06:23:11.358207886Z level=warn msg="No modules registered..."
loki-grafana-1   | logger=http.server t=2023-08-22T06:23:11.363756711Z level=info msg="HTTP Server Listen" address=0.0.0.0:3000 protocol=http subUrl= socket=
loki-grafana-1   | logger=ngalert.state.manager t=2023-08-22T06:23:11.385327796Z level=info msg="State cache has been initialized" states=0 duration=27.13053ms
loki-grafana-1   | logger=ticker t=2023-08-22T06:23:11.385392338Z level=info msg=starting first_tick=2023-08-22T06:23:20Z
loki-grafana-1   | logger=ngalert.multiorg.alertmanager t=2023-08-22T06:23:11.38539853Z level=info msg="starting MultiOrg Alertmanager"
loki-grafana-1   | logger=grafana.update.checker t=2023-08-22T06:23:11.393996688Z level=info msg="Update check succeeded" duration=35.574708ms
loki-grafana-1   | logger=plugins.update.checker t=2023-08-22T06:23:11.429935985Z level=info msg="Update check succeeded" duration=71.611468ms
loki-grafana-1   | logger=context userId=0 orgId=1 uname= t=2023-08-22T06:23:14.611460871Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=77.16.211.21 time_ms=0 duration=761.569µs size=0 referer= handler=/api/live/ws
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877248888Z caller=filetargetmanager.go:295 msg="new target" labels="{__address__=\"localhost\"}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877305495Z caller=filetargetmanager.go:355 msg="Adding target" key="/var/log/*log:{job=\"varlogs\"}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.87763624Z caller=filetarget.go:280 msg="watching new directory" directory=/var/log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877652281Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/alternatives.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877702445Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/apport.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877735237Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/bootstrap.log
loki-promtail-1  | ts=2023-08-22T06:23:14.8777683Z caller=log.go:168 level=info msg="Seeked /var/log/alternatives.log - &{Offset:26965 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877761507Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/alternatives.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877777657Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init-output.log
loki-promtail-1  | ts=2023-08-22T06:23:14.877783959Z caller=log.go:168 level=info msg="Seeked /var/log/apport.log - &{Offset:4917 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877816371Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877808826Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/bootstrap.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877820689Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init-output.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877796714Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/apport.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877874991Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dpkg.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877892314Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init.log
loki-promtail-1  | ts=2023-08-22T06:23:14.87791125Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init.log - &{Offset:1880701 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.87791103Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init-output.log - &{Offset:87040 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.877913204Z caller=log.go:168 level=info msg="Seeked /var/log/bootstrap.log - &{Offset:64549 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877934885Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dummy.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877936257Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dpkg.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877978747Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fail2ban.log
loki-promtail-1  | ts=2023-08-22T06:23:14.87796441Z caller=log.go:168 level=info msg="Seeked /var/log/dpkg.log - &{Offset:913581 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.877996541Z caller=log.go:168 level=info msg="Seeked /var/log/dummy.log - &{Offset:150 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878009987Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/faillog
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878016349Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fail2ban.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878036657Z caller=log.go:168 level=info msg="Seeked /var/log/fail2ban.log - &{Offset:495573 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877993626Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dummy.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878076061Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fontconfig.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878113151Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/lastlog
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878114414Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878129993Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/faillog
loki-promtail-1  | ts=2023-08-22T06:23:14.878150532Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:484 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878153898Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/openvpn-protected.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878161653Z caller=log.go:168 level=info msg="Seeked /var/log/faillog - &{Offset:0 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.878178295Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878190608Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/lastlog
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878199394Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage-timer.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.8782023Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/openvpn-protected.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878232757Z caller=log.go:168 level=info msg="Seeked /var/log/openvpn-protected.log - &{Offset:94773 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878270499Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878283173Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage-timer.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878299524Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage-timer.log - &{Offset:14025 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878326495Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878338798Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage.log - &{Offset:168393 Whence:0}"

Loki, Grafana and Promtail logs

Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.78770515Z level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787837199Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787845886Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787848961Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787851917Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787854732Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787857157Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787859782Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787862507Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787865623Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787868599Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787872205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787875421Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787878527Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ENABLED=true"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787881232Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787884128Z level=info msg=Target target=[all]
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787888646Z level=info msg="Path Home" path=/usr/share/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787891522Z level=info msg="Path Data" path=/var/lib/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787896471Z level=info msg="Path Logs" path=/var/log/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787899246Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787902042Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787904627Z level=info msg="App mode production"
Aug 22 06:23:09 home-server grafana[1196549]: logger=sqlstore t=2023-08-22T06:23:09.788035754Z level=info msg="Connecting to DB" dbtype=sqlite3
Aug 22 06:23:09 home-server grafana[1196549]: logger=migrator t=2023-08-22T06:23:09.814328341Z level=info msg="Starting DB migrations"
Aug 22 06:23:09 home-server grafana[1196549]: logger=migrator t=2023-08-22T06:23:09.826870944Z level=info msg="migrations completed" performed=0 skipped=485 duration=1.066916ms
Aug 22 06:23:09 home-server grafana[1196549]: logger=secrets t=2023-08-22T06:23:09.827522275Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
Aug 22 06:23:09 home-server grafana[1196549]: logger=local.finder t=2023-08-22T06:23:09.877037417Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
Aug 22 06:23:09 home-server grafana[1196549]: logger=query_data t=2023-08-22T06:23:09.879240009Z level=info msg="Query Service initialization"
Aug 22 06:23:09 home-server grafana[1196549]: logger=live.push_http t=2023-08-22T06:23:09.8816113Z level=info msg="Live Push Gateway initialization"
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.876198031Z caller=promtail.go:125 msg="Reloading configuration file"
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.876678348Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=ddc18cca7f5fcf4eb48e967dd00065aa
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.876901029Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[system:{}]
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.877024342Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.879091148Z caller=server.go:323 http=0.0.0.0:9080 grpc=0.0.0.0:45349 msg="server listening on addresses"
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.879261119Z caller=main.go:174 msg="Starting Promtail" version="(version=2.8.0, branch=HEAD, revision=90888a0cc)"
Aug 22 06:23:09 home-server promtail[1196549]: level=warn ts=2023-08-22T06:23:09.879317656Z caller=promtail.go:265 msg="enable watchConfig"
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.879382368Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.884346981Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.884370595Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
Aug 22 06:23:11 home-server grafana[1196549]: logger=infra.usagestats.collector t=2023-08-22T06:23:11.33850898Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
Aug 22 06:23:11 home-server grafana[1196549]: logger=provisioning.alerting t=2023-08-22T06:23:11.358026002Z level=info msg="starting to provision alerting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=provisioning.alerting t=2023-08-22T06:23:11.358035971Z level=info msg="finished to provision alerting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.state.manager t=2023-08-22T06:23:11.358196054Z level=info msg="Warming state cache for startup"
Aug 22 06:23:11 home-server grafana[1196549]: logger=grafanaStorageLogger t=2023-08-22T06:23:11.358270274Z level=info msg="storage starting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=modules t=2023-08-22T06:23:11.358207886Z level=warn msg="No modules registered..."
Aug 22 06:23:11 home-server grafana[1196549]: logger=http.server t=2023-08-22T06:23:11.363756711Z level=info msg="HTTP Server Listen" address=0.0.0.0:3000 protocol=http subUrl= socket=
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.state.manager t=2023-08-22T06:23:11.385327796Z level=info msg="State cache has been initialized" states=0 duration=27.13053ms
Aug 22 06:23:11 home-server grafana[1196549]: logger=ticker t=2023-08-22T06:23:11.385392338Z level=info msg=starting first_tick=2023-08-22T06:23:20Z
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.multiorg.alertmanager t=2023-08-22T06:23:11.38539853Z level=info msg="starting MultiOrg Alertmanager"
Aug 22 06:23:11 home-server grafana[1196549]: logger=grafana.update.checker t=2023-08-22T06:23:11.393996688Z level=info msg="Update check succeeded" duration=35.574708ms
Aug 22 06:23:11 home-server grafana[1196549]: logger=plugins.update.checker t=2023-08-22T06:23:11.429935985Z level=info msg="Update check succeeded" duration=71.611468ms
Aug 22 06:23:14 home-server grafana[1196549]: logger=context userId=0 orgId=1 uname= t=2023-08-22T06:23:14.611460871Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=77.16.211.21 time_ms=0 duration=761.569µs size=0 referer= handler=/api/live/ws
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877248888Z caller=filetargetmanager.go:295 msg="new target" labels="{__address__=\"localhost\"}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877305495Z caller=filetargetmanager.go:355 msg="Adding target" key="/var/log/*log:{job=\"varlogs\"}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.87763624Z caller=filetarget.go:280 msg="watching new directory" directory=/var/log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877652281Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/alternatives.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877702445Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/apport.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877735237Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/bootstrap.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.8777683Z caller=log.go:168 level=info msg="Seeked /var/log/alternatives.log - &{Offset:26965 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877761507Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/alternatives.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877777657Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init-output.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877783959Z caller=log.go:168 level=info msg="Seeked /var/log/apport.log - &{Offset:4917 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877816371Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877808826Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/bootstrap.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877820689Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init-output.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877796714Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/apport.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877874991Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dpkg.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877892314Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87791125Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init.log - &{Offset:1880701 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87791103Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init-output.log - &{Offset:87040 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877913204Z caller=log.go:168 level=info msg="Seeked /var/log/bootstrap.log - &{Offset:64549 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877934885Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dummy.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877936257Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dpkg.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877978747Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fail2ban.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87796441Z caller=log.go:168 level=info msg="Seeked /var/log/dpkg.log - &{Offset:913581 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877996541Z caller=log.go:168 level=info msg="Seeked /var/log/dummy.log - &{Offset:150 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878009987Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/faillog
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878016349Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fail2ban.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878036657Z caller=log.go:168 level=info msg="Seeked /var/log/fail2ban.log - &{Offset:495573 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877993626Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dummy.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878076061Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fontconfig.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878113151Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/lastlog
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878114414Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878129993Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/faillog
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878150532Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:484 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878153898Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/openvpn-protected.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878161653Z caller=log.go:168 level=info msg="Seeked /var/log/faillog - &{Offset:0 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878178295Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878190608Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/lastlog
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878199394Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage-timer.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.8782023Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/openvpn-protected.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878232757Z caller=log.go:168 level=info msg="Seeked /var/log/openvpn-protected.log - &{Offset:94773 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878270499Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878283173Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage-timer.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878299524Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage-timer.log - &{Offset:14025 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878326495Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878338798Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage.log - &{Offset:168393 Whence:0}"

siment · August 22, 2023, 11:47am

SOLVED: I was an idiot. I had an active filter in Grafana (wrong host) that prevented me from seeing logs.

system · August 21, 2024, 11:48am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logs showing in Docker Desktop but not in Grafana Grafana Loki loki	1	165	May 15, 2024
Loki logging docker driver not working with docker compose? Grafana Loki docker , loki	1	1445	January 28, 2023
Dockerized Grafana not able to use Dockerized Loki+Promtail as data source Grafana Loki docker , loki , grafana	5	660	April 2, 2024
Promtail syslog receiver , no data in Loki Grafana Loki loki , grafana-ui , promtail	1	1797	December 7, 2023
"Data source connected and labels found." But no logs is found from my Docker containers in Grafana/Loki Grafana Loki docker , loki	1	1831	April 18, 2022