[SOLVED] Grafana empty: "No Data". And no error logs

siment · August 22, 2023, 6:37am

Hi, guys. Sorry to bother you with this, but I just cannot figure this one out. I am testing Grafana Loki in a Docker Compose setup, but not seeing any data ingested into Grafana dashboard.

I am trying my hardest to find any hints as to where ingestion is blocked, but with no luck.

I will post complete Grafana, Loki and Promtail logs below.

Config

Docker Compose config

Here is my Docker Compose file:

# docker-compose.yaml
version: "3"

networks:
  loki:

services:
  loki:
    logging:
      driver: journald
      options:
        tag: loki
    image: grafana/loki:2.8.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /etc/loki:/etc/loki
      - /var/lib/loki:/loki
    networks:
      - loki

  promtail:
    logging:
      driver: journald
      options:
        tag: promtail
    image: grafana/promtail:2.8.0
    volumes:
      - /var/log:/var/log
      - /etc/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    logging:
      driver: journald
      options:
        tag: grafana
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy 
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    networks:
      - loki

Loki config

Here is my Loki config:

# /etc/loki/local-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  # You can omit this:
  log_level: warn

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Promtail config

Here is my Promtail config:

# config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
  # You can omit this:
  log_level: debug

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Grafana config

The Grafana config file is too big to post, but it is the default one copied from /usr/share/grafana/conf/defaults.ini inside the docker container.

Steps taken

In Grafana web UI

Grafana is successfully connecting to the data source “Loki” on http://loki:3100: Data source successfully connected.

In Promtail container

I can send logs from Promtail container to Loki:

cat /var/log/dummy.log | promtail --stdin --dry-run --client.url http://loki:3100/loki/api/v1/push

Output:

Clients configured:
----------------------
url: http://loki:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
enable_http2: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
drop_rate_limited_batches: false
stream_lag_labels: ""

2023-08-22T06:18:00.60657654+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:35 2023: this
2023-08-22T06:18:00.606577712+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:36 2023: is
2023-08-22T06:18:00.606579966+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:37 2023: a
2023-08-22T06:18:00.606582932+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:38 2023: dummy
2023-08-22T06:18:00.606585066+0000	{hostname="2bb288d2d3b3", job="stdin"}	Mon Aug 14 04:28:39 2023: log

I can also send without --dry-run:

cat /var/log/dummy.log | promtail --stdin --client.url http://loki:3100/loki/api/v1/push

No output, which indicates success.

I have also deleted /tmp/positions.yaml to ensure logs are re ingested:

rm /tmp/positions.yaml

I can see that /tmp/positions.yaml re populates shortly after being deleted:

# /tmp/positions.yaml:
positions:
  /var/log/alternatives.log: "23471"
  /var/log/apport.log: "4917"
  /var/log/bootstrap.log: "64549"
  /var/log/cloud-init-output.log: "87040"
  /var/log/cloud-init.log: "1880701"
  /var/log/dpkg.log: "909892"
  /var/log/dummy.log: "150"
  /var/log/fail2ban.log: "495573"
  /var/log/faillog: "0"
  /var/log/fontconfig.log: "484"
  /var/log/lastlog: "0"
  /var/log/openvpn-protected.log: "94773"
  /var/log/ubuntu-advantage-timer.log: "14025"
  /var/log/ubuntu-advantage.log: "168393"

Summary and questions

After all this, dashboards are still empty in Grafana

What kind of obvious (or not) rookie mistake am I doing here? I would greatly appreciate any tips.

Full logs

Docker Compose logs

[+] Running 3/0
 ✔ Container loki-grafana-1   Created                                                                                                                                                                                                                                                        0.0s 
 ✔ Container loki-loki-1      Created                                                                                                                                                                                                                                                        0.0s 
 ✔ Container loki-promtail-1  Created                                                                                                                                                                                                                                                        0.0s 
Attaching to loki-grafana-1, loki-loki-1, loki-promtail-1
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.78770515Z level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787837199Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787845886Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787848961Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787851917Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787854732Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787857157Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787859782Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787862507Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787865623Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787868599Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787872205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787875421Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787878527Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ENABLED=true"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787881232Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787884128Z level=info msg=Target target=[all]
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787888646Z level=info msg="Path Home" path=/usr/share/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787891522Z level=info msg="Path Data" path=/var/lib/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787896471Z level=info msg="Path Logs" path=/var/log/grafana
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787899246Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787902042Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
loki-grafana-1   | logger=settings t=2023-08-22T06:23:09.787904627Z level=info msg="App mode production"
loki-grafana-1   | logger=sqlstore t=2023-08-22T06:23:09.788035754Z level=info msg="Connecting to DB" dbtype=sqlite3
loki-grafana-1   | logger=migrator t=2023-08-22T06:23:09.814328341Z level=info msg="Starting DB migrations"
loki-grafana-1   | logger=migrator t=2023-08-22T06:23:09.826870944Z level=info msg="migrations completed" performed=0 skipped=485 duration=1.066916ms
loki-grafana-1   | logger=secrets t=2023-08-22T06:23:09.827522275Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
loki-grafana-1   | logger=local.finder t=2023-08-22T06:23:09.877037417Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
loki-grafana-1   | logger=query_data t=2023-08-22T06:23:09.879240009Z level=info msg="Query Service initialization"
loki-grafana-1   | logger=live.push_http t=2023-08-22T06:23:09.8816113Z level=info msg="Live Push Gateway initialization"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.876198031Z caller=promtail.go:125 msg="Reloading configuration file"
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.876678348Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=ddc18cca7f5fcf4eb48e967dd00065aa
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.876901029Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[system:{}]
loki-promtail-1  | level=debug ts=2023-08-22T06:23:09.877024342Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.879091148Z caller=server.go:323 http=0.0.0.0:9080 grpc=0.0.0.0:45349 msg="server listening on addresses"
loki-promtail-1  | level=info ts=2023-08-22T06:23:09.879261119Z caller=main.go:174 msg="Starting Promtail" version="(version=2.8.0, branch=HEAD, revision=90888a0cc)"
loki-promtail-1  | level=warn ts=2023-08-22T06:23:09.879317656Z caller=promtail.go:265 msg="enable watchConfig"
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.879382368Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.884346981Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
loki-loki-1      | level=warn ts=2023-08-22T06:23:09.884370595Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
loki-grafana-1   | logger=infra.usagestats.collector t=2023-08-22T06:23:11.33850898Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
loki-grafana-1   | logger=provisioning.alerting t=2023-08-22T06:23:11.358026002Z level=info msg="starting to provision alerting"
loki-grafana-1   | logger=provisioning.alerting t=2023-08-22T06:23:11.358035971Z level=info msg="finished to provision alerting"
loki-grafana-1   | logger=ngalert.state.manager t=2023-08-22T06:23:11.358196054Z level=info msg="Warming state cache for startup"
loki-grafana-1   | logger=grafanaStorageLogger t=2023-08-22T06:23:11.358270274Z level=info msg="storage starting"
loki-grafana-1   | logger=modules t=2023-08-22T06:23:11.358207886Z level=warn msg="No modules registered..."
loki-grafana-1   | logger=http.server t=2023-08-22T06:23:11.363756711Z level=info msg="HTTP Server Listen" address=0.0.0.0:3000 protocol=http subUrl= socket=
loki-grafana-1   | logger=ngalert.state.manager t=2023-08-22T06:23:11.385327796Z level=info msg="State cache has been initialized" states=0 duration=27.13053ms
loki-grafana-1   | logger=ticker t=2023-08-22T06:23:11.385392338Z level=info msg=starting first_tick=2023-08-22T06:23:20Z
loki-grafana-1   | logger=ngalert.multiorg.alertmanager t=2023-08-22T06:23:11.38539853Z level=info msg="starting MultiOrg Alertmanager"
loki-grafana-1   | logger=grafana.update.checker t=2023-08-22T06:23:11.393996688Z level=info msg="Update check succeeded" duration=35.574708ms
loki-grafana-1   | logger=plugins.update.checker t=2023-08-22T06:23:11.429935985Z level=info msg="Update check succeeded" duration=71.611468ms
loki-grafana-1   | logger=context userId=0 orgId=1 uname= t=2023-08-22T06:23:14.611460871Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=77.16.211.21 time_ms=0 duration=761.569µs size=0 referer= handler=/api/live/ws
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877248888Z caller=filetargetmanager.go:295 msg="new target" labels="{__address__=\"localhost\"}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877305495Z caller=filetargetmanager.go:355 msg="Adding target" key="/var/log/*log:{job=\"varlogs\"}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.87763624Z caller=filetarget.go:280 msg="watching new directory" directory=/var/log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877652281Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/alternatives.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877702445Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/apport.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877735237Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/bootstrap.log
loki-promtail-1  | ts=2023-08-22T06:23:14.8777683Z caller=log.go:168 level=info msg="Seeked /var/log/alternatives.log - &{Offset:26965 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877761507Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/alternatives.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877777657Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init-output.log
loki-promtail-1  | ts=2023-08-22T06:23:14.877783959Z caller=log.go:168 level=info msg="Seeked /var/log/apport.log - &{Offset:4917 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877816371Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877808826Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/bootstrap.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877820689Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init-output.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877796714Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/apport.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877874991Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dpkg.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877892314Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init.log
loki-promtail-1  | ts=2023-08-22T06:23:14.87791125Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init.log - &{Offset:1880701 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.87791103Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init-output.log - &{Offset:87040 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.877913204Z caller=log.go:168 level=info msg="Seeked /var/log/bootstrap.log - &{Offset:64549 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877934885Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dummy.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877936257Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dpkg.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.877978747Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fail2ban.log
loki-promtail-1  | ts=2023-08-22T06:23:14.87796441Z caller=log.go:168 level=info msg="Seeked /var/log/dpkg.log - &{Offset:913581 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.877996541Z caller=log.go:168 level=info msg="Seeked /var/log/dummy.log - &{Offset:150 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878009987Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/faillog
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878016349Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fail2ban.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878036657Z caller=log.go:168 level=info msg="Seeked /var/log/fail2ban.log - &{Offset:495573 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.877993626Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dummy.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878076061Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fontconfig.log
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878113151Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/lastlog
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878114414Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878129993Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/faillog
loki-promtail-1  | ts=2023-08-22T06:23:14.878150532Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:484 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878153898Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/openvpn-protected.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878161653Z caller=log.go:168 level=info msg="Seeked /var/log/faillog - &{Offset:0 Whence:0}"
loki-promtail-1  | ts=2023-08-22T06:23:14.878178295Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878190608Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/lastlog
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878199394Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage-timer.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.8782023Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/openvpn-protected.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878232757Z caller=log.go:168 level=info msg="Seeked /var/log/openvpn-protected.log - &{Offset:94773 Whence:0}"
loki-promtail-1  | level=debug ts=2023-08-22T06:23:14.878270499Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage.log
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878283173Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage-timer.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878299524Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage-timer.log - &{Offset:14025 Whence:0}"
loki-promtail-1  | level=info ts=2023-08-22T06:23:14.878326495Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage.log
loki-promtail-1  | ts=2023-08-22T06:23:14.878338798Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage.log - &{Offset:168393 Whence:0}"

Loki, Grafana and Promtail logs

Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.78770515Z level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787837199Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787845886Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787848961Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787851917Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787854732Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787857157Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787859782Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787862507Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787865623Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787868599Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787872205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787875421Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787878527Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ENABLED=true"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787881232Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787884128Z level=info msg=Target target=[all]
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787888646Z level=info msg="Path Home" path=/usr/share/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787891522Z level=info msg="Path Data" path=/var/lib/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787896471Z level=info msg="Path Logs" path=/var/log/grafana
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787899246Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787902042Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
Aug 22 06:23:09 home-server grafana[1196549]: logger=settings t=2023-08-22T06:23:09.787904627Z level=info msg="App mode production"
Aug 22 06:23:09 home-server grafana[1196549]: logger=sqlstore t=2023-08-22T06:23:09.788035754Z level=info msg="Connecting to DB" dbtype=sqlite3
Aug 22 06:23:09 home-server grafana[1196549]: logger=migrator t=2023-08-22T06:23:09.814328341Z level=info msg="Starting DB migrations"
Aug 22 06:23:09 home-server grafana[1196549]: logger=migrator t=2023-08-22T06:23:09.826870944Z level=info msg="migrations completed" performed=0 skipped=485 duration=1.066916ms
Aug 22 06:23:09 home-server grafana[1196549]: logger=secrets t=2023-08-22T06:23:09.827522275Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
Aug 22 06:23:09 home-server grafana[1196549]: logger=local.finder t=2023-08-22T06:23:09.877037417Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
Aug 22 06:23:09 home-server grafana[1196549]: logger=query_data t=2023-08-22T06:23:09.879240009Z level=info msg="Query Service initialization"
Aug 22 06:23:09 home-server grafana[1196549]: logger=live.push_http t=2023-08-22T06:23:09.8816113Z level=info msg="Live Push Gateway initialization"
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.876198031Z caller=promtail.go:125 msg="Reloading configuration file"
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.876678348Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=ddc18cca7f5fcf4eb48e967dd00065aa
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.876901029Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[system:{}]
Aug 22 06:23:09 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:09.877024342Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.879091148Z caller=server.go:323 http=0.0.0.0:9080 grpc=0.0.0.0:45349 msg="server listening on addresses"
Aug 22 06:23:09 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:09.879261119Z caller=main.go:174 msg="Starting Promtail" version="(version=2.8.0, branch=HEAD, revision=90888a0cc)"
Aug 22 06:23:09 home-server promtail[1196549]: level=warn ts=2023-08-22T06:23:09.879317656Z caller=promtail.go:265 msg="enable watchConfig"
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.879382368Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.884346981Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
Aug 22 06:23:09 home-server loki[1196549]: level=warn ts=2023-08-22T06:23:09.884370595Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
Aug 22 06:23:11 home-server grafana[1196549]: logger=infra.usagestats.collector t=2023-08-22T06:23:11.33850898Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
Aug 22 06:23:11 home-server grafana[1196549]: logger=provisioning.alerting t=2023-08-22T06:23:11.358026002Z level=info msg="starting to provision alerting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=provisioning.alerting t=2023-08-22T06:23:11.358035971Z level=info msg="finished to provision alerting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.state.manager t=2023-08-22T06:23:11.358196054Z level=info msg="Warming state cache for startup"
Aug 22 06:23:11 home-server grafana[1196549]: logger=grafanaStorageLogger t=2023-08-22T06:23:11.358270274Z level=info msg="storage starting"
Aug 22 06:23:11 home-server grafana[1196549]: logger=modules t=2023-08-22T06:23:11.358207886Z level=warn msg="No modules registered..."
Aug 22 06:23:11 home-server grafana[1196549]: logger=http.server t=2023-08-22T06:23:11.363756711Z level=info msg="HTTP Server Listen" address=0.0.0.0:3000 protocol=http subUrl= socket=
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.state.manager t=2023-08-22T06:23:11.385327796Z level=info msg="State cache has been initialized" states=0 duration=27.13053ms
Aug 22 06:23:11 home-server grafana[1196549]: logger=ticker t=2023-08-22T06:23:11.385392338Z level=info msg=starting first_tick=2023-08-22T06:23:20Z
Aug 22 06:23:11 home-server grafana[1196549]: logger=ngalert.multiorg.alertmanager t=2023-08-22T06:23:11.38539853Z level=info msg="starting MultiOrg Alertmanager"
Aug 22 06:23:11 home-server grafana[1196549]: logger=grafana.update.checker t=2023-08-22T06:23:11.393996688Z level=info msg="Update check succeeded" duration=35.574708ms
Aug 22 06:23:11 home-server grafana[1196549]: logger=plugins.update.checker t=2023-08-22T06:23:11.429935985Z level=info msg="Update check succeeded" duration=71.611468ms
Aug 22 06:23:14 home-server grafana[1196549]: logger=context userId=0 orgId=1 uname= t=2023-08-22T06:23:14.611460871Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=77.16.211.21 time_ms=0 duration=761.569µs size=0 referer= handler=/api/live/ws
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877248888Z caller=filetargetmanager.go:295 msg="new target" labels="{__address__=\"localhost\"}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877305495Z caller=filetargetmanager.go:355 msg="Adding target" key="/var/log/*log:{job=\"varlogs\"}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.87763624Z caller=filetarget.go:280 msg="watching new directory" directory=/var/log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877652281Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/alternatives.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877702445Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/apport.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877735237Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/bootstrap.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.8777683Z caller=log.go:168 level=info msg="Seeked /var/log/alternatives.log - &{Offset:26965 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877761507Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/alternatives.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877777657Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init-output.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877783959Z caller=log.go:168 level=info msg="Seeked /var/log/apport.log - &{Offset:4917 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877816371Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/cloud-init.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877808826Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/bootstrap.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877820689Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init-output.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877796714Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/apport.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877874991Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dpkg.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877892314Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/cloud-init.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87791125Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init.log - &{Offset:1880701 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87791103Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init-output.log - &{Offset:87040 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877913204Z caller=log.go:168 level=info msg="Seeked /var/log/bootstrap.log - &{Offset:64549 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877934885Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/dummy.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877936257Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dpkg.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.877978747Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fail2ban.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.87796441Z caller=log.go:168 level=info msg="Seeked /var/log/dpkg.log - &{Offset:913581 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.877996541Z caller=log.go:168 level=info msg="Seeked /var/log/dummy.log - &{Offset:150 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878009987Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/faillog
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878016349Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fail2ban.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878036657Z caller=log.go:168 level=info msg="Seeked /var/log/fail2ban.log - &{Offset:495573 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.877993626Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/dummy.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878076061Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/fontconfig.log
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878113151Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/lastlog
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878114414Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878129993Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/faillog
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878150532Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:484 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878153898Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/openvpn-protected.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878161653Z caller=log.go:168 level=info msg="Seeked /var/log/faillog - &{Offset:0 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878178295Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878190608Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/lastlog
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878199394Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage-timer.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.8782023Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/openvpn-protected.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878232757Z caller=log.go:168 level=info msg="Seeked /var/log/openvpn-protected.log - &{Offset:94773 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=debug ts=2023-08-22T06:23:14.878270499Z caller=filetarget.go:330 msg="tailing new file" filename=/var/log/ubuntu-advantage.log
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878283173Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage-timer.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878299524Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage-timer.log - &{Offset:14025 Whence:0}"
Aug 22 06:23:14 home-server promtail[1196549]: level=info ts=2023-08-22T06:23:14.878326495Z caller=tailer.go:143 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage.log
Aug 22 06:23:14 home-server promtail[1196549]: ts=2023-08-22T06:23:14.878338798Z caller=log.go:168 level=info msg="Seeked /var/log/ubuntu-advantage.log - &{Offset:168393 Whence:0}"

siment · August 22, 2023, 11:47am

SOLVED: I was an idiot. I had an active filter in Grafana (wrong host) that prevented me from seeing logs.