No labels being outputted in Grafana Data Source

  • What Grafana version and what operating system are you using?
    Grafana v11.1.3 , Debian 12 bookworm on VMware ESXi

  • What are you trying to achieve?
    Want to ingest the file and make it appear in the WebApp

  • What happened?
    I’m trying to host a local server for internal use which the server has no internet connection, all the binaries I downloaded from a local machine and transferred there to be installed:

  1. grafana-11.1.3.linux-amd64.tar.gz
  2. loki_3.0.1_amd64.deb
  3. promtail_3.0.1_amd64.deb

Then with sudo dpkg -i files.deb and tar -zxvf file.tar.gz

Loki and Promtail is installed to /usr/bin/ and grafana just extracts into a folder and in grafana/bin contains the binary, which I used # > ./grafana server

I could acces the website on localhost:3000 and navigate around. Using Loki as a data source and save & test was successful.

Configuration wise also able to fetch the log file inside the path but nothing pops up. I ran this on my VM and everything was okay. The only difference is the internet access factor.

  • What did you expect to happen?
    I expect the varlogs label to be appeared in the WebApp

  • Can you copy/paste the configuration(s) that you are having problems with?

/etc/loki/config.yml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

/etc/promtail/config.yml:


server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
- url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/grafana/ingest/*log

localhost:3000/api/health returned okay
localhost:3100/ready returned ready

localhost:9080:

  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
  http_listen_conn_limit: 0
  grpc_listen_network: tcp
  grpc_listen_address: ""
  grpc_listen_port: 0
  grpc_listen_conn_limit: 0
  tls_cipher_suites: ""
  tls_min_version: ""
  http_tls_config:
    cert: ""
    key: null
    client_ca: ""
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert: ""
    key: null
    client_ca: ""
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: true
  report_grpc_codes_in_instrumentation_label_enabled: false
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 30s
  http_server_read_header_timeout: 0s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 2m0s
  http_log_closed_connections_without_response_enabled: false
  grpc_server_max_recv_msg_size: 4194304
  grpc_server_max_send_msg_size: 4194304
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_connection_idle: 2562047h47m16.854775807s
  grpc_server_max_connection_age: 2562047h47m16.854775807s
  grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
  grpc_server_keepalive_time: 2h0m0s
  grpc_server_keepalive_timeout: 20s
  grpc_server_min_time_between_pings: 5m0s
  grpc_server_ping_without_stream_allowed: false
  grpc_server_num_workers: 0
  log_format: logfmt
  log_level: debug
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  log_request_headers: false
  log_request_at_info_level_enabled: false
  log_request_exclude_headers_list: ""
  http_path_prefix: ""
  external_url: ""
  health_check_target: null
  disable: false
  profiling_enabled: false
  enable_runtime_reload: false
client:
  url: ""
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  enable_http2: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  drop_rate_limited_batches: false
clients:
- url: http://localhost:3100/loki/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: true
  enable_http2: true
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  drop_rate_limited_batches: false
positions:
  sync_period: 10s
  filename: /tmp/positions.yaml
  ignore_invalid_yaml: false
scrape_configs:
- job_name: system
  static_configs:
  - targets:
    - localhost
    labels:
      __path__: /var/log/grafana/ingest/*log
      job: varlogs
target_config:
  sync_period: 10s
  stdin: false
limits_config:
  readline_rate: 10000
  readline_burst: 10000
  readline_rate_drop: true
  max_streams: 0
  max_line_size: 0B
  max_line_size_truncate: false
tracing:
  enabled: true
  profiling_enabled: true
wal:
  enabled: false
  dir: ""
  cleanSegmentsOlderThan: 0s
  watchConfig:
    minReadFrequency: 0s
    maxReadFrequency: 0s```

Sytem (1/1) ready with `discovered labels` and `target labels` correctly mentioned.

Is there any problem with the grafana server binary in this case? 


* Did you follow any online instructions? If so, what is the URL?
https://grafana.com/docs/loki/latest/operations/troubleshooting/