Alloy don't find targets of logfiles

Grafana Alloy
1

Hi all,
I’m totally confused at the moment.

I setted up alloy, loki and on another host grafana and want to tail my logs and show them in Grafana.

First I set up the loki config

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

ingester_rf1:
  enabled: false

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    enabled: true
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

Second I have defined the alloy config

logging {
  level = "debug"
  format = "logfmt"
}

livedebugging {
  enabled = true
}

loki.source.syslog "local" {
  listener {
    address  = "127.0.0.1:1514"
    labels   = { component = "loki.source.syslog", protocol = "tcp" }
  }

  listener {
    address  = "127.0.0.1:1514"
    protocol = "udp"
    labels   = { component = "loki.source.syslog", protocol = "udp"}
  }

  forward_to = [loki.write.grafana_loki.receiver]
}

prometheus.exporter.unix "default" {
  include_exporter_metrics = true
  disable_collectors       = ["mdadm"]
}

prometheus.scrape "default" {
  targets = array.concat(
    prometheus.exporter.unix.default.targets,
    [{
      // Self-collect metrics
      job         = "alloy",
      __address__ = "127.0.0.1:12345",
    }],
  )


  forward_to = [
  // TODO: components to forward metrics to (like prometheus.remote_write or
  // prometheus.relabel).
  ]
}

local.file_match "local_files" {
   path_targets = [{"__path__" = "/var/log/*.log"}]
   sync_period = "5s"
}

local.file_match "local_syslog" {
   path_targets = [{"__path__" = "/var/log/syslog"}]
   sync_period = "5s"
}

local.file_match "local_subfolder_log" {
    path_targets = [{"__path__" = "/var/log/**/*.log"}]
    sync_period = "5s"
 }

local.file_match "local_router_log" {
    path_targets = [{"__path__" = "/var/log/router/**/*.log"}]
    sync_period = "5s"
 }


loki.source.file "log_scrape" {
  targets    = local.file_match.local_files.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "syslog_scrape" {
  targets    = local.file_match.local_syslog.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "subfolder_log_scrape" {
  targets    = local.file_match.local_subfolder_log.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "router_log_scrape" {
  targets    = local.file_match.local_router_log.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.write "grafana_loki" {
  endpoint {
    url = "http://localhost:3100/loki/api/v1/push"
  }
}

After this a few of my logfiles will be shown in Grafana Explore and also in Alloy Metrics

metrics 
# HELP alloy_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which alloy was built, and the goos and goarch for the build.
# TYPE alloy_build_info gauge
alloy_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.22.7",revision="f14249012",tags="netgo,builtinassets,promtail_journal_enabled",version="v1.4.1"} 1
# HELP alloy_component_controller_evaluating Tracks if the controller is currently in the middle of a graph evaluation
# TYPE alloy_component_controller_evaluating gauge
alloy_component_controller_evaluating{controller_id="",controller_path="/"} 0
alloy_component_controller_evaluating{controller_id="remotecfg",controller_path="/"} 0
# HELP alloy_component_controller_running_components Total number of running components.
# TYPE alloy_component_controller_running_components gauge
alloy_component_controller_running_components{controller_id="",controller_path="/",health_type="healthy"} 12
# HELP alloy_component_evaluation_queue_size Tracks the number of components waiting to be evaluated in the worker pool
# TYPE alloy_component_evaluation_queue_size gauge
alloy_component_evaluation_queue_size{controller_id="",controller_path="/"} 1
alloy_component_evaluation_queue_size{controller_id="remotecfg",controller_path="/"} 0
# HELP alloy_component_evaluation_seconds Time spent performing component evaluation
# TYPE alloy_component_evaluation_seconds histogram
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="0.005"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="0.025"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="0.1"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="0.5"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="1"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="5"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="10"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="30"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="60"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="120"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="300"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="600"} 10
alloy_component_evaluation_seconds_bucket{controller_id="",controller_path="/",le="+Inf"} 10
alloy_component_evaluation_seconds_sum{controller_id="",controller_path="/"} 0.010221227000000001
alloy_component_evaluation_seconds_count{controller_id="",controller_path="/"} 10
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="0.005"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="0.025"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="0.1"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="0.5"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="1"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="5"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="10"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="30"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="60"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="120"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="300"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="600"} 0
alloy_component_evaluation_seconds_bucket{controller_id="remotecfg",controller_path="/",le="+Inf"} 0
alloy_component_evaluation_seconds_sum{controller_id="remotecfg",controller_path="/"} 0
alloy_component_evaluation_seconds_count{controller_id="remotecfg",controller_path="/"} 0
# HELP alloy_config_hash Hash of the currently active config file.
# TYPE alloy_config_hash gauge
alloy_config_hash{cluster_name="",sha256="315b7a327ec84490db2b6e561a3dc752e9dea1cb58c52729a48452d968dc6f0a"} 1
# HELP alloy_config_last_load_success_timestamp_seconds Timestamp of the last successful configuration load.
# TYPE alloy_config_last_load_success_timestamp_seconds gauge
alloy_config_last_load_success_timestamp_seconds 1.7279661694588945e+09
# HELP alloy_config_last_load_successful Config loaded successfully.
# TYPE alloy_config_last_load_successful gauge
alloy_config_last_load_successful 1
# HELP alloy_config_load_failures_total Configuration load failures.
# TYPE alloy_config_load_failures_total counter
alloy_config_load_failures_total 0
# HELP alloy_labelstore_global_ids_count Total number of global ids.
# TYPE alloy_labelstore_global_ids_count gauge
alloy_labelstore_global_ids_count 1169
# HELP alloy_labelstore_last_stale_check_timestamp Last time stale check was ran expressed in unix timestamp.
# TYPE alloy_labelstore_last_stale_check_timestamp gauge
alloy_labelstore_last_stale_check_timestamp 0
# HELP alloy_resources_machine_rx_bytes_total Total bytes, host-wide, received across all network interfaces.
# TYPE alloy_resources_machine_rx_bytes_total counter
alloy_resources_machine_rx_bytes_total 8.6907084e+07
# HELP alloy_resources_machine_tx_bytes_total Total bytes, host-wide, sent across all given network interface.
# TYPE alloy_resources_machine_tx_bytes_total counter
alloy_resources_machine_tx_bytes_total 9.6651541e+07
# HELP alloy_resources_process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE alloy_resources_process_cpu_seconds_total counter
alloy_resources_process_cpu_seconds_total 5.16
# HELP alloy_resources_process_resident_memory_bytes Current resident memory size in bytes.
# TYPE alloy_resources_process_resident_memory_bytes gauge
alloy_resources_process_resident_memory_bytes 1.85729024e+08
# HELP alloy_resources_process_start_time_seconds Start time of the process since Unix epoch in seconds.
# TYPE alloy_resources_process_start_time_seconds gauge
alloy_resources_process_start_time_seconds 1.7283297e+09
# HELP alloy_resources_process_virtual_memory_bytes Current virtual memory size in bytes.
# TYPE alloy_resources_process_virtual_memory_bytes gauge
alloy_resources_process_virtual_memory_bytes 2.23418368e+09
# HELP deprecated_flags_inuse_total The number of deprecated flags currently set.
# TYPE deprecated_flags_inuse_total counter
deprecated_flags_inuse_total 0
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.729e-05
go_gc_duration_seconds{quantile="0.25"} 5.0213e-05
go_gc_duration_seconds{quantile="0.5"} 0.000113165
go_gc_duration_seconds{quantile="0.75"} 0.000113459
go_gc_duration_seconds{quantile="1"} 0.000129325
go_gc_duration_seconds_sum 0.000561475
go_gc_duration_seconds_count 6
# HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent
# TYPE go_gc_gogc_percent gauge
go_gc_gogc_percent 100
# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes
# TYPE go_gc_gomemlimit_bytes gauge
go_gc_gomemlimit_bytes 9.223372036854776e+18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 226
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.22.7"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 5.409408e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.12430832e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.53346e+06
# HELP go_memstats_frees_total Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 611191
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.309048e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 5.409408e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 9.66656e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.0014592e+07
# HELP go_memstats_heap_objects Number of currently allocated objects. Equals to /gc/heap/objects:objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 185252
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.145728e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.9681152e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7279665838448875e+09
# HELP go_memstats_mallocs_total Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 796443
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 1200
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 480000
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 538560
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 9.3478144e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 460316
# HELP go_memstats_stack_inuse_bytes Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.605632e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.605632e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system. Equals to /memory/classes/total:byte.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.8143768e+07
# HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads
# TYPE go_sched_gomaxprocs_threads gauge
go_sched_gomaxprocs_threads 1
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 11
# HELP loki_experimental_features_in_use_total The number of experimental features in use.
# TYPE loki_experimental_features_in_use_total counter
loki_experimental_features_in_use_total 0
# HELP loki_logql_querystats_duplicates_total Total count of duplicates found while executing LogQL queries.
# TYPE loki_logql_querystats_duplicates_total counter
loki_logql_querystats_duplicates_total 0
# HELP loki_logql_querystats_ingester_sent_lines_total Total count of lines sent from ingesters while executing LogQL queries.
# TYPE loki_logql_querystats_ingester_sent_lines_total counter
loki_logql_querystats_ingester_sent_lines_total 0
# HELP loki_panic_total The total number of panic triggered
# TYPE loki_panic_total counter
loki_panic_total 0
# HELP loki_querier_index_cache_corruptions_total The number of cache corruptions for the index cache.
# TYPE loki_querier_index_cache_corruptions_total counter
loki_querier_index_cache_corruptions_total 0
# HELP loki_querier_index_cache_encode_errors_total The number of errors for the index cache while encoding the body.
# TYPE loki_querier_index_cache_encode_errors_total counter
loki_querier_index_cache_encode_errors_total 0
# HELP loki_querier_index_cache_gets_total The number of gets for the index cache.
# TYPE loki_querier_index_cache_gets_total counter
loki_querier_index_cache_gets_total 0
# HELP loki_querier_index_cache_hits_total The number of cache hits for the index cache.
# TYPE loki_querier_index_cache_hits_total counter
loki_querier_index_cache_hits_total 0
# HELP loki_querier_index_cache_puts_total The number of puts for the index cache.
# TYPE loki_querier_index_cache_puts_total counter
loki_querier_index_cache_puts_total 0
# HELP loki_source_file_file_bytes_total Number of bytes total.
# TYPE loki_source_file_file_bytes_total gauge
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/alternatives.log"} 4831
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/auth.log"} 57263
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/cron.log"} 14320
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/daemon.log"} 5.0073898e+07
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/dpkg.log"} 85211
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/mail.log"} 3206
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/ubuntu-advantage-timer.log"} 1860
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 669864
loki_source_file_file_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/user.log"} 422
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/alternatives.log"} 4831
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/access.log"} 0
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/error.log"} 694
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/other_vhosts_access.log"} 0
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apt/history.log"} 8598
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apt/term.log"} 45808
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/auth.log"} 57263
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/cron.log"} 14320
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/daemon.log"} 5.0073898e+07
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/dpkg.log"} 85211
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/mail.log"} 3206
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/ubuntu-advantage-timer.log"} 1860
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 669864
loki_source_file_file_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/user.log"} 422
loki_source_file_file_bytes_total{component_id="loki.source.file.syslog_scrape",component_path="/",path="/var/log/syslog"} 4.8786179e+07
# HELP loki_source_file_files_active_total Number of active files.
# TYPE loki_source_file_files_active_total gauge
loki_source_file_files_active_total{component_id="loki.source.file.log_scrape",component_path="/"} 9
loki_source_file_files_active_total{component_id="loki.source.file.router_log_scrape",component_path="/"} 0
loki_source_file_files_active_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/"} 14
loki_source_file_files_active_total{component_id="loki.source.file.syslog_scrape",component_path="/"} 1
# HELP loki_source_file_read_bytes_total Number of bytes read.
# TYPE loki_source_file_read_bytes_total gauge
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/alternatives.log"} 4831
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/auth.log"} 57263
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/cron.log"} 14320
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/daemon.log"} 5.0073898e+07
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/dpkg.log"} 85211
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/mail.log"} 3206
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/ubuntu-advantage-timer.log"} 1860
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 669864
loki_source_file_read_bytes_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/user.log"} 422
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/alternatives.log"} 4831
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/access.log"} 0
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/error.log"} 694
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apache2/other_vhosts_access.log"} 0
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apt/history.log"} 8598
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/apt/term.log"} 45808
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/auth.log"} 57263
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/cron.log"} 14320
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/daemon.log"} 5.0073898e+07
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/dpkg.log"} 85211
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/mail.log"} 3206
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/ubuntu-advantage-timer.log"} 1860
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 669864
loki_source_file_read_bytes_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/user.log"} 422
loki_source_file_read_bytes_total{component_id="loki.source.file.syslog_scrape",component_path="/",path="/var/log/syslog"} 4.8786179e+07
# HELP loki_source_file_read_lines_total Number of lines read.
# TYPE loki_source_file_read_lines_total counter
loki_source_file_read_lines_total{component_id="loki.source.file.log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 13
loki_source_file_read_lines_total{component_id="loki.source.file.subfolder_log_scrape",component_path="/",path="/var/log/ubuntu-advantage.log"} 13
loki_source_file_read_lines_total{component_id="loki.source.file.syslog_scrape",component_path="/",path="/var/log/syslog"} 3
# HELP loki_source_syslog_empty_messages_total Total number of empty messages received from syslog
# TYPE loki_source_syslog_empty_messages_total counter
loki_source_syslog_empty_messages_total{component_id="loki.source.syslog.local",component_path="/"} 0
# HELP loki_source_syslog_entries_total Total number of successful entries sent to the syslog target
# TYPE loki_source_syslog_entries_total counter
loki_source_syslog_entries_total{component_id="loki.source.syslog.local",component_path="/"} 137
# HELP loki_source_syslog_parsing_errors_total Total number of parsing errors while receiving syslog messages
# TYPE loki_source_syslog_parsing_errors_total counter
loki_source_syslog_parsing_errors_total{component_id="loki.source.syslog.local",component_path="/"} 0
# HELP loki_write_batch_retries_total Number of times batches has had to be retried.
# TYPE loki_write_batch_retries_total counter
loki_write_batch_retries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",tenant=""} 0
# HELP loki_write_dropped_bytes_total Number of bytes dropped because failed to be sent to the ingester after all retries.
# TYPE loki_write_dropped_bytes_total counter
loki_write_dropped_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="ingester_error",tenant=""} 0
loki_write_dropped_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="line_too_long",tenant=""} 0
loki_write_dropped_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="rate_limited",tenant=""} 0
loki_write_dropped_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="stream_limited",tenant=""} 0
# HELP loki_write_dropped_entries_total Number of log entries dropped because failed to be sent to the ingester after all retries.
# TYPE loki_write_dropped_entries_total counter
loki_write_dropped_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="ingester_error",tenant=""} 0
loki_write_dropped_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="line_too_long",tenant=""} 0
loki_write_dropped_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="rate_limited",tenant=""} 0
loki_write_dropped_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="stream_limited",tenant=""} 0
# HELP loki_write_encoded_bytes_total Number of bytes encoded and ready to send.
# TYPE loki_write_encoded_bytes_total counter
loki_write_encoded_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100"} 17811
# HELP loki_write_mutated_bytes_total The total number of bytes that have been mutated.
# TYPE loki_write_mutated_bytes_total counter
loki_write_mutated_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="ingester_error",tenant=""} 0
loki_write_mutated_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="line_too_long",tenant=""} 0
loki_write_mutated_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="rate_limited",tenant=""} 0
loki_write_mutated_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="stream_limited",tenant=""} 0
# HELP loki_write_mutated_entries_total The total number of log entries that have been mutated.
# TYPE loki_write_mutated_entries_total counter
loki_write_mutated_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="ingester_error",tenant=""} 0
loki_write_mutated_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="line_too_long",tenant=""} 0
loki_write_mutated_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="rate_limited",tenant=""} 0
loki_write_mutated_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",reason="stream_limited",tenant=""} 0
# HELP loki_write_request_duration_seconds Duration of send requests.
# TYPE loki_write_request_duration_seconds histogram
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.005"} 61
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.01"} 64
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.025"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.05"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.1"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.25"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="0.5"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="1"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="2.5"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="5"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="10"} 65
loki_write_request_duration_seconds_bucket{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204",le="+Inf"} 65
loki_write_request_duration_seconds_sum{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204"} 0.2094018419999999
loki_write_request_duration_seconds_count{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100",status_code="204"} 65
# HELP loki_write_sent_bytes_total Number of bytes sent.
# TYPE loki_write_sent_bytes_total counter
loki_write_sent_bytes_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100"} 17811
# HELP loki_write_sent_entries_total Number of log entries sent to the ingester.
# TYPE loki_write_sent_entries_total counter
loki_write_sent_entries_total{component_id="loki.write.grafana_loki",component_path="/",host="localhost:3100"} 166
# HELP net_conntrack_dialer_conn_attempted_total Total number of connections attempted by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_attempted_total counter
net_conntrack_dialer_conn_attempted_total{dialer_name="Alloy"} 1
net_conntrack_dialer_conn_attempted_total{dialer_name="prometheus.scrape.default"} 2
# HELP net_conntrack_dialer_conn_closed_total Total number of connections closed which originated from the dialer of a given name.
# TYPE net_conntrack_dialer_conn_closed_total counter
net_conntrack_dialer_conn_closed_total{dialer_name="Alloy"} 0
net_conntrack_dialer_conn_closed_total{dialer_name="prometheus.scrape.default"} 0
# HELP net_conntrack_dialer_conn_established_total Total number of connections successfully established by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_established_total counter
net_conntrack_dialer_conn_established_total{dialer_name="Alloy"} 1
net_conntrack_dialer_conn_established_total{dialer_name="prometheus.scrape.default"} 2
# HELP net_conntrack_dialer_conn_failed_total Total number of connections failed to dial by the dialer a given name.
# HELP net_conntrack_listener_conn_accepted_total Total number of connections opened to the listener of a given name.
# TYPE net_conntrack_listener_conn_accepted_total counter
net_conntrack_listener_conn_accepted_total{listener_name="syslog_target/127.0.0.1:1514"} 1
# HELP net_conntrack_listener_conn_closed_total Total number of connections closed that were made to the listener of a given name.
# TYPE net_conntrack_listener_conn_closed_total counter
net_conntrack_listener_conn_closed_total{listener_name="syslog_target/127.0.0.1:1514"} 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 5.14
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 524288
# HELP process_network_receive_bytes_total Number of bytes received by the process over the network.
# TYPE process_network_receive_bytes_total counter
process_network_receive_bytes_total 8.0541631e+07
# HELP process_network_transmit_bytes_total Number of bytes sent by the process over the network.
# TYPE process_network_transmit_bytes_total counter
process_network_transmit_bytes_total 9.0819808e+07
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 40
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.85729024e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.72796616848e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.23418368e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP prometheus_fanout_latency Write latency for sending to direct and indirect components
# TYPE prometheus_fanout_latency histogram
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.005"} 15
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.01"} 15
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.025"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.05"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.1"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.25"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="0.5"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="1"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="2.5"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="5"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="10"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="30"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="60"} 16
prometheus_fanout_latency_bucket{component_id="prometheus.scrape.default",component_path="/",le="+Inf"} 16
prometheus_fanout_latency_sum{component_id="prometheus.scrape.default",component_path="/"} 0.051084818000000004
prometheus_fanout_latency_count{component_id="prometheus.scrape.default",component_path="/"} 16

# HELP prometheus_scrape_targets_gauge Number of targets this component is configured to scrape
# TYPE prometheus_scrape_targets_gauge gauge
prometheus_scrape_targets_gauge{component_id="prometheus.scrape.default",component_path="/"} 2

# HELP prometheus_target_interval_length_seconds Actual intervals between scrapes.
# TYPE prometheus_target_interval_length_seconds summary
prometheus_target_interval_length_seconds{component_id="prometheus.scrape.default",component_path="/",interval="1m0s",quantile="0.01"} 59.999398403
prometheus_target_interval_length_seconds{component_id="prometheus.scrape.default",component_path="/",interval="1m0s",quantile="0.05"} 59.999398403
prometheus_target_interval_length_seconds{component_id="prometheus.scrape.default",component_path="/",interval="1m0s",quantile="0.5"} 60.00005116
prometheus_target_interval_length_seconds{component_id="prometheus.scrape.default",component_path="/",interval="1m0s",quantile="0.9"} 60.000822276
prometheus_target_interval_length_seconds{component_id="prometheus.scrape.default",component_path="/",interval="1m0s",quantile="0.99"} 60.000842513
prometheus_target_interval_length_seconds_sum{component_id="prometheus.scrape.default",component_path="/",interval="1m0s"} 840.001796565
prometheus_target_interval_length_seconds_count{component_id="prometheus.scrape.default",component_path="/",interval="1m0s"} 14
# HELP prometheus_target_metadata_cache_bytes The number of bytes that are currently used for storing metric metadata in the cache
# TYPE prometheus_target_metadata_cache_bytes gauge
prometheus_target_metadata_cache_bytes{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 35674
# HELP prometheus_target_metadata_cache_entries Total number of metric metadata entries in the cache
# TYPE prometheus_target_metadata_cache_entries gauge
prometheus_target_metadata_cache_entries{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 680

# HELP prometheus_target_scrape_pool_sync_total Total number of syncs that were executed on a scrape pool.
# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 1

# HELP prometheus_target_scrape_pool_targets Current number of targets in this scrape pool.
# TYPE prometheus_target_scrape_pool_targets gauge
prometheus_target_scrape_pool_targets{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 2

# HELP prometheus_target_scrape_pools_total Total number of scrape pool creation attempts.
# TYPE prometheus_target_scrape_pools_total counter
prometheus_target_scrape_pools_total{component_id="prometheus.scrape.default",component_path="/"} 1

# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default",quantile="0.01"} 0.00031688
prometheus_target_sync_length_seconds{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default",quantile="0.05"} 0.00031688
prometheus_target_sync_length_seconds{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default",quantile="0.5"} 0.00031688
prometheus_target_sync_length_seconds{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default",quantile="0.9"} 0.00031688
prometheus_target_sync_length_seconds{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default",quantile="0.99"} 0.00031688
prometheus_target_sync_length_seconds_sum{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 0.00031688
prometheus_target_sync_length_seconds_count{component_id="prometheus.scrape.default",component_path="/",scrape_job="prometheus.scrape.default"} 1

But for example my router logs, which will be sent to /var/log/$HOST/yyyy_mm_dd.log won’t be found.
In the Debug log I found the following entry

Oct 03 14:36:09 logserver alloy[1874]: ts=2024-10-03T14:36:09.440314657Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f2050b23ed5f66a817c32086665fc651 node_id=local.file_match.local_router_log duration=114.855<C2><B5>s
Oct 03 14:36:09 logserver alloy[1874]: ts=2024-10-03T14:36:09.441105009Z level=debug msg="no files targets were passed, nothing will be tailed" component_path=/ component_id=loki.source.file.router_log_scrape

But the logs are definitly present on host and readable.
For Example

root@logserver:/var/log/router/WR1300v3.lan# ls -al
total 644
drw-r--r-- 2 root root   4096 Oct  3 14:53 .
drwxr--r-- 4 root root   4096 Oct  3 14:52 ..
-rw-r--r-- 1 root root 643932 Oct  3 14:53 2024_10_03.log

I have already rebooted, checked several configs and so on but the logs won’t be shown in Grafana / metrics

Does anybody have an idea?

2

Shouldn’t this be path_targets = [{"__path__" = "/var/log/router/*.log"}]?

3

No.
The logs are in a folder per router. E.g. /var/log/WR1300-ap.lan/2024_10_03.log

4

Apologie, misread.

I don’t see anything obviously wrong then. Couple of things I’d suggest trying:

  1. Remove tail_from_end = true when debugging.
  2. Make sure you remove Alloy agent’s state file when debugging.
  3. Try changing the targets to path_targets = [{"__path__" = "/var/log/router/**"}] and see if that picks it up.
5

I have done all you suggestions.

I think with state file you mean the positions.yml. they are located at /var/lib/alloy/
Deleted them and restarted.
But nothing happens

Metrics shows

loki_source_file_files_active_total{component_id="loki.source.file.router_log_scrape",component_path="/"} 0

Journatlctl -u alloy shows

Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.928720101Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=prometheus.scrape.default duration=422.003<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.929178209Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.syslog.local duration=321.666<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.929396744Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.log_scrape duration=166.656<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.929546877Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.router_log_scrape duration=111.322<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.929737041Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.syslog_scrape duration=144.016<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.929873688Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.subfolder_log_scrape duration=100.302<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.935998458Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.936139342Z level=debug msg="logserver @3: participant" service=cluster
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.937003434Z level=info msg="peers changed" service=cluster peers_count=1 peers=logserver
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.938805426Z level=debug msg="passed new targets to scrape manager" component_path=/ component_id=prometheus.scrape.default
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.942327778Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/alternatives.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.94318362Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/auth.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.943760859Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/cron.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.944331532Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/daemon.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.94484838Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/dpkg.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945066887Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/mail.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945285242Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/ubuntu-advantage-timer.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945448412Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/ubuntu-advantage.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945627896Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.log_scrape filename=/var/log/user.log
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945711059Z level=info msg="retrieved and stored the position of the last line" component_path=/ component_id=loki.source.file.log_scrape
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.945816131Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.log_scrape duration=3.791313ms
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.946155024Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.syslog_scrape filename=/var/log/syslog
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.946301873Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.syslog_scrape duration=428.021<C2><B5>s
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.946429459Z level=debug msg="no files targets were passed, nothing will be tailed" component_path=/ component_id=loki.source.file.router_log_scrape
Oct 04 07:00:10 logserver alloy[19993]: ts=2024-10-04T07:00:10.946464339Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.router_log_scrape duration=106.084<C2><B5>s
6

I opened a issue at github for this. Because I think this can be a bug and not a misconfig

7

I made a several test for the subfolder log.
Created a new directory named alloy in /var/log/
Made chmod 766
Stderr of alloy.service to /var/log/alloy_err.log
Restarted alloy service
In my perspective the local.file_match "local_subfolder_log" should find this new files at startup or all 5 seconds

But it won’t be found
Still the service found only Apache and apt logs in subfolders

8

I tried, and I couldn’t reproduce your problem. One thing I’ll point out, after looking at your config more closely, is that your configuration actually generate twice the amount of logs for anything under /var/log/router, because:

local.file_match "local_subfolder_log" {
    path_targets = [{"__path__" = "/var/log/**/*.log"}]
    sync_period = "5s"
 }

local.file_match "local_router_log" {
    path_targets = [{"__path__" = "/var/log/router/**/*.log"}]
    sync_period = "5s"
}

/var/log/**/*.log will also match /var/log/router/**/*.log, which in my test did generate twice the amount of logs for anything under /var/log/router.

Here is my configuration (identical to yours, except I added password for my Loki cluster, and I removed tail_from_end for router logs for testing purpose):

logging {
  level = "debug"
  format = "logfmt"
}

loki.source.syslog "local" {
  listener {
    address  = "127.0.0.1:1514"
    labels   = { component = "loki.source.syslog", protocol = "tcp" }
  }

  listener {
    address  = "127.0.0.1:1514"
    protocol = "udp"
    labels   = { component = "loki.source.syslog", protocol = "udp"}
  }

  forward_to = [loki.write.grafana_loki.receiver]
}

local.file_match "local_files" {
   path_targets = [{"__path__" = "/var/log/*.log"}]
   sync_period = "5s"
}

local.file_match "local_syslog" {
   path_targets = [{"__path__" = "/var/log/syslog"}]
   sync_period = "5s"
}

local.file_match "local_subfolder_log" {
    path_targets = [{"__path__" = "/var/log/**/*.log"}]
    sync_period = "5s"
 }

local.file_match "local_router_log" {
    path_targets = [{"__path__" = "/var/log/router/**/*.log"}]
    sync_period = "5s"
 }


loki.source.file "log_scrape" {
  targets    = local.file_match.local_files.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "syslog_scrape" {
  targets    = local.file_match.local_syslog.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "subfolder_log_scrape" {
  targets    = local.file_match.local_subfolder_log.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.source.file "router_log_scrape" {
  targets    = local.file_match.local_router_log.targets
  forward_to = [loki.write.grafana_loki.receiver]
  # tail_from_end = true
}

loki.write "grafana_loki" {
  external_labels = {
      app = "tonytest",
  }

  endpoint {
    url = "https://<URL>/loki/api/v1/push"

    basic_auth {
      username = "<USER>"
      password = "<PASS>"
    }
  }
}

Label view from my cluster:

Screenshot 2024-10-07 at 3.29.09 PM
Screenshot 2024-10-07 at 3.29.09 PM2756×820 141 KB

Count summed by filename:

Screenshot 2024-10-07 at 3.30.21 PM
Screenshot 2024-10-07 at 3.30.21 PM3296×1294 286 KB

9

I think I’m seeing something similar, I’ll post config and details in the morning.

10

No, it was me, there was a mistake in my config for the file_match, but one that was not flagged by alloy.

This works:

local.file_match "docker" {
  path_targets = [{__path__ = "/var/log/docker.log"}]
}

Notice there are no quotes around path - maybe that makes a difference?

11

I am quite sure in my testing I had quoted __path__.