Self Host Grafana sending logs to Promtail and then to Loki

Grafana Configuration
1

  • What Grafana version and what operating system are you using?
    v10

  • What are you trying to achieve?
    Sending the logs via Grafana Agent to Promtail and then to Loki

  • How are you trying to achieve it?
    Install and configre grafana agent on Linux server with sending logs to http://promtail:9089/api/v1/prom
    Promtail already configure and pointing to Loki

  • What happened?
    msg=“final error sending batch” status=404 tenant= error=“server returned HTTP status 404 Not Found (404): 404 page not found”

  • What did you expect to happen?
    Expect transfer the logs to Loki

  • Can you copy/paste the configuration(s) that you are having problems with?

Blockquote

Grafana agent
integrations:
node_exporter:
enabled: true
# disable unused collectors
disable_collectors:
- ipvs #high cardinality on kubelet
- btrfs
- infiniband
- xfs
- zfs
# exclude dynamic interfaces
netclass_ignored_devices: “^(veth.|cali.|[a-f0-9]{15})$”
netdev_device_exclude: “^(veth.|cali.|[a-f0-9]{15})$”
# disable tmpfs
filesystem_fs_types_exclude: “^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$”
# drop extensive scrape statistics
metric_relabel_configs:
- action: drop
regex: node_scrape_collector_.+
source_labels: [name]
relabel_configs:
- replacement: testvm01
target_label: instance
prometheus_remote_write:

  • basic_auth:
    password: 12521521
    username: 12521521
    url: http://prometheus:9090/api/v1/write
    agent:
    enabled: true
    relabel_configs:
    • action: replace
      source_labels:
      • agent_hostname
        target_label: instance
    • action: replace
      target_label: job
      replacement: “integrations/agent-check”
      metric_relabel_configs:
    • action: keep
      regex: (prometheus_target_.|prometheus_sd_discovered_targets|agent_build.|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      • name

logs:
configs:

  • clients:

    • url: http://promtail:3100/loki/api/v1/push
      name: integrations
      positions:
      filename: /tmp/positions.yaml
      scrape_configs:

    • job_name: integrations/node_exporter_journal_scrape
      journal:
      max_age: 24h
      labels:
      instance: testvm01
      job: integrations/node_exporter
      relabel_configs:

      • source_labels: [‘__journal__systemd_unit’]
        target_label: ‘unit’
      • source_labels: [‘__journal__boot_id’]
        target_label: ‘boot_id’
      • source_labels: [‘__journal__transport’]
        target_label: ‘transport’
      • source_labels: [‘__journal_priority_keyword’]
        target_label: ‘level’

    • job_name: integrations/node_exporter_direct_scrape
      static_configs:

      • targets:
        • localhost
          labels:
          instance: testvm01
          path: /var/log/{syslog,messages}
          job: integrations/node_exporter
          metrics:
          configs:
  • name: integrations
    remote_write:

global:
scrape_interval: 60s
wal_directory: /tmp/grafana-agent-wal

Promtail configmap
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
external_labels:
job: promtail

Blockquote

2

Can you edit your post its unreadable

3

updated, sorry for mistakes