401 final error sending batch server returned HTTP status 401 Unauthorized (401)

Anybody familiar with this error?
I guess I am not replacing a value somewhere, but I can’t find the correct item to replace.

Nov 26 23:50:45 raspberry grafana-agent[2862]: ts=2023-11-26T22:50:45.231201431Z caller=client.go:430 level=error component=logs logs_config=integrations component=client host=logs-prod-012.grafana.net msg=“final error sending batch” status=401 tenant= error=“server returned HTTP status 401 Unauthorized (401): {"status":"error","error":"authentication error: invalid token"}”

I try to install the Grafana agent on a raspberry pi version 3b

  1. Install Grafana Agent : test ok

  2. Make configuration selections

Choose your OS

Select your OS
Debian - based
Architecture
Arm64

  1. Prepare your agent configuration file

yaml file:

integrations:
  prometheus_remote_write:
  - basic_auth:
      password: xxx 
      username: 123456
    url: https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
    - action: replace
      target_label: job
      replacement: "integrations/agent-check"
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: 'raspberrypi3'
      target_label: instance
    - replacement: integrations/raspberrypi-node
      target_label: job
    metric_relabel_configs:
    - action: keep
      regex: node_cpu_seconds_total|node_disk_io_time_seconds_total|node_disk_io_time_weighted_seconds_total|node_disk_read_bytes_total|node_disk_written_bytes_total|node_filesystem_avail_bytes|node_filesystem_files|node_filesystem_files_free|node_filesystem_readonly|node_filesystem_size_bytes|node_hwmon_temp_celsius|node_load1|node_load15|node_load5|node_memory_Buffers_bytes|node_memory_Cached_bytes|node_memory_MemAvailable_bytes|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_memory_Slab_bytes|node_network_receive_bytes_total|node_network_receive_drop_total|node_network_receive_errs_total|node_network_receive_packets_total|node_network_transmit_bytes_total|node_network_transmit_drop_total|node_network_transmit_errs_total|node_network_transmit_packets_total|node_uname_info|node_vmstat_pgmajfault
      source_labels:
      - __name__

logs:
  configs:
  - clients:
    - basic_auth:
        password: xxxx
        username: 745xxx (=different user name)
      url: https://logs-prod-012.grafana.net/loki/api/v1/push
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        max_age: 24h
        labels:
          instance: 'raspberrypi3'
          job: integrations/raspberrypi-node
      relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__boot_id']
        target_label: 'boot_id'
      - source_labels: ['__journal__transport']
        target_label: 'transport'
      - source_labels: ['__journal_priority_keyword']
        target_label: 'level'
    - job_name: integrations/node_exporter_direct_scrape
      static_configs:
      - targets:
        - localhost
        labels:
          instance: 'raspberrypi3'
          __path__: /var/log/{syslog,messages,*.log}
          job: integrations/raspberrypi-node

metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: xxx
        username: 129xxxx
      url: https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
    scrape_configs:
      # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.

  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal

Any ideas/suggestions?

Have you tested the username / password on your Loki endpoint via the API?

Thanks for the tip, @tonyswumac .

I didn’t, but I’m trying now.

Can I use Postman?

POST https://logs-prod-012.grafana.net/loki/api/v1/push
Authorisation : Bearer Token = token ID in my Access Policy on Grafana Cloud
returns 401 Unauthorized

POST https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
Authorisation : Bearer Token = token ID in my Access Policy on Grafana Cloud
returns 401 Unauthorized

POST https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
Authorisation : Basic Auth with username & pwd in Yaml file (generated by Grafana, looks encrypted)
returns 401 Unauthorized

FYI: I also added my public IP address to my access policy in Grafana cloud, no change

I’d double check the username / password you are using for Loki then and make sure they are correct.

Couldn’t find the right values to use.
So I removed the grafana service, did an update of the system, even deleted the grafana token on the cloud, and reinstallled everything.
I had to manually create a /etc/grafana-agent.yaml (error: invalid permissions)
now it works fine

No idea what was going wrong, but it works now.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.