401 final error sending batch server returned HTTP status 401 Unauthorized (401)

Grafana Loki
1

Anybody familiar with this error?
I guess I am not replacing a value somewhere, but I can’t find the correct item to replace.

Nov 26 23:50:45 raspberry grafana-agent[2862]: ts=2023-11-26T22:50:45.231201431Z caller=client.go:430 level=error component=logs logs_config=integrations component=client host=logs-prod-012.grafana.net msg=“final error sending batch” status=401 tenant= error=“server returned HTTP status 401 Unauthorized (401): {"status":"error","error":"authentication error: invalid token"}”

I try to install the Grafana agent on a raspberry pi version 3b

  1. Install Grafana Agent : test ok

  2. Make configuration selections

Choose your OS

Select your OS
Debian - based
Architecture
Arm64

  1. Prepare your agent configuration file

yaml file:

integrations:
  prometheus_remote_write:
  - basic_auth:
      password: xxx 
      username: 123456
    url: https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
    - action: replace
      target_label: job
      replacement: "integrations/agent-check"
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: 'raspberrypi3'
      target_label: instance
    - replacement: integrations/raspberrypi-node
      target_label: job
    metric_relabel_configs:
    - action: keep
      regex: node_cpu_seconds_total|node_disk_io_time_seconds_total|node_disk_io_time_weighted_seconds_total|node_disk_read_bytes_total|node_disk_written_bytes_total|node_filesystem_avail_bytes|node_filesystem_files|node_filesystem_files_free|node_filesystem_readonly|node_filesystem_size_bytes|node_hwmon_temp_celsius|node_load1|node_load15|node_load5|node_memory_Buffers_bytes|node_memory_Cached_bytes|node_memory_MemAvailable_bytes|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_memory_Slab_bytes|node_network_receive_bytes_total|node_network_receive_drop_total|node_network_receive_errs_total|node_network_receive_packets_total|node_network_transmit_bytes_total|node_network_transmit_drop_total|node_network_transmit_errs_total|node_network_transmit_packets_total|node_uname_info|node_vmstat_pgmajfault
      source_labels:
      - __name__

logs:
  configs:
  - clients:
    - basic_auth:
        password: xxxx
        username: 745xxx (=different user name)
      url: https://logs-prod-012.grafana.net/loki/api/v1/push
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        max_age: 24h
        labels:
          instance: 'raspberrypi3'
          job: integrations/raspberrypi-node
      relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__boot_id']
        target_label: 'boot_id'
      - source_labels: ['__journal__transport']
        target_label: 'transport'
      - source_labels: ['__journal_priority_keyword']
        target_label: 'level'
    - job_name: integrations/node_exporter_direct_scrape
      static_configs:
      - targets:
        - localhost
        labels:
          instance: 'raspberrypi3'
          __path__: /var/log/{syslog,messages,*.log}
          job: integrations/raspberrypi-node

metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: xxx
        username: 129xxxx
      url: https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
    scrape_configs:
      # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.

  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal
2

Any ideas/suggestions?

3

Have you tested the username / password on your Loki endpoint via the API?

4

Thanks for the tip, @tonyswumac .

I didn’t, but I’m trying now.

Can I use Postman?

POST https://logs-prod-012.grafana.net/loki/api/v1/push
Authorisation : Bearer Token = token ID in my Access Policy on Grafana Cloud
returns 401 Unauthorized

POST https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
Authorisation : Bearer Token = token ID in my Access Policy on Grafana Cloud
returns 401 Unauthorized

POST https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push
Authorisation : Basic Auth with username & pwd in Yaml file (generated by Grafana, looks encrypted)
returns 401 Unauthorized

FYI: I also added my public IP address to my access policy in Grafana cloud, no change

5

I’d double check the username / password you are using for Loki then and make sure they are correct.

6

Couldn’t find the right values to use.
So I removed the grafana service, did an update of the system, even deleted the grafana token on the cloud, and reinstallled everything.
I had to manually create a /etc/grafana-agent.yaml (error: invalid permissions)
now it works fine

No idea what was going wrong, but it works now.