Messages Log showing "authentication error: invalid authentication credentials\" every second

Grafana Authentication
1

  • What Grafana version and what operating system are you using?
    Grafana Cloud

  • What are you trying to achieve?
    Collecting logs using the promtail service.

  • How are you trying to achieve it?
    I created a Loki data source. The promtail service is installed on each instance.

  • What happened?
    Logs are being collected successfully–everything looks good from the Grafana Cloud UI.

  • What did you expect to happen?
    Logs being collected successfully.

  • Can you copy/paste the configuration(s) that you are having problems with?
    The promtail config.yml contains:
    ==============
    server:
    http_listen_port: 0
    grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

client:
url: https://:@/api/prom/push
[ Using data/credentials from the Loki datasource set up in the Grafana Cloud UI]

scrape_configs:

  • job_name: classworks_common
    pipeline_stages:
    • replace:
      expression: “password (\S+)”
      replace: “****”
    • replace:
      expression: “password=(\S+)”
      replace: “****”
    • replace:
      expression: “passwd (\S+)”
      replace: “****”
    • replace:
      expression: “passwd=(\S+)”
      replace: “****”
    • replace:
      expression: “sessionPassword=(\S+)”
      replace: “****”
    • replace:
      expression: “key: (\S+)”
      replace: “****”
    • replace:
      expression: “secret: (\S+)”
      replace: “****”
    • replace:
      expression: “secret-id: (\S+)”
      replace: “****”
    • replace:
      expression: “userid=(\S+)”
      replace: “****”
      static_configs:
    • labels:
      job: catalina.log
      stack: Production
      layer: cw-production-application
      instance: cw-production-application3
      path: /usr/share/apache-tomcat-9.0.70/logs/catalina.log
    • labels:
      job: catalina.out
      stack: Production
      layer: cw-production-application
      instance: cw-production-application3
      path: /usr/share/apache-tomcat-9.0.70/logs/catalina.out
      ==============
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    The /var/log/messages log is receiving the following log record once per second:

Jan 25 08:45:29 cw-production-application3 agent-linux-amd64: ts=2023-01-25T13:45:29.161780764Z caller=client.go:390 level=error component=logs logs_config=integrations component=client host= msg=“final error sending batch” status=401 error=“server returned HTTP status 401 Unauthorized (401): {"status":"error","error":"authentication error: invalid authentication credentials"}”

  • Did you follow any online instructions? If so, what is the URL?