Forwarding logs to external lokistack with vector

Grafana Loki
1

Hello, I having been trying to setup a grafana loki stack instance on an OpenShift cluster that aggregates logs from many OpenShift cluster. I am using vector as the log forwarder on the source clusters which forwards to the lokistack on the one destination cluster, but I have been running into 302 errors when trying to forward the logs to the external lokistack route endpoint. Does anyone have any ideas on what I am doing wrong here? I am using openshift-logging tenant mode which creates application, infrastructure, and audit tenants. I tried exposing the gateway http service without tls, but I get a 400 error saying that I am sending http packets to an http server. Thank you in advance.

LokiStack:

apiVersion: loki.grafana.com/v1

kind: LokiStack

metadata:

name: logging-loki-grafana

namespace: loki-logging-grafana

spec:

hashRing:

type: memberlist

limits:

global:

  ingestion:

    ingestionBurstSize: 32

    ingestionRate: 24

    maxLineSize: 256000

    perStreamRateLimit: 5

    perStreamRateLimitBurst: 20

  queries:

    queryTimeout: 10m

  retention:

    days: 7

managementState: Managed

size: 1x.pico

storage:

schemas:

  - effectiveDate: '2020-10-11'

    version: v13

secret:

  name: logging-loki-s3-grafana

  type: s3

tls:

  caName: loki-s3-ca-grafana

storageClassName: volumes-primary

tenants:

mode: openshift-logging

Vector Cluster Log Forwarder:

apiVersion: observability.openshift.io/v1

kind: ClusterLogForwarder

metadata:

name: lokistack-logging

namespace: loki-log-forwarder

spec:

managementState: Managed

outputs:

- loki:

    tenantKey: infrastructure

    url: 'https://logging-loki-grafana-loki-logging-grafana.apps.ocpadm.fedins.com/api/logs/v1/infrastructure'

  name: loki-infra

  tls:

    ca:

      key: ca-bundle.crt

      secretName: fed-ca-bundle

  type: loki

pipelines:

- inputRefs:

    - infrastructure

  name: send-infra-logs

  outputRefs:

    - loki-infra

serviceAccount:

name: loki-forwarder

Cluster Log Forwarder Pods Logs:

2025-12-05T16:07:00.711934Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

2025-12-05T16:07:09.704677Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been suppressed 7 times.

2025-12-05T16:07:09.704907Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 302 Found internal_log_rate_limit=true

2025-12-05T16:07:09.705288Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] has been suppressed 7 times.

2025-12-05T16:07:09.705301Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServerError { code: 302 }) request_id=1795 error_type=“request_failed” stage=“sending” internal_log_rate_limit=true

2025-12-05T16:07:09.705589Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 7 times.

2025-12-05T16:07:09.705594Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=2 reason=“Service call failed. No retries or retries exhausted.” internal_log_rate_limit=true

2025-12-05T16:07:10.734796Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding.

2025-12-05T16:07:10.734924Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.

2025-12-05T16:07:10.734968Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

2025-12-05T16:07:20.728491Z ERROR sink{component_kind=“sink” component_id=output_loki_infra component_type=loki}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been suppressed 8 times.