Remote_write configuration to push metrics from loki ruler to prometheus tsdb endpoint

Grafana Alerting
, , ,
1

Hi,
I’m trying to configure Loki Recording rules, sending metrics to Prometheus write endpoint.
Loki (distributed Mode) and Prometheus are deployed via helm.

Below is the simple rules configured,

name: "testlogpushedrule-7thApril"
interval: 1m
rules:
  - record: testlog_7thApril
    expr: |
        count_over_time({job="test"}[1m])

I’m trying to load rules into Rules S3 via
https://loki-test.abc.com/loki/api/v1/rules/my_namespace

When I try to get rules via loki endpoint -
https://loki-test.abc.com/loki/api/v1/rules

I’m able to see rules configured, same thing is visible in GrafanaUI aswell under datasource managed recording rule

image
image2794×1483 290 KB

Ruler configuration passed via loki settings -

loki:
  ruler:
      storage:
        type: s3
        s3:
          bucketnames: loki-123-test-west-test
          region: us-west-2 
          s3forcepathstyle: true
      rule_path: /rules  # Temporary path to store rule files
      wal:
        dir: /var/loki/ruler-wal
      evaluation_interval: 2m
      ring:
        kvstore:
          store: memberlist
      enable_api: true
      enable_sharding: true
      remote_write:
        enabled: true
        config_refresh_period: 10s
        add_org_id_header: true  # true
        clients:
          - url: http://prometheus-test-server.prometheus-test.svc.cluster.local:80/api/v1/write   #"https://prometheus-test.abc.com/api/v1/write"
            name: "prometheus-test-AVGexecutionTime"
            remote_timeout: 30s
            send_exemplars: false
            send_native_histograms: false
            round_robin_dns: false
            # write_relabel_configs:
            #   - source_labels: [__name__]
            #     target_label: source
            #     replacement: loki_recording_rules
            # queue_config:
            #   capacity: 10000 # 500
            #   max_shards: 50  # 100
            #   min_shards: 1
            #   max_samples_per_send: 10000
            #   batch_send_deadline: 5s
        # evaluation:
        #   mode: remote
        #   query_frontend:
        #     address: <QF_ADDRESS>

Query -

  1. I’m unable to see metric -testlog_7thApril in Prometheus. its says NODATA, which means either loki is not sending metrics or prometheus is unable to receive metrics.
  2. looked into all possible workaround, nothing works,
  3. Unfortunatly, I don’t see any logs related to remote_write either in loki ruler/prometheus/

Please Assist,

Thanks Much for the support.