Hi All,
We are trying to configure loki recording rule to send metrics to mimir backend and view metric via grafana UI.
unforunatly we were not able to setup this asexpected, notsure where we are failing,
belowi is the configuration from loki ruler :
ruler:
# external_url: http://grafana-test.grafana-test.svc.cluster.local
enable_api: true
evaluation_interval: 1m
poll_interval: 1m
storage:
type: s3
s3:
bucketnames: loki-7boss-test-west-test
region: us-west-2
s3forcepathstyle: true
# local:
# directory: /rules
rule_path: /rules
# wal:
# dir: /loki/ruler-wal
# ring:
# kvstore:
# store: memberlist
# enable_sharding: true
# alertmanager_url: http://prometheus-test-alertmanager.prometheus-test.svc.cluster.local:9093
# enable_alertmanager_v2: true
remote_write:
enabled: true
config_refresh_period: 10s
add_org_id_header: true # true
clients:
- url: http://mimir-test-distributor.mimir-test.svc.cluster.local:8080/api/v1/push
name: "mimir"
remote_timeout: 30s
send_exemplars: false
send_native_histograms: false
round_robin_dns: false
write_relabel_configs:
- source_labels: [__name__]
target_label: source
replacement: testRule
# queue_config:
# capacity: 10000 # 500
# max_shards: 50 # 100
# min_shards: 1
# max_samples_per_send: 10000
# batch_send_deadline: 5s
# evaluation:
# mode: remote
# query_frontend:
# address: <QF_ADDRESS>
Mimir configuration:
api:
prometheus_http_prefix: /prometheus
response_compression_enabled: true
limits:
out_of_order_time_window: 5m
ingestion_rate: 10000
ingestion_burst_size: 200000
serviceAccount:
# -- Whether to create a service account or not. In case 'create' is false, do set 'name' to an existing service account name.
create: true
annotations:
"eks.amazonaws.com/role-arn": "arn:aws:iam::<11111>:role/mimirServiceAccountRole-testCluster"
alertmanager:
enabled: false
persistentVolume:
enabled: true
storageClass: mimir-ebs-sc
size: 2Gi
accessModes:
- ReadWriteOnce
name: storage
whenDeleted: Delete
whenScaled: Delete
replicas: 1
statefulSet:
enabled: false
compactor:
enabled: true
replicas: 1
persistentVolume:
enabled: true
storageClass: mimir-ebs-sc
size: 2Gi
accessModes:
- ReadWriteOnce
name: storage
whenDeleted: Retain
whenScaled: Retain
distributor:
replicas: 3
replication_factor: 1
ring:
kvstore:
store: memberlist
ingester:
persistentVolume:
enabled: true
storageClass: mimir-ebs-sc
size: 2Gi # size: 8Gi
accessModes:
- ReadWriteOnce
name: storage
whenDeleted: Delete
whenScaled: Retain
replicas: 1
replication_factor: 1
lifecycler:
ring:
kvstore:
store: memberlist
topologySpreadConstraints: {}
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: target # support for enterprise.legacyLabels
operator: In
values:
- ingester
topologyKey: 'kubernetes.io/hostname'
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- ingester
topologyKey: 'kubernetes.io/hostname'
zoneAwareReplication:
enabled: true
topologyKey: 'kubernetes.io/hostname'
extraArgs:
"ingester.max-global-series-per-user": 990000
# - --ingester.max-series-per-user=500000
admin-cache:
enabled: false
replicas: 1
chunks-cache:
enabled: false
replicas: 1
index-cache:
enabled: false
replicas: 1
metadata-cache:
enabled: false
replicas: 1
results-cache:
enabled: false
replicas: 1
minio:
enabled: false
mimir:
structuredConfig:
common:
storage:
backend: s3
s3:
endpoint: s3.us-west-2.amazonaws.com
region: us-west-2
blocks_storage:
s3:
bucket_name: mimir-testcluster-blocks
alertmanager_storage:
s3:
bucket_name: mimir-testcluster-alertmanager
ruler_storage:
s3:
bucket_name: mimir-testcluster-ruler
overrides_exporter:
replicas: 1
querier:
replicas: 2
replication_factor: 1
query_frontend:
replicas: 1
ruler:
replicas: 2
alertmanager_url: http://prometheus-test-alertmanager.prometheus-test.svc.cluster.local:9093
replication_factor: 1
# kubectl delete statefulset mimir-test-store-gateway -n mimir-test --cascade=orphan
store_gateway:
persistentVolume:
enabled: true
storageClass: mimir-ebs-sc
size: 2Gi # size: 8Gi
accessModes:
- ReadWriteOnce
name: storage
whenDeleted: Delete
whenScaled: Retain
replicas: 2
replication_factor: 1
topologySpreadConstraints: {}
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: target # support for enterprise.legacyLabels
operator: In
values:
- store-gateway
topologyKey: 'kubernetes.io/hostname'
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- store-gateway
topologyKey: 'kubernetes.io/hostname'
zoneAwareReplication:
enabled: true
# topologyKey: 'kubernetes.io/hostname'
nginx:
replicas: 1
# Grafana Enterprise Metrics feature related
admin_api:
replicas: 1
gateway:
replicas: 1
loki recording rule:
name: "testRule"
interval: 1m
rules:
- record: avg:execution_time:5m
expr: |
avg_over_time(
{job="test"}
| logfmt
| regexp "executionTime: (?P<execution_time>[0-9]+)"
| unwrap execution_time [5m])
labels:
job: test
environment: dev
loki ruler logs -
I see ruler is evaluation the recording rule, but not sure, whethere its trying to send anything to mimir,
No related logs in mimir,
level=info ts=2025-04-25T13:20:18.225955479Z caller=metrics.go:230 component=ruler evaluation_mode=local org_id=fake latency=fast query=“avg_over_time({job="test"} | logfmt | regexp "executionTime: (?P<execution_time>[0-9]+)" | unwrap execution_time[5m])” query_hash=1335626740 query_type=metric range_type=instant length=0s start_delta=4.447843ms end_delta=4.448034ms step=0s duration=3.318371ms status=200 limit=0 returned_lines=0 throughput=1.4MB total_bytes=4.6kB total_bytes_structured_metadata=912B lines_per_second=34354 total_lines=114 post_filter_lines=114 total_entries=1 store_chunks_download_time=0s queue_time=0s splits=0 shards=0 query_referenced_structured_metadata=false pipeline_wrapper_filtered_lines=0 chunk_refs_fetch_time=2.167259ms cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_volume_results_req=0 cache_volume_results_hit=0 cache_volume_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s cache_result_query_length_served=0s ingester_chunk_refs=0 ingester_chunk_downloaded=0 ingester_chunk_matches=1 ingester_requests=6 ingester_chunk_head_bytes=4.6kB ingester_chunk_compressed_bytes=0B ingester_chunk_decompressed_bytes=0B ingester_post_filter_lines=114 congestion_control_latency=0s index_total_chunks=0 index_post_bloom_filter_chunks=0 index_bloom_filter_ratio=0.00 index_used_bloom_filters=false index_shard_resolver_duration=0s disable_pipeline_wrappers=false has_labelfilter_before_parser=false
ts=2025-04-25T13:20:26.2888288Z caller=memberlist_logger
Can anyone please assist, what are we missing here,
ThanksMuch,
