Hi All
We are using Tempo version 2.1.1. and Mimir 2.6.0 .
We have enabled tempo metrics generator in tempo and we are sending the metrics to Mimir .
We use otel agent for pulling traces from applications/browsers and sending it to Mimir .
Problem Statement
While we are able to see the metrcis pushed from tempo metrics generator to Mimir , however we see a gap at regular interval in our graphs.
Query we use in our graph to visualize errors is as mentioned below -
(sum(
irate(
traces_spanmetrics_calls_total{
deployment_environment=“$deployment_environment”,
service=“$service_name”,
status_code=“STATUS_CODE_ERROR”
}[$__rate_interval]
)
))
Below are our settings for Tempo Application(copied from Argo CD console … Config Map section of tempo.distributed application )
Any inputs would be really helpfull.
apiVersion: v1
data:
overrides.yaml: |
overrides: {}
tempo-query.yaml: |
backend: 127.0.0.1:3100
tempo.yaml: |
compactor:
compaction:
block_retention: 336h
compacted_block_retention: 1h
compaction_cycle: 60s
compaction_window: 1h
max_block_bytes: 107374182400
max_compaction_objects: 6000000
max_time_per_tenant: 60m
retention_concurrency: 10
v2_in_buffer_bytes: 5242880
v2_out_buffer_bytes: 20971520
v2_prefetch_traces_count: 1000
ring:
kvstore:
store: memberlist
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
ring:
kvstore:
store: memberlist
ingester:
flush_check_period: 10s
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 3
tokens_file_path: /var/tempo/tokens.json
max_block_bytes: 524288000
max_block_duration: 20m
trace_idle_period: 10s
memberlist:
abort_if_cluster_join_fails: false
join_members:
- tempo-gossip-ring
metrics_generator:
processor:
service_graphs:
dimensions: []
histogram_buckets:
- 0.1
- 0.2
- 0.4
- 0.8
- 1.6
- 3.2
- 6.4
- 12.8
max_items: 10000
wait: 10s
workers: 10
span_metrics:
dimensions:
- service.namespace
- deployment.environment
- http.method
- http.route
- http.status_code
histogram_buckets:
- 0.002
- 0.004
- 0.008
- 0.016
- 0.032
- 0.064
- 0.128
- 0.256
- 0.512
- 1.02
- 2.05
- 4.1
intrinsic_dimensions:
service: true
span_kind: true
span_name: true
status_code: true
status_message: false
registry:
collection_interval: 60s
external_labels: {}
stale_duration: 15m
ring:
kvstore:
store: memberlist
storage:
path: /var/tempo/wal
remote_write:
- remote_timeout: 30s
send_exemplars: true
url: http://mimir-gateway.mimir-distributed.svc.cluster.local/api/v1/push
remote_write_flush_deadline: 1m
multitenancy_enabled: false
overrides:
ingestion_burst_size_bytes: 50000000
ingestion_rate_limit_bytes: 45000000
ingestion_rate_strategy: global
max_bytes_per_trace: 75000000
max_traces_per_user: 100000
metrics_generator_forwarder_queue_size: 400
metrics_generator_forwarder_workers: 8
metrics_generator_max_active_series: 0
metrics_generator_processors:
- span-metrics
metrics_generator_ring_size: 0
per_tenant_override_config: /conf/overrides.yaml
querier:
frontend_worker:
frontend_address: tempo-query-frontend-discovery:9095
max_concurrent_queries: 100
search:
external_endpoints: []
external_hedge_requests_at: 8s
external_hedge_requests_up_to: 2
prefer_self: 10
query_timeout: 60s
trace_by_id:
query_timeout: 60s
query_frontend:
max_retries: 6
search:
concurrent_jobs: 2000
target_bytes_per_job: 209715200
tolerate_failed_blocks: 0
trace_by_id:
hedge_requests_at: 2s
hedge_requests_up_to: 2
query_shards: 80
server:
grpc_server_max_recv_msg_size: 4194304
grpc_server_max_send_msg_size: 4194304
http_listen_port: 3100
http_server_read_timeout: 30s
http_server_write_timeout: 30s
log_format: logfmt
log_level: info
storage:
trace:
backend: s3
block:
version: vParquet2
blocklist_poll: 5m
cache: memcached
local:
path: /var/tempo/traces
memcached:
consistent_hash: true
host: tempo-memcached
service: memcached-client
timeout: 500ms
s3:
access_key: tempoUser
bucket: tempo-traces-target
endpoint: ${TEMPO_MINIO_ENDPOINT:-NotUsed}
insecure: true
secret_key: ${TEMPO_USER_MINIO_PASSWORD:-NotUsed}
wal:
path: /var/tempo/wal
usage_report:
reporting_enabled: false
@sureshk300567 can you share metrics generator logs around the time range when you are missing metrics??
Hi @surajsidh - thanks for the response .
I couldnt find the options for attaching logs , hence pasted here .
Common labels: {"app_kubernetes_io_component":"metrics-generator","app_kubernetes_io_instance":"tempo-distributed","app_kubernetes_io_managed_by":"Helm","app_kubernetes_io_name":"tempo","app_kubernetes_io_part_of":"memberlist","app_kubernetes_io_version":"2.0.1","cluster":"YOUR_CLUSTER_NAME","container":"metrics-generator","helm_sh_chart":"tempo-distributed-1.2.3","namespace":"tempo-distributed"}
Line limit: "1000 reached, received logs cover 36.77% (1h 28min 15sec) of your selected time range (3h 59min 59sec)"
Total bytes processed: "186 kB"
2023-06-05T19:11:56+02:00 2023-06-05T20:11:56.319642494+03:00 stderr F level=info ts=2023-06-05T17:11:56.319421104Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:56+02:00 2023-06-05T20:11:56.309538837+03:00 stderr F level=info ts=2023-06-05T17:11:56.309341022Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:54+02:00 2023-06-05T20:11:54.354835512+03:00 stderr F level=info ts=2023-06-05T17:11:54.354582166Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98503
2023-06-05T19:11:54+02:00 2023-06-05T20:11:54.292895098+03:00 stderr F level=info ts=2023-06-05T17:11:54.292678461Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:11:46+02:00 2023-06-05T20:11:46.320016064+03:00 stderr F level=info ts=2023-06-05T17:11:46.31978587Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:46+02:00 2023-06-05T20:11:46.303191163+03:00 stderr F level=info ts=2023-06-05T17:11:46.302877188Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:44+02:00 2023-06-05T20:11:44.333286747+03:00 stderr F level=info ts=2023-06-05T17:11:44.332949031Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98503
2023-06-05T19:11:44+02:00 2023-06-05T20:11:44.324021428+03:00 stderr F level=info ts=2023-06-05T17:11:44.323749577Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:11:36+02:00 2023-06-05T20:11:36.31746113+03:00 stderr F level=info ts=2023-06-05T17:11:36.317080808Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:36+02:00 2023-06-05T20:11:36.300447467+03:00 stderr F level=info ts=2023-06-05T17:11:36.300144048Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:34+02:00 2023-06-05T20:11:34.321012497+03:00 stderr F level=info ts=2023-06-05T17:11:34.320779115Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:11:34+02:00 2023-06-05T20:11:34.293855318+03:00 stderr F level=info ts=2023-06-05T17:11:34.293590134Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:11:26+02:00 2023-06-05T20:11:26.319192279+03:00 stderr F level=info ts=2023-06-05T17:11:26.318977413Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:26+02:00 2023-06-05T20:11:26.297996878+03:00 stderr F level=info ts=2023-06-05T17:11:26.297747513Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:24+02:00 2023-06-05T20:11:24.375895155+03:00 stderr F level=info ts=2023-06-05T17:11:24.375605956Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:11:24+02:00 2023-06-05T20:11:24.298504186+03:00 stderr F level=info ts=2023-06-05T17:11:24.298269219Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:11:16+02:00 2023-06-05T20:11:16.304312177+03:00 stderr F level=info ts=2023-06-05T17:11:16.304004729Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:16+02:00 2023-06-05T20:11:16.320054595+03:00 stderr F level=info ts=2023-06-05T17:11:16.319879668Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:14+02:00 2023-06-05T20:11:14.59850607+03:00 stderr F level=info ts=2023-06-05T17:11:14.596353643Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:11:14+02:00 2023-06-05T20:11:14.307602385+03:00 stderr F level=info ts=2023-06-05T17:11:14.307372371Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:11:06+02:00 2023-06-05T20:11:06.296671938+03:00 stderr F level=info ts=2023-06-05T17:11:06.296337254Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:11:06+02:00 2023-06-05T20:11:06.332771265+03:00 stderr F level=info ts=2023-06-05T17:11:06.332584842Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:11:04+02:00 2023-06-05T20:11:04.310771875+03:00 stderr F level=info ts=2023-06-05T17:11:04.310583109Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:11:04+02:00 2023-06-05T20:11:04.308687463+03:00 stderr F level=info ts=2023-06-05T17:11:04.30849698Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:10:56+02:00 2023-06-05T20:10:56.305369403+03:00 stderr F level=info ts=2023-06-05T17:10:56.305174442Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:10:56+02:00 2023-06-05T20:10:56.314716849+03:00 stderr F level=info ts=2023-06-05T17:10:56.314465042Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:10:54+02:00 2023-06-05T20:10:54.315957722+03:00 stderr F level=info ts=2023-06-05T17:10:54.315791077Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:10:54+02:00 2023-06-05T20:10:54.367391539+03:00 stderr F level=info ts=2023-06-05T17:10:54.367128911Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:10:46+02:00 2023-06-05T20:10:46.307343174+03:00 stderr F level=info ts=2023-06-05T17:10:46.307059037Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:10:46+02:00 2023-06-05T20:10:46.322953281+03:00 stderr F level=info ts=2023-06-05T17:10:46.32274248Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99269
2023-06-05T19:10:44+02:00 2023-06-05T20:10:44.305558087+03:00 stderr F level=info ts=2023-06-05T17:10:44.305378428Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:10:44+02:00 2023-06-05T20:10:44.349152324+03:00 stderr F level=info ts=2023-06-05T17:10:44.348879547Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:10:36+02:00 2023-06-05T20:10:36.334305035+03:00 stderr F level=info ts=2023-06-05T17:10:36.334055567Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99269
2023-06-05T19:10:36+02:00 2023-06-05T20:10:36.304799014+03:00 stderr F level=info ts=2023-06-05T17:10:36.304500503Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98190
2023-06-05T19:10:34+02:00 2023-06-05T20:10:34.384476474+03:00 stderr F level=info ts=2023-06-05T17:10:34.384239007Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:10:34+02:00 2023-06-05T20:10:34.295574921+03:00 stderr F level=info ts=2023-06-05T17:10:34.295309377Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:10:26+02:00 2023-06-05T20:10:26.338129905+03:00 stderr F level=info ts=2023-06-05T17:10:26.337941369Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:10:26+02:00 2023-06-05T20:10:26.321596973+03:00 stderr F level=info ts=2023-06-05T17:10:26.321349028Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=99269
2023-06-05T19:10:26+02:00 2023-06-05T20:10:26.299130328+03:00 stderr F level=info ts=2023-06-05T17:10:26.298960567Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:10:26+02:00 2023-06-05T20:10:26.143216496+03:00 stderr F level=info ts=2023-06-05T17:10:26.143051502Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=98190
2023-06-05T19:10:24+02:00 2023-06-05T20:10:24.347325993+03:00 stderr F level=info ts=2023-06-05T17:10:24.346939271Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98503
2023-06-05T19:10:24+02:00 2023-06-05T20:10:24.3385552+03:00 stderr F level=info ts=2023-06-05T17:10:24.338179256Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=98503
2023-06-05T19:10:24+02:00 2023-06-05T20:10:24.314680459+03:00 stderr F level=info ts=2023-06-05T17:10:24.314494039Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97882
2023-06-05T19:10:24+02:00 2023-06-05T20:10:24.284721076+03:00 stderr F level=info ts=2023-06-05T17:10:24.284514553Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=97850
2023-06-05T19:10:16+02:00 2023-06-05T20:10:16.311221627+03:00 stderr F level=info ts=2023-06-05T17:10:16.310898948Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:10:16+02:00 2023-06-05T20:10:16.299454089+03:00 stderr F level=info ts=2023-06-05T17:10:16.299246543Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:10:14+02:00 2023-06-05T20:10:14.317572462+03:00 stderr F level=info ts=2023-06-05T17:10:14.317363694Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:10:14+02:00 2023-06-05T20:10:14.304689111+03:00 stderr F level=info ts=2023-06-05T17:10:14.30441272Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98503
2023-06-05T19:10:06+02:00 2023-06-05T20:10:06.318262802+03:00 stderr F level=info ts=2023-06-05T17:10:06.31799214Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:10:06+02:00 2023-06-05T20:10:06.305663802+03:00 stderr F level=info ts=2023-06-05T17:10:06.30545904Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:10:04+02:00 2023-06-05T20:10:04.340525369+03:00 stderr F level=info ts=2023-06-05T17:10:04.340176885Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98503
2023-06-05T19:10:04+02:00 2023-06-05T20:10:04.31012607+03:00 stderr F level=info ts=2023-06-05T17:10:04.309905842Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:56+02:00 2023-06-05T20:09:56.642529027+03:00 stderr F level=info ts=2023-06-05T17:09:56.642295853Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:56+02:00 2023-06-05T20:09:56.306156713+03:00 stderr F level=info ts=2023-06-05T17:09:56.305943144Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:54+02:00 2023-06-05T20:09:54.325849448+03:00 stderr F level=info ts=2023-06-05T17:09:54.325609969Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:09:54+02:00 2023-06-05T20:09:54.308773125+03:00 stderr F level=info ts=2023-06-05T17:09:54.308573244Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:46+02:00 2023-06-05T20:09:46.303858859+03:00 stderr F level=info ts=2023-06-05T17:09:46.303678977Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:46+02:00 2023-06-05T20:09:46.311528937+03:00 stderr F level=info ts=2023-06-05T17:09:46.31130016Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:44+02:00 2023-06-05T20:09:44.339318198+03:00 stderr F level=info ts=2023-06-05T17:09:44.339082626Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:44+02:00 2023-06-05T20:09:44.320201143+03:00 stderr F level=info ts=2023-06-05T17:09:44.319962329Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:09:36+02:00 2023-06-05T20:09:36.304735691+03:00 stderr F level=info ts=2023-06-05T17:09:36.304512091Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:36+02:00 2023-06-05T20:09:36.313337557+03:00 stderr F level=info ts=2023-06-05T17:09:36.313068383Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:34+02:00 2023-06-05T20:09:34.309365139+03:00 stderr F level=info ts=2023-06-05T17:09:34.309109719Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:34+02:00 2023-06-05T20:09:34.354056575+03:00 stderr F level=info ts=2023-06-05T17:09:34.353731932Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:09:26+02:00 2023-06-05T20:09:26.301738964+03:00 stderr F level=info ts=2023-06-05T17:09:26.301534204Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:26+02:00 2023-06-05T20:09:26.313373953+03:00 stderr F level=info ts=2023-06-05T17:09:26.313139156Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:24+02:00 2023-06-05T20:09:24.302060072+03:00 stderr F level=info ts=2023-06-05T17:09:24.300923505Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:24+02:00 2023-06-05T20:09:24.353427628+03:00 stderr F level=info ts=2023-06-05T17:09:24.352708902Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:09:16+02:00 2023-06-05T20:09:16.30267745+03:00 stderr F level=info ts=2023-06-05T17:09:16.302460627Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:16+02:00 2023-06-05T20:09:16.317358577+03:00 stderr F level=info ts=2023-06-05T17:09:16.317118545Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:14+02:00 2023-06-05T20:09:14.699493433+03:00 stderr F level=info ts=2023-06-05T17:09:14.699245419Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:09:14+02:00 2023-06-05T20:09:14.289900631+03:00 stderr F level=info ts=2023-06-05T17:09:14.289720276Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:06+02:00 2023-06-05T20:09:06.300789361+03:00 stderr F level=info ts=2023-06-05T17:09:06.300673373Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:09:06+02:00 2023-06-05T20:09:06.308575645+03:00 stderr F level=info ts=2023-06-05T17:09:06.308323213Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:09:04+02:00 2023-06-05T20:09:04.292588408+03:00 stderr F level=info ts=2023-06-05T17:09:04.292363622Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:09:04+02:00 2023-06-05T20:09:04.321508614+03:00 stderr F level=info ts=2023-06-05T17:09:04.321244168Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:08:56+02:00 2023-06-05T20:08:56.304298255+03:00 stderr F level=info ts=2023-06-05T17:08:56.304112898Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:56+02:00 2023-06-05T20:08:56.301533625+03:00 stderr F level=info ts=2023-06-05T17:08:56.301315445Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:54+02:00 2023-06-05T20:08:54.30035436+03:00 stderr F level=info ts=2023-06-05T17:08:54.300123946Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97884
2023-06-05T19:08:54+02:00 2023-06-05T20:08:54.32788277+03:00 stderr F level=info ts=2023-06-05T17:08:54.327554683Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:08:46+02:00 2023-06-05T20:08:46.311964118+03:00 stderr F level=info ts=2023-06-05T17:08:46.311773274Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:46+02:00 2023-06-05T20:08:46.317270187+03:00 stderr F level=info ts=2023-06-05T17:08:46.317040198Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:44+02:00 2023-06-05T20:08:44.336736241+03:00 stderr F level=info ts=2023-06-05T17:08:44.336515119Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:08:44+02:00 2023-06-05T20:08:44.315284754+03:00 stderr F level=info ts=2023-06-05T17:08:44.315001725Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:08:36+02:00 2023-06-05T20:08:36.296763641+03:00 stderr F level=info ts=2023-06-05T17:08:36.296531931Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:36+02:00 2023-06-05T20:08:36.303608922+03:00 stderr F level=info ts=2023-06-05T17:08:36.303451872Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:34+02:00 2023-06-05T20:08:34.350893215+03:00 stderr F level=info ts=2023-06-05T17:08:34.350714032Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98486
2023-06-05T19:08:34+02:00 2023-06-05T20:08:34.294568212+03:00 stderr F level=info ts=2023-06-05T17:08:34.294358856Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:08:26+02:00 2023-06-05T20:08:26.306734561+03:00 stderr F level=info ts=2023-06-05T17:08:26.306462165Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:26+02:00 2023-06-05T20:08:26.318473424+03:00 stderr F level=info ts=2023-06-05T17:08:26.318232559Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:24+02:00 2023-06-05T20:08:24.324416182+03:00 stderr F level=info ts=2023-06-05T17:08:24.324264187Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:08:24+02:00 2023-06-05T20:08:24.30312002+03:00 stderr F level=info ts=2023-06-05T17:08:24.302904976Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:08:16+02:00 2023-06-05T20:08:16.30238294+03:00 stderr F level=info ts=2023-06-05T17:08:16.302142429Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:16+02:00 2023-06-05T20:08:16.306736385+03:00 stderr F level=info ts=2023-06-05T17:08:16.306485709Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:14+02:00 2023-06-05T20:08:14.341515829+03:00 stderr F level=info ts=2023-06-05T17:08:14.341310506Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:08:14+02:00 2023-06-05T20:08:14.325825221+03:00 stderr F level=info ts=2023-06-05T17:08:14.325545944Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:08:06+02:00 2023-06-05T20:08:06.362658698+03:00 stderr F level=info ts=2023-06-05T17:08:06.362442348Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:08:06+02:00 2023-06-05T20:08:06.323832544+03:00 stderr F level=info ts=2023-06-05T17:08:06.323574113Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:08:04+02:00 2023-06-05T20:08:04.304658855+03:00 stderr F level=info ts=2023-06-05T17:08:04.304447944Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:08:04+02:00 2023-06-05T20:08:04.408183929+03:00 stderr F level=info ts=2023-06-05T17:08:04.407968979Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:56+02:00 2023-06-05T20:07:56.319592672+03:00 stderr F level=info ts=2023-06-05T17:07:56.319319335Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:56+02:00 2023-06-05T20:07:56.299788244+03:00 stderr F level=info ts=2023-06-05T17:07:56.299459258Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:54+02:00 2023-06-05T20:07:54.307825257+03:00 stderr F level=info ts=2023-06-05T17:07:54.30762527Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:07:54+02:00 2023-06-05T20:07:54.338603915+03:00 stderr F level=info ts=2023-06-05T17:07:54.33838275Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:46+02:00 2023-06-05T20:07:46.317658625+03:00 stderr F level=info ts=2023-06-05T17:07:46.317397853Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:46+02:00 2023-06-05T20:07:46.311100625+03:00 stderr F level=info ts=2023-06-05T17:07:46.310894096Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:44+02:00 2023-06-05T20:07:44.288356024+03:00 stderr F level=info ts=2023-06-05T17:07:44.288180844Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:07:44+02:00 2023-06-05T20:07:44.316892453+03:00 stderr F level=info ts=2023-06-05T17:07:44.316664347Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:36+02:00 2023-06-05T20:07:36.313290515+03:00 stderr F level=info ts=2023-06-05T17:07:36.31306722Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:36+02:00 2023-06-05T20:07:36.313930853+03:00 stderr F level=info ts=2023-06-05T17:07:36.313725114Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:34+02:00 2023-06-05T20:07:34.299135667+03:00 stderr F level=info ts=2023-06-05T17:07:34.298893479Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:07:34+02:00 2023-06-05T20:07:34.321717251+03:00 stderr F level=info ts=2023-06-05T17:07:34.321487065Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:26+02:00 2023-06-05T20:07:26.32048791+03:00 stderr F level=info ts=2023-06-05T17:07:26.320247777Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:26+02:00 2023-06-05T20:07:26.312710265+03:00 stderr F level=info ts=2023-06-05T17:07:26.312489268Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:24+02:00 2023-06-05T20:07:24.296874019+03:00 stderr F level=info ts=2023-06-05T17:07:24.296672708Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:07:24+02:00 2023-06-05T20:07:24.321620165+03:00 stderr F level=info ts=2023-06-05T17:07:24.321355725Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:16+02:00 2023-06-05T20:07:16.606079957+03:00 stderr F level=info ts=2023-06-05T17:07:16.605877758Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:16+02:00 2023-06-05T20:07:16.308905634+03:00 stderr F level=info ts=2023-06-05T17:07:16.308704704Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:14+02:00 2023-06-05T20:07:14.709059629+03:00 stderr F level=info ts=2023-06-05T17:07:14.708757467Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:14+02:00 2023-06-05T20:07:14.305340966+03:00 stderr F level=info ts=2023-06-05T17:07:14.305124869Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:07:06+02:00 2023-06-05T20:07:06.323486738+03:00 stderr F level=info ts=2023-06-05T17:07:06.323244074Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:07:06+02:00 2023-06-05T20:07:06.317086689+03:00 stderr F level=info ts=2023-06-05T17:07:06.316866871Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:07:04+02:00 2023-06-05T20:07:04.410566632+03:00 stderr F level=info ts=2023-06-05T17:07:04.410281958Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:07:04+02:00 2023-06-05T20:07:04.303292475+03:00 stderr F level=info ts=2023-06-05T17:07:04.303023911Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:06:56+02:00 2023-06-05T20:06:56.313141291+03:00 stderr F level=info ts=2023-06-05T17:06:56.312889319Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:56+02:00 2023-06-05T20:06:56.30367408+03:00 stderr F level=info ts=2023-06-05T17:06:56.303420071Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:54+02:00 2023-06-05T20:06:54.340474949+03:00 stderr F level=info ts=2023-06-05T17:06:54.340234686Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:06:54+02:00 2023-06-05T20:06:54.305741697+03:00 stderr F level=info ts=2023-06-05T17:06:54.305536354Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:06:46+02:00 2023-06-05T20:06:46.617252194+03:00 stderr F level=info ts=2023-06-05T17:06:46.617020959Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:46+02:00 2023-06-05T20:06:46.3066729+03:00 stderr F level=info ts=2023-06-05T17:06:46.306433332Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:44+02:00 2023-06-05T20:06:44.351338363+03:00 stderr F level=info ts=2023-06-05T17:06:44.351100659Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:06:44+02:00 2023-06-05T20:06:44.3166352+03:00 stderr F level=info ts=2023-06-05T17:06:44.314270146Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:06:36+02:00 2023-06-05T20:06:36.319994987+03:00 stderr F level=info ts=2023-06-05T17:06:36.31965749Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:36+02:00 2023-06-05T20:06:36.302846899+03:00 stderr F level=info ts=2023-06-05T17:06:36.302665675Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:34+02:00 2023-06-05T20:06:34.319358003+03:00 stderr F level=info ts=2023-06-05T17:06:34.319150126Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97867
2023-06-05T19:06:34+02:00 2023-06-05T20:06:34.327010211+03:00 stderr F level=info ts=2023-06-05T17:06:34.326635224Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:06:26+02:00 2023-06-05T20:06:26.317099136+03:00 stderr F level=info ts=2023-06-05T17:06:26.316831921Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:26+02:00 2023-06-05T20:06:26.305875455+03:00 stderr F level=info ts=2023-06-05T17:06:26.305661509Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:24+02:00 2023-06-05T20:06:24.315128878+03:00 stderr F level=info ts=2023-06-05T17:06:24.314869415Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:06:24+02:00 2023-06-05T20:06:24.31696265+03:00 stderr F level=info ts=2023-06-05T17:06:24.316696634Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:06:16+02:00 2023-06-05T20:06:16.32395504+03:00 stderr F level=info ts=2023-06-05T17:06:16.323699043Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:16+02:00 2023-06-05T20:06:16.304163773+03:00 stderr F level=info ts=2023-06-05T17:06:16.303893153Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:14+02:00 2023-06-05T20:06:14.313346598+03:00 stderr F level=info ts=2023-06-05T17:06:14.313110519Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:06:14+02:00 2023-06-05T20:06:14.380894048+03:00 stderr F level=info ts=2023-06-05T17:06:14.380645673Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:06:06+02:00 2023-06-05T20:06:06.306516332+03:00 stderr F level=info ts=2023-06-05T17:06:06.306294437Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:06:06+02:00 2023-06-05T20:06:06.315268194+03:00 stderr F level=info ts=2023-06-05T17:06:06.315023664Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:06:04+02:00 2023-06-05T20:06:04.336370267+03:00 stderr F level=info ts=2023-06-05T17:06:04.336132219Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:06:04+02:00 2023-06-05T20:06:04.334012129+03:00 stderr F level=info ts=2023-06-05T17:06:04.333788712Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:05:56+02:00 2023-06-05T20:05:56.302842643+03:00 stderr F level=info ts=2023-06-05T17:05:56.302540611Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:05:56+02:00 2023-06-05T20:05:56.306895477+03:00 stderr F level=info ts=2023-06-05T17:05:56.306711957Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:05:54+02:00 2023-06-05T20:05:54.306695624+03:00 stderr F level=info ts=2023-06-05T17:05:54.306491308Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:05:54+02:00 2023-06-05T20:05:54.315257951+03:00 stderr F level=info ts=2023-06-05T17:05:54.315011356Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:05:46+02:00 2023-06-05T20:05:46.307712374+03:00 stderr F level=info ts=2023-06-05T17:05:46.307451817Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:05:46+02:00 2023-06-05T20:05:46.29720669+03:00 stderr F level=info ts=2023-06-05T17:05:46.296825359Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:05:44+02:00 2023-06-05T20:05:44.308508696+03:00 stderr F level=info ts=2023-06-05T17:05:44.308293107Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:05:44+02:00 2023-06-05T20:05:44.336457848+03:00 stderr F level=info ts=2023-06-05T17:05:44.336254432Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:05:36+02:00 2023-06-05T20:05:36.308504092+03:00 stderr F level=info ts=2023-06-05T17:05:36.308245492Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99286
2023-06-05T19:05:36+02:00 2023-06-05T20:05:36.299968547+03:00 stderr F level=info ts=2023-06-05T17:05:36.299701657Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:05:34+02:00 2023-06-05T20:05:34.351562921+03:00 stderr F level=info ts=2023-06-05T17:05:34.351185619Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98469
2023-06-05T19:05:34+02:00 2023-06-05T20:05:34.306779603+03:00 stderr F level=info ts=2023-06-05T17:05:34.306530236Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:05:26+02:00 2023-06-05T20:05:26.314495584+03:00 stderr F level=info ts=2023-06-05T17:05:26.314262384Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99334
2023-06-05T19:05:26+02:00 2023-06-05T20:05:26.271764828+03:00 stderr F level=info ts=2023-06-05T17:05:26.271503424Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=99286
2023-06-05T19:05:26+02:00 2023-06-05T20:05:26.307075955+03:00 stderr F level=info ts=2023-06-05T17:05:26.306780587Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98191
2023-06-05T19:05:26+02:00 2023-06-05T20:05:26.108511311+03:00 stderr F level=info ts=2023-06-05T17:05:26.10803129Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=98191
2023-06-05T19:05:24+02:00 2023-06-05T20:05:24.394038134+03:00 stderr F level=info ts=2023-06-05T17:05:24.393741597Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:05:24+02:00 2023-06-05T20:05:24.366195973+03:00 stderr F level=info ts=2023-06-05T17:05:24.365950164Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=98469
2023-06-05T19:05:24+02:00 2023-06-05T20:05:24.308391725+03:00 stderr F level=info ts=2023-06-05T17:05:24.308181542Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97850
2023-06-05T19:05:24+02:00 2023-06-05T20:05:24.096322126+03:00 stderr F level=info ts=2023-06-05T17:05:24.096101362Z caller=registry.go:243 tenant=single-tenant msg="deleted stale series" active_series=97850
2023-06-05T19:05:16+02:00 2023-06-05T20:05:16.673034293+03:00 stderr F level=info ts=2023-06-05T17:05:16.6727676Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:05:16+02:00 2023-06-05T20:05:16.321777094+03:00 stderr F level=info ts=2023-06-05T17:05:16.32154299Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:05:14+02:00 2023-06-05T20:05:14.348923107+03:00 stderr F level=info ts=2023-06-05T17:05:14.348648482Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:05:14+02:00 2023-06-05T20:05:14.306047491+03:00 stderr F level=info ts=2023-06-05T17:05:14.30572364Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:05:06+02:00 2023-06-05T20:05:06.345893579+03:00 stderr F level=info ts=2023-06-05T17:05:06.345659788Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:05:06+02:00 2023-06-05T20:05:06.314578909+03:00 stderr F level=info ts=2023-06-05T17:05:06.314351043Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:05:04+02:00 2023-06-05T20:05:04.311328421+03:00 stderr F level=info ts=2023-06-05T17:05:04.311048491Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:05:04+02:00 2023-06-05T20:05:04.29536305+03:00 stderr F level=info ts=2023-06-05T17:05:04.295172492Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:56+02:00 2023-06-05T20:04:56.31362599+03:00 stderr F level=info ts=2023-06-05T17:04:56.313394373Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:56+02:00 2023-06-05T20:04:56.306527463+03:00 stderr F level=info ts=2023-06-05T17:04:56.306276636Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:54+02:00 2023-06-05T20:04:54.31195599+03:00 stderr F level=info ts=2023-06-05T17:04:54.311749906Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:54+02:00 2023-06-05T20:04:54.33360892+03:00 stderr F level=info ts=2023-06-05T17:04:54.3333059Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:46+02:00 2023-06-05T20:04:46.309214464+03:00 stderr F level=info ts=2023-06-05T17:04:46.309021772Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:46+02:00 2023-06-05T20:04:46.350140193+03:00 stderr F level=info ts=2023-06-05T17:04:46.34972344Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:44+02:00 2023-06-05T20:04:44.313245337+03:00 stderr F level=info ts=2023-06-05T17:04:44.313062558Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:44+02:00 2023-06-05T20:04:44.398900485+03:00 stderr F level=info ts=2023-06-05T17:04:44.398653186Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:36+02:00 2023-06-05T20:04:36.311581616+03:00 stderr F level=info ts=2023-06-05T17:04:36.311367827Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:36+02:00 2023-06-05T20:04:36.31476974+03:00 stderr F level=info ts=2023-06-05T17:04:36.314434215Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:34+02:00 2023-06-05T20:04:34.549983111+03:00 stderr F level=info ts=2023-06-05T17:04:34.549723318Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:34+02:00 2023-06-05T20:04:34.340852628+03:00 stderr F level=info ts=2023-06-05T17:04:34.340560068Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:26+02:00 2023-06-05T20:04:26.30144595+03:00 stderr F level=info ts=2023-06-05T17:04:26.301211307Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:26+02:00 2023-06-05T20:04:26.325316384+03:00 stderr F level=info ts=2023-06-05T17:04:26.325064402Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:24+02:00 2023-06-05T20:04:24.362373334+03:00 stderr F level=info ts=2023-06-05T17:04:24.362102033Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:24+02:00 2023-06-05T20:04:24.393381415+03:00 stderr F level=info ts=2023-06-05T17:04:24.393177789Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:16+02:00 2023-06-05T20:04:16.323346027+03:00 stderr F level=info ts=2023-06-05T17:04:16.323021399Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:16+02:00 2023-06-05T20:04:16.301528048+03:00 stderr F level=info ts=2023-06-05T17:04:16.301359706Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:14+02:00 2023-06-05T20:04:14.317272507+03:00 stderr F level=info ts=2023-06-05T17:04:14.316931239Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:04:14+02:00 2023-06-05T20:04:14.322261706+03:00 stderr F level=info ts=2023-06-05T17:04:14.322108236Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:06+02:00 2023-06-05T20:04:06.318990437+03:00 stderr F level=info ts=2023-06-05T17:04:06.318713126Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:04:06+02:00 2023-06-05T20:04:06.304899458+03:00 stderr F level=info ts=2023-06-05T17:04:06.304676843Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:04:04+02:00 2023-06-05T20:04:04.371002823+03:00 stderr F level=info ts=2023-06-05T17:04:04.370818913Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:04:04+02:00 2023-06-05T20:04:04.326291445+03:00 stderr F level=info ts=2023-06-05T17:04:04.325806384Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:56+02:00 2023-06-05T20:03:56.322133953+03:00 stderr F level=info ts=2023-06-05T17:03:56.321925059Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:03:56+02:00 2023-06-05T20:03:56.296147561+03:00 stderr F level=info ts=2023-06-05T17:03:56.295922094Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:54+02:00 2023-06-05T20:03:54.374853984+03:00 stderr F level=info ts=2023-06-05T17:03:54.374606813Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:03:54+02:00 2023-06-05T20:03:54.311813979+03:00 stderr F level=info ts=2023-06-05T17:03:54.310685186Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:46+02:00 2023-06-05T20:03:46.363709008+03:00 stderr F level=info ts=2023-06-05T17:03:46.363387971Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:03:46+02:00 2023-06-05T20:03:46.306587945+03:00 stderr F level=info ts=2023-06-05T17:03:46.306401163Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:44+02:00 2023-06-05T20:03:44.405911145+03:00 stderr F level=info ts=2023-06-05T17:03:44.405601924Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:03:44+02:00 2023-06-05T20:03:44.313867777+03:00 stderr F level=info ts=2023-06-05T17:03:44.313674843Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:36+02:00 2023-06-05T20:03:36.308476877+03:00 stderr F level=info ts=2023-06-05T17:03:36.308240503Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:36+02:00 2023-06-05T20:03:36.327577487+03:00 stderr F level=info ts=2023-06-05T17:03:36.327392187Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:03:34+02:00 2023-06-05T20:03:34.356079024+03:00 stderr F level=info ts=2023-06-05T17:03:34.355727566Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:03:34+02:00 2023-06-05T20:03:34.323660249+03:00 stderr F level=info ts=2023-06-05T17:03:34.323406526Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:26+02:00 2023-06-05T20:03:26.61778001+03:00 stderr F level=info ts=2023-06-05T17:03:26.617563215Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:26+02:00 2023-06-05T20:03:26.323663894+03:00 stderr F level=info ts=2023-06-05T17:03:26.323141088Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99337
2023-06-05T19:03:24+02:00 2023-06-05T20:03:24.313885643+03:00 stderr F level=info ts=2023-06-05T17:03:24.313672078Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:24+02:00 2023-06-05T20:03:24.32382893+03:00 stderr F level=info ts=2023-06-05T17:03:24.323540986Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:03:16+02:00 2023-06-05T20:03:16.319268768+03:00 stderr F level=info ts=2023-06-05T17:03:16.319039437Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:16+02:00 2023-06-05T20:03:16.331512792+03:00 stderr F level=info ts=2023-06-05T17:03:16.331226179Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99285
2023-06-05T19:03:14+02:00 2023-06-05T20:03:14.321103392+03:00 stderr F level=info ts=2023-06-05T17:03:14.320886013Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:14+02:00 2023-06-05T20:03:14.382751163+03:00 stderr F level=info ts=2023-06-05T17:03:14.382489484Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:03:06+02:00 2023-06-05T20:03:06.770587916+03:00 stderr F level=info ts=2023-06-05T17:03:06.770339459Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99268
2023-06-05T19:03:06+02:00 2023-06-05T20:03:06.310013989+03:00 stderr F level=info ts=2023-06-05T17:03:06.309799837Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:03:04+02:00 2023-06-05T20:03:04.325081312+03:00 stderr F level=info ts=2023-06-05T17:03:04.324754781Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:03:04+02:00 2023-06-05T20:03:04.35966507+03:00 stderr F level=info ts=2023-06-05T17:03:04.359363322Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:02:56+02:00 2023-06-05T20:02:56.318617672+03:00 stderr F level=info ts=2023-06-05T17:02:56.318336377Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99268
2023-06-05T19:02:56+02:00 2023-06-05T20:02:56.297474719+03:00 stderr F level=info ts=2023-06-05T17:02:56.297276205Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:02:54+02:00 2023-06-05T20:02:54.307154626+03:00 stderr F level=info ts=2023-06-05T17:02:54.306970672Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=97901
2023-06-05T19:02:54+02:00 2023-06-05T20:02:54.322860557+03:00 stderr F level=info ts=2023-06-05T17:02:54.32250997Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98520
2023-06-05T19:02:46+02:00 2023-06-05T20:02:46.318034921+03:00 stderr F level=info ts=2023-06-05T17:02:46.317765569Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=99268
2023-06-05T19:02:46+02:00 2023-06-05T20:02:46.307942508+03:00 stderr F level=info ts=2023-06-05T17:02:46.307732549Z caller=registry.go:222 tenant=single-tenant msg="collecting metrics" active_series=98208
2023-06-05T19:02:44+02:00 2023-06-05T20:02:44.401073637+03:00 stderr F level=info ts=2023-06-05T17:02:44.400816888Z caller=registry.go:222
logs are looking okay to me, I see bunch of deleted stale series which can happen when series go stale, and is normal.
It might be that you don’t have enough traces coming in to keep the series active, and you se missing data when series goes stale, read Staleness and PromQL – Robust Perception | Prometheus Monitoring Experts for more on staleness.
you can trying pushing more data or try running it with only namespace and environment dimensions and see if that works 
We get the same impressions too post looking at the logs.
However applications have been sending metrics continuously.
Today I tried sending span metrics generated from one of our trace gateway collector (otel) and send to mimir .
When we compared the two graphs( one we are getting metrics from metrics generator and the other coming from otel) we see a huge difference.
We don’t see any gaps when it comes to span metrics coming from otel .
I will try attaching screen shot tomorrow .
Thanks
K Suresh
spanmetrics are generated from the traces sent, and they are generated with the dimensions and intrinsic_dimensions as labels to the metrics generated. are you using same dimensions and intrinsic_dimensions in your gateway collector?
another thing that can happen is the samping? is you gateway collector generating these metrics before or after sampling? if it’s before sampling then the metrics at collector level are not sampled and will be differnt.
If issue is the metrics going stale, reduing stale_duration should mean that your metrics will have bigger gaps (they will go stale faster), and bumping up stale_duration will make the gaps smaller. try playing around with stale_duration, and see if it helps.
We are using the same dimensions , but not the histogram buckets, i have now made sure we have same values under histogram buckets .I dont see any gaps now .
However for strange reasons , the graph is lagging behind by 30 mins .
I am not sure how this can happen. I would probaly check scape config and remote write configs for the promethous instance, otel collector and tempo metrics generator. also look out for clock drift
