When sending metrics to Prometheus, it generates the metrics:
traces_spanmetrics_latency_sum, traces_spanmetrics_latency_count, traces_spanmetrics_latency_bucket.
However, when I run the following queries:
sum by (le,host_name)(traces_spanmetrics_latency_count{service="vb_sme_payment_platform_online", span_name="com.livi.sme.payment.api.PlatformService/paymentTran"})
sum by (le,host_name)(traces_spanmetrics_latency_sum{service="vb_sme_payment_platform_online", span_name="com.livi.sme.payment.api.PlatformService/paymentTran"})
While I understand that rate() or increase() should be used to view increments, I’m only asking why these metrics are not monotonically increasing.
I find their values do not keep increasing; they go up and down and even drop out entirely. Prometheus has not restarted, and no errors appear in the logs.
Aren’t these supposed to be counter types? Or have I misunderstood something?
Ttempo, version 2.8.1, configuration is as follows：
metrics_generator:
ring:
kvstore:
store: memberlist
instance_addr: ${HOSTNAME}
instance_port: 9095
processor:
service_graphs:
dimensions: [server.address, client.address, network.peer.address, ip.address, protocol, host.name]
enable_client_server_prefix: true
enable_messaging_system_latency_histogram: true
peer_attributes: []
enable_virtual_node_label: true
histogram_buckets: [1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 6, 7, 8, 9, 10, 30]
span_metrics:
dimensions: [server.address, client.address, network.peer.address, ip.address, protocol, host.name]
intrinsic_dimensions:
status_message: true
enable_target_info: true
histogram_buckets: [1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 6, 7, 8, 9, 10, 30]
local_blocks:
block:
version: vParquet3
filter_server_spans: true
flush_to_storage: true
storage:
path: /export/tempo_data/metrics_storage
remote_write:
- url: http://prome.dev.local/api/v1/write
send_exemplars: true
traces_storage:
path: /export/tempo_data/metrics_traces_storage
version: vParquet3
