-
What Grafana version and what operating system are you using?
- alloy, version v1.10.2 (branch: HEAD, revision: b19f8c5); linux
-
What are you trying to achieve?
- trying to get request per second rate graph.
-
How are you trying to achieve it?
- We are sending counter metrics from our application. We are a grafana_alloy collector running in ECS fargate centrally for all our services and application
-
What happened?
- i see it does not go down. It keeps monotonically increasing for most part. It gets reset after deployment.
-
What did you expect to happen?
- It should follow the rps pattern based on the traffic.
-
Can you copy/paste the configuration(s) that you are having problems with?
-
logging { level = "info" format = "logfmt" } otelcol.receiver.otlp "default" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/ // configures the default grpc endpoint "0.0.0.0:4317" grpc { } // configures the default http/protobuf endpoint "0.0.0.0:4318" http { } output { metrics = [otelcol.processor.resourcedetection.default.input] logs = [otelcol.processor.resourcedetection.default.input] traces = [otelcol.processor.resourcedetection.default.input] } } otelcol.processor.resourcedetection "default" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.resourcedetection/ detectors = ["env", "system"] system { hostname_sources = ["os"] } output { metrics = [otelcol.processor.transform.drop_unneeded_resource_attributes.input] logs = [otelcol.processor.transform.drop_unneeded_resource_attributes.input] traces = [otelcol.processor.transform.drop_unneeded_resource_attributes.input] } } otelcol.processor.transform "drop_unneeded_resource_attributes" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/ error_mode = "ignore" trace_statements { context = "resource" statements = [ "delete_key(attributes, \"k8s.pod.start_time\")", "delete_key(attributes, \"os.description\")", "delete_key(attributes, \"os.type\")", "delete_key(attributes, \"process.command_args\")", "delete_key(attributes, \"process.executable.path\")", "delete_key(attributes, \"process.pid\")", "delete_key(attributes, \"process.runtime.description\")", "delete_key(attributes, \"process.runtime.name\")", "delete_key(attributes, \"process.runtime.version\")", ] } metric_statements { context = "resource" statements = [ "delete_key(attributes, \"k8s.pod.start_time\")", "delete_key(attributes, \"os.description\")", "delete_key(attributes, \"os.type\")", "delete_key(attributes, \"process.command_args\")", "delete_key(attributes, \"process.executable.path\")", "delete_key(attributes, \"process.pid\")", "delete_key(attributes, \"process.runtime.description\")", "delete_key(attributes, \"process.runtime.name\")", "delete_key(attributes, \"process.runtime.version\")", ] } log_statements { context = "resource" statements = [ "delete_key(attributes, \"k8s.pod.start_time\")", "delete_key(attributes, \"os.description\")", "delete_key(attributes, \"os.type\")", "delete_key(attributes, \"process.command_args\")", "delete_key(attributes, \"process.executable.path\")", "delete_key(attributes, \"process.pid\")", "delete_key(attributes, \"process.runtime.description\")", "delete_key(attributes, \"process.runtime.name\")", "delete_key(attributes, \"process.runtime.version\")", ] } output { metrics = [otelcol.processor.transform.add_resource_attributes_as_metric_attributes.input] logs = [otelcol.processor.batch.default.input] traces = [ otelcol.processor.batch.default.input, otelcol.connector.host_info.default.input, ] } } otelcol.connector.host_info "default" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.host_info/ host_identifiers = ["host.name"] output { metrics = [otelcol.processor.batch.default.input] } } otelcol.processor.transform "add_resource_attributes_as_metric_attributes" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/ error_mode = "ignore" metric_statements { context = "datapoint" statements = [ "set(attributes[\"deployment.environment\"], resource.attributes[\"deployment.environment\"])", "set(attributes[\"service.version\"], resource.attributes[\"service.version\"])", ] } output { metrics = [otelcol.processor.batch.default.input] } } otelcol.processor.batch "default" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/ output { metrics = [otelcol.exporter.otlphttp.grafana_cloud.input] logs = [otelcol.exporter.otlphttp.grafana_cloud.input] traces = [otelcol.exporter.otlphttp.grafana_cloud.input] } } otelcol.exporter.otlphttp "grafana_cloud" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/ client { endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT") auth = otelcol.auth.basic.grafana_cloud.handler } } otelcol.auth.basic "grafana_cloud" { // https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/ username = env("GRAFANA_CLOUD_INSTANCE_ID") password = env("GRAFANA_CLOUD_API_KEY") }
-
-
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
- none
-
Did you follow any online instructions? If so, what is the URL?
- I have followed the instructions on how to plot rate chart.
Correct, that’s “counter” metric type. Request per second is usually calculated with rate function from counter metric type (if you are using Prometheus like metric storage). E. g.
rate(http_requests_total[5m])
That’s on the query level - e. g. panel query in the Grafana.
See basic doc about metric types:
Correct, that’s “counter” metric type. Request per second is usually calculated with rate function from counter metric type (if you are using Prometheus like metric storage)
I understand that the counter metrics keep going up. I am referring to the “rate” graph. The rate graph doesn’t go down. Here are some screenshots. There are occasional dips and they align with the deployments.
no, that’s deployments. I am mainly referring to the monotonous upward trend from 200c/s to 800c/s. I see that trend on pretty much all the counter metrics and I dont expect it.
Are there documents that i can refer to for debugging further?
I would show:
- raw data
- rate() data
- longer time range
…
How can you prove that those rate numbers are not correct?
here is the raw data: raw data - Pastebin.com
here is the rate() data rate data points - Pastebin.com
i know it is wrong because i am generating the traffic for testing and the graph does not match what i am generating.
And what’s the query?
i think for most part the rate being calculated is correct except when no new data point is being emitted for the metrics. In that case, it just stays flat, as i am seeing in this. There are multiple dips, and the request rate is correct. However, after i stopped generating any new traffic, the graph just stayed on the top. I expect that to go down.
this is for rate raw data :
rate(http_requests_total{ deployment_environment="dev",service_name="****", method="GET",status_code="200",path=”******"}[$__rate_interval])
this is for raw data
http_requests_total{ deployment_environment="dev",service_name="****", method="GET",status_code="200",path=”******"}
You are pasting data/screenshots without any context. Be always specific. WHAT stays FLAT? Counter, rate, …?
from further investigation, I was able to narrow this down to the fact that the rate graph is working fine when there is consistently new data points for the underlying metrics. But when there is no new data points, the rate function keeps using the last know data point forever.
Ideally it should forget that last data point after sometime. no?