Alloy lose some metrics

Grafana Alloy
1

When i push metrics from .NET application in otelcol.receiver.otlp some metrics disappear. When same application push metrics into OpenTelemetry Collector i see this metrics:
ec_Npgsql_busy_connections
ec_Npgsql_bytes_read_per_second
ec_Npgsql_bytes_read_per_second_total
ec_Npgsql_bytes_written_per_second
ec_Npgsql_bytes_written_per_second_total
ec_Npgsql_commands_per_second
ec_Npgsql_commands_per_second_total
ec_Npgsql_connection_pools
ec_Npgsql_current_commands
ec_Npgsql_failed_commands
ec_Npgsql_idle_connections
ec_Npgsql_multiplexing_average_commands_per_batch
ec_Npgsql_multiplexing_average_waits_per_batch
ec_Npgsql_multiplexing_average_write_time_per_batch
ec_Npgsql_prepared_commands_ratio
ec_Npgsql_total_command
http_client_duration_bucket
http_client_duration_count
http_client_duration_milliseconds_bucket
http_client_duration_milliseconds_count
http_client_duration_milliseconds_sum
http_client_duration_sum
http_client_request_duration_bucket
http_client_request_duration_count
http_client_request_duration_seconds_bucket
http_client_request_duration_seconds_count
http_client_request_duration_seconds_sum
http_client_request_duration_sum
http_client_response_size_bucket
http_client_response_size_count
http_client_response_size_sum
http_request_duration_microseconds
http_request_duration_microseconds_count
http_request_duration_microseconds_sum
http_request_size_bytes
http_request_size_bytes_count
http_request_size_bytes_sum
http_requests_bytes_total
http_requests_count_total
http_requests_milliseconds_total
http_requests_total
http_response_size_bytes
http_response_size_bytes_count
http_response_size_bytes_sum
http_server_active_requests
http_server_duration_bucket
http_server_duration_count
http_server_duration_milliseconds_bucket
http_server_duration_milliseconds_count
http_server_duration_milliseconds_sum
http_server_duration_sum
http_server_request_duration_bucket
http_server_request_duration_count
http_server_request_duration_seconds_bucket
http_server_request_duration_seconds_count
http_server_request_duration_seconds_sum
http_server_request_duration_sum
http_server_request_size_bucket
http_server_request_size_bytes_bucket
http_server_request_size_bytes_count
http_server_request_size_bytes_sum
http_server_request_size_count
http_server_request_size_sum
http_server_response_size_bucket
http_server_response_size_bytes_bucket
http_server_response_size_bytes_count
http_server_response_size_bytes_sum
http_server_response_size_count
http_server_response_size_sum
process_runtime_dotnet_assemblies_count
process_runtime_dotnet_exceptions_count
process_runtime_dotnet_exceptions_count_total
process_runtime_dotnet_gc_allocations_size
process_runtime_dotnet_gc_allocations_size_bytes_total
process_runtime_dotnet_gc_collections_count
process_runtime_dotnet_gc_collections_count_total
process_runtime_dotnet_gc_committed_memory_size
process_runtime_dotnet_gc_committed_memory_size_bytes
process_runtime_dotnet_gc_heap_size
process_runtime_dotnet_gc_heap_size_bytes
process_runtime_dotnet_gc_objects_size
process_runtime_dotnet_gc_objects_size_bytes
process_runtime_dotnet_jit_compilation_time
process_runtime_dotnet_jit_compilation_time_nanoseconds_total
process_runtime_dotnet_jit_il_compiled_size
process_runtime_dotnet_jit_il_compiled_size_bytes_total
process_runtime_dotnet_jit_methods_compiled_count
process_runtime_dotnet_jit_methods_compiled_count_total
process_runtime_dotnet_monitor_lock_contention_count
process_runtime_dotnet_monitor_lock_contention_count_total
process_runtime_dotnet_thread_pool_completed_items_count
process_runtime_dotnet_thread_pool_completed_items_count_total
process_runtime_dotnet_thread_pool_queue_length
process_runtime_dotnet_thread_pool_threads_count
process_runtime_dotnet_timer_count
When push into Alloy:
ec_Npgsql_busy_connections
ec_Npgsql_bytes_read_per_second_total
ec_Npgsql_bytes_written_per_second_total
ec_Npgsql_commands_per_second_total
ec_Npgsql_connection_pools
ec_Npgsql_current_commands
ec_Npgsql_failed_commands
ec_Npgsql_idle_connections
ec_Npgsql_prepared_commands_ratio
ec_Npgsql_total_commands
geoserver_request_count_total
geoserver_request_sum_milliseconds_total
http_client_duration_milliseconds_bucket
http_client_duration_milliseconds_count
http_client_duration_milliseconds_sum
http_server_duration_milliseconds_bucket
http_server_duration_milliseconds_count
http_server_duration_milliseconds_sum
process_runtime_dotnet_assemblies_count
process_runtime_dotnet_exceptions_count_total
process_runtime_dotnet_gc_allocations_size_bytes_total
process_runtime_dotnet_gc_collections_count_total
process_runtime_dotnet_gc_committed_memory_size_bytes
process_runtime_dotnet_gc_heap_size_bytes
process_runtime_dotnet_gc_objects_size_bytes
process_runtime_dotnet_jit_compilation_time_nanoseconds_total
process_runtime_dotnet_jit_il_compiled_size_bytes_total
process_runtime_dotnet_jit_methods_compiled_count_total
process_runtime_dotnet_monitor_lock_contention_count_total
process_runtime_dotnet_thread_pool_completed_items_count_total
process_runtime_dotnet_thread_pool_queue_length
process_runtime_dotnet_thread_pool_threads_count
process_runtime_dotnet_timer_count

As I understand, i’m missing something in configuration. That’s my Alloy config:

otelcol.receiver.otlp "otlp"{
  http {endpoint = "0.0.0.0:4318"}
  output {
    metrics = [otelcol.processor.attributes.attributes.input]
  }
}
otelcol.processor.attributes "attributes" {
    action {
        key = "prod.name"
        value = "dev_devops74"
        action = "insert"
    }
    output {
        metrics = [otelcol.processor.transform.resources.input]
    }
}
otelcol.processor.transform "resources" {
    metric_statements {
      context = "metric"
        statements = [
          "set(resource.attributes[\"prod_name\"], \"dev_devops74\")",
        ]
    }

    output {
      metrics = [otelcol.exporter.prometheus.prometheus.input]
    }
}
otelcol.exporter.prometheus "prometheus" {
  forward_to = [prometheus.remote_write.local_prom.receiver]
  include_target_info = true
}

prometheus.remote_write "local_prom" {
    endpoint { url = "http://10.1.1.2:9090/api/v1/write" }
}

Alloy version v1.1.1

2

Hello! I don’t see anything in your config which could lead to dropping metrics. You could use the OTel debug exporter or the UI to see if the metrics are being ingested, and to see if they’re still there before being translated to Prometheus metrics.

3

I found some regularity in this case. Metrics with DataType: Sum don’t appear in prometheus. I guess it’s about how Alloy convert OpenTelemetry format in Prometheus Format. What can i do to transform metrics right?

4

Hello, metrics being ingested in otelcol.receiver.otlp, but lost during transmission in prometheus.