Forwarding Metrics from Grafana Alloy to Prometeus (using remote write)

I have a local Prometheus instance (on a Mac) with a remote_write feature enabled. I have also installed Grafana Alloy installed locally on the same computer using Homebrew.

 --config. file /usr/local/etc/prometheus.yml
 --web.listen-address=127.0.0.1:9090
 --storage.tsdb.path /usr/local/var/prometheus
 --web.config.file=/usr/local/etc/web.yml
 --web.enable-remote-write-receiver

I have also deployed Grafana Alloy locally. I want to send metrics from local code, using Opentelemetry SDK, to Grafana Alloy and forward them to Prometheus.

Here is my AlAlloy’sonfig file (config.al Alloy

logging {
  level  = "debug"
  format = "logfmt"
}


otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/v1/write"
    basic_auth {
      username = "admin"
      password = "password"
    }
  }
}

My code has OTLP exporter and sends the metrics to AlAlloy’s318 HTTP port:

using System.Diagnostics.Metrics;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);
var collectorUrl = "http://localhost:4318";
builder.Services.AddOpenTelemetry().WithMetrics(m =>
{
    m.SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService("OrderService"))
        .AddConsoleExporter()
        .AddOtlpExporter(o=>
        {
            o.Endpoint = new Uri(collectorUrl);
            o.Protocol = OtlpExportProtocol.HttpProtobuf;
        })
        .AddMeter("OrderServiceMeter");
    
});
builder.Services.AddMetrics();

var app = builder.Build();


Counter<int> otelOrderCount= null;

app.Use(async (context, next) =>
{
    var metricFactory = context.RequestServices.GetService<IMeterFactory>();
    var meter = metricFactory?.Create(new MeterOptions("OrderServiceMeter"));
    otelOrderCount = meter.CreateCounter<int>("otel_order_count");
    otelOrderCount?.Add(1);
    await next();
});

app.MapGet("/", () =>
{
    otelOrderCount?.Add(1);
    return "OK";
});

app.Run();

I don’t get the otel_order_count metric in Prometheus.

When I look at alloy.err.log I see this entry:

ts=2024-04-14T10:17:34.462683Z level=debug msg="Watcher is reading the WAL due to timeout, haven't received any write notifications recently" component_path=/ component_id=prometheus.remote_write.default subcomponent=rw remote_name=110ae3 url=http://localhost:9090/api/v1/write timeout=15s

Where in my configuration have I made a mistake? Unfortunately, Allog’s components do not emit debug information.

prometheus_remote_write_wal_storage_created_series_total is 0
prometheus_remote_write_wal_storage_active_series is 0
prometheus_remote_storage_samples_total is 0

Does it mean that no metric has been received by prometheus.remote_write ?