Forwarding Metrics from Grafana Alloy to Prometeus (using remote write)

Grafana
1

I have a local Prometheus instance (on a Mac) with a remote_write feature enabled. I have also installed Grafana Alloy installed locally on the same computer using Homebrew.

 --config. file /usr/local/etc/prometheus.yml
 --web.listen-address=127.0.0.1:9090
 --storage.tsdb.path /usr/local/var/prometheus
 --web.config.file=/usr/local/etc/web.yml
 --web.enable-remote-write-receiver

I have also deployed Grafana Alloy locally. I want to send metrics from local code, using Opentelemetry SDK, to Grafana Alloy and forward them to Prometheus.

Here is my AlAlloy’sonfig file (config.al Alloy

logging {
  level  = "debug"
  format = "logfmt"
}


otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/v1/write"
    basic_auth {
      username = "admin"
      password = "password"
    }
  }
}

My code has OTLP exporter and sends the metrics to AlAlloy’s318 HTTP port:

using System.Diagnostics.Metrics;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);
var collectorUrl = "http://localhost:4318";
builder.Services.AddOpenTelemetry().WithMetrics(m =>
{
    m.SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService("OrderService"))
        .AddConsoleExporter()
        .AddOtlpExporter(o=>
        {
            o.Endpoint = new Uri(collectorUrl);
            o.Protocol = OtlpExportProtocol.HttpProtobuf;
        })
        .AddMeter("OrderServiceMeter");
    
});
builder.Services.AddMetrics();

var app = builder.Build();


Counter<int> otelOrderCount= null;

app.Use(async (context, next) =>
{
    var metricFactory = context.RequestServices.GetService<IMeterFactory>();
    var meter = metricFactory?.Create(new MeterOptions("OrderServiceMeter"));
    otelOrderCount = meter.CreateCounter<int>("otel_order_count");
    otelOrderCount?.Add(1);
    await next();
});

app.MapGet("/", () =>
{
    otelOrderCount?.Add(1);
    return "OK";
});

app.Run();

I don’t get the otel_order_count metric in Prometheus.

When I look at alloy.err.log I see this entry:

ts=2024-04-14T10:17:34.462683Z level=debug msg="Watcher is reading the WAL due to timeout, haven't received any write notifications recently" component_path=/ component_id=prometheus.remote_write.default subcomponent=rw remote_name=110ae3 url=http://localhost:9090/api/v1/write timeout=15s

Where in my configuration have I made a mistake? Unfortunately, Allog’s components do not emit debug information.

prometheus_remote_write_wal_storage_created_series_total is 0
prometheus_remote_write_wal_storage_active_series is 0
prometheus_remote_storage_samples_total is 0

Does it mean that no metric has been received by prometheus.remote_write ?

2

I have the same problem

ts=2024-06-25T06:01:52.187851811Z level=debug msg="Watcher is reading the WAL due to timeout, haven't received any write notifications recently" component_path=/ component_id=prometheus.remote_write.default subcomponent=rw remote_name=8f4ada url=http://admin:password@prometheus:9090/api/v1/write timeout=15s

My Alloy is

logging {
  level  = "debug"
  format = "logfmt"
}

otelcol.receiver.otlp "default" {
  http {
    endpoint = "0.0.0.0:4318"
  }
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
    logs    = [otelcol.exporter.loki.default.input]
    traces  = [otelcol.exporter.otlphttp.tempo.input]
  }
}

otelcol.exporter.otlphttp "tempo" {
  client {
    endpoint = "http://tempo:4318"
    tls {
      insecure             = true
      insecure_skip_verify = true
    }
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus:9090/api/v1/write"

	tls_config {
		server_name = "prometheus.portal.domain.tld"
		insecure_skip_verify = true
	}

    basic_auth {
      username = "admin"
      password = "password"
    }
  }
}


otelcol.exporter.loki "default" {
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
    tenant_id = "tenant1"
  }
  external_labels = {}
}

and use Docker

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./shared/prometheus:/etc/prometheus
    ports:
      - "9090:9090"
    command:
      - --config.file=/etc/prometheus/prometheus-local.yml
      - --web.enable-remote-write-receiver
    networks:
      - mynet

I already made a lot of tricks and I have the same problem