Enabling service graphs

Grafana Alloy
1

I see traces in tempo datasource but can’t see the service graphs in that datasource. I must be missing something simple but I have no idea what.
Error is:

No service graph datasource selected

Please set up a service graph datasource in the datasource settings according to the Tempo documentation.

Prometheus:
traces_spanmetrics_calls_total, traces_service_graph_request_total and so on are seen in prometheus-app datasource.

Tempo config contains:

overrides:
per_tenant_override_config: /conf/overrides.yaml
metrics_generator_processors:
- ‘service-graphs’
- ‘span-metrics’
metrics_generator:
storage:
path: “/tmp/tempo”
remote_write:
- url: (REDACTED)

Tempo datasource (not sure if needed to have prometheus-app):

apiVersion: v1
data:
tempo.yaml: |-
apiVersion: 1
datasources:
- name: tempo
type: tempo
uid: tempo
url: redacted
isDefault: false
editable: false
access: proxy
orgId: 1
version: 1
jsonData:
httpMethod: POST
timeInterval: 30s
manageAlerts: false
datasourceUid: ‘prometheus-app’
kind: ConfigMap

Here is the alloy config:

logging {
level = “info”
format = “logfmt”
}

discovery.kubernetes “pods” {
role = “pod”
}

discovery.kubernetes “nodes” {
role = “node”
}

discovery.kubernetes “services” {
role = “service”
}

discovery.kubernetes “endpoints” {
role = “endpoints”
}

discovery.kubernetes “endpointslices” {
role = “endpointslice”
}

discovery.kubernetes “ingresses” {
role = “ingress”
}

//// METRICS
otelcol.receiver.otlp “default” {
grpc {
endpoint = “0.0.0.0:4317”
}
http {
endpoint = “0.0.0.0:4318”
}
output {
metrics = [otelcol.processor.k8sattributes.default.input]
// logs = [otelcol.processor.batch.default.input]
traces = [otelcol.processor.batch.default.input,otelcol.connector.servicegraph.default.input]
}
}

otelcol.processor.k8sattributes “default” {
extract {
label {
from = “pod”
}

metadata = [
  "k8s.namespace.name",
  "k8s.pod.name",
]

}

output {
metrics = [otelcol.processor.transform.add_kube_attrs.input]
}
}

otelcol.processor.transform “add_kube_attrs” {
error_mode = “ignore”

metric_statements {
context = “datapoint”
statements = [
“set(attributes["k8s.pod.name"], resource.attributes["k8s.pod.name"])”,
“set(attributes["k8s.namespace.name"], resource.attributes["k8s.namespace.name"])”,
]
}

output {
metrics = [otelcol.processor.batch.default.input]
}
}

otelcol.processor.batch “default” {
output {
metrics = [otelcol.exporter.prometheus.default.input]
// logs = [otelcol.exporter.loki.default.input]
traces = [otelcol.exporter.otlp.default.input]
}
}

// Tempo traces
otelcol.connector.servicegraph “default” {
dimensions = [“http.method”, “http.target”]
output {
metrics = [otelcol.exporter.prometheus.default.input]
}
}
otelcol.exporter.otlp “default” {
client {
endpoint = “redacted:4317”
tls {
insecure = true
insecure_skip_verify = true
}
}
}
otelcol.exporter.prometheus “default” {
forward_to = [prometheus.relabel.instance.receiver]
}

// HOST
prometheus.exporter.unix “default” {}

prometheus.scrape “host” {
targets = prometheus.exporter.unix.default.targets
forward_to = [prometheus.relabel.instance.receiver]
extra_metrics = true
}

prometheus.relabel “instance” {
forward_to = [prometheus.remote_write.default.receiver]
rule {
action = “labeldrop”
regex = “instance”
}
}

prometheus.remote_write “default” {
endpoint {
// REPLACE
url = “redacted”
}
}

I have a feeling I haven’t configured alloy correctly because the example given in the documentation asks for the following flow:
otelcol.receiver.otlp → otelcol.connector.servicegraph.default → otelcol.exporter.otlp.default → otelcol.exporter.otlp

But mine is: otelcol.receiver.otlp → otelcol.connector.servicegraph.default → otelcol.exporter.prometheus.default → prometheus.relabel.instance → prometheus.remote_write.default

If I change otelcol.exporter.prometheus.default to otelcol.exporter.otlp.default my alloy instance starts throwing errors like:

ts=2024-10-15T13:09:16.783691406Z level=error msg=“Exporting failed. Dropping data.” component_path=/ component_id=otelcol.exporter.otlp.default error=“not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService” dropped_items=45

Any advice is welcome!

Thanks

2

Bump. This issue is still ongoing.