How to poll X-Ray data from AWS to Grafana?

Grafana Developers & API
,
1

We are using Grafana to monitor our AWS resources. Our current structure:

  • YACE → Prometheus (CloudWatch metrics)

  • Fluentd → Loki (CloudWatch Logs)

  • Thanos (Long-term storage)

  • Grafana (Visualization)

    We also have AWS X-Ray active tracing enabled on all of our Lambda functions, and Grafana can query X-Ray directly for live trace data. However, X-Ray only retains traces for 30 days, and we need long-term storage to match what we already have for metrics and logs.

    Ideally, we’d like a tool that can poll or scrape X-Ray trace data — similar to how YACE polls CloudWatch metrics — so we can store it long-term. Modifying our Lambda functions to send traces to an external collector isn’t feasible since they all use AWS-native active tracing with no custom instrumentation. We haven’t been able to find any open-source project that pulls trace data from the X-Ray API. Does anyone know of a tool or approach that fits this use case?

TL;DR: How to scrabe/poll the AWS X-Ray Data?

2

I guess you are using ADOT, so there is OTEL collector under the hood and you can configure it to export traces also to other destinations, e.g. to your Tempo instance.

3

We considered the ADOT dual-export approach, but it requires structural changes across all our Lambda functions - adding ADOT layers. Since we have many Lambdas already running with native X-Ray tracing (tracing_mode = Active), we’d rather keep the existing setup untouched and instead poll the X-Ray API from the observability side to pull traces into Tempo for long-term storage. But I could not find any tool (or project) to poll the X-Ray.

Note: I actually need the same approach in the metrics (YACE and Prometheus) for the X-RAY data.

4

Quick search:

IMHO pulling is fine for small load, but it may have problem with higher load.

1 Like