Linking Traces to Logs with Clickhouse as data source for all Telemetry

Grafana Dashboards
, ,
1

  • What Grafana version and what operating system are you using?
    Grafana Version : 10.0.3
    OS : Arch Linux

  • What are you trying to achieve?

Context : We are using OTeL collector (contrib clickhouse exporter) to export the Telemetry data (Logs, Metrics and Traces) to clickhouse, and displaying the traces in Traces Panel using query. Following is the query used.

WITH
('e98e9b25211e06aa75afc1835e1dc5b7','e3b8098b03f80d8be19a546442b77a5a', '9944890d82136a0641b0af8349d154b0', '7177256df5eb7a2cf0582c32a1b11e22') AS trace_ids,
(
SELECT min(Start)
FROM otel.otel_traces_trace_id_ts
WHERE TraceId in trace_ids
) AS start,
(
SELECT array(map('key', 'error', 'value', 'true'))
) AS errorTrue, 
(
SELECT array(map('key', 'error', 'value', 'false'))
) AS errorFalse, 
(
SELECT array(map('timestamp', '1692615893', 'fields', NULL))
) AS traceLogs,
(
SELECT array(map('traceID', 'a92c186f8050b2af62b19aae4b3591d2', 'spanID', 'bff0d8de38dfa333')))
) AS traceRef,
(
SELECT max(End) + 1
FROM otel.otel_traces_trace_id_ts
WHERE TraceId in trace_ids
) AS end
SELECT
TraceId AS traceID,
SpanId AS spanID,
SpanName AS operationName,
ParentSpanId AS parentSpanID,
ServiceName AS serviceName,
Duration / 1000000 AS duration,
Timestamp AS startTime,
traceLogs AS logs,
traceRef AS references, 
arrayConcat(arrayMap(key -> map('key', key, 'value', SpanAttributes[key]), mapKeys(SpanAttributes)), errorTrue) AS tags,
arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags
FROM otel.otel_traces
WHERE (TraceId in trace_ids) AND (Timestamp >= start) AND (Timestamp <= end)
ORDER BY startTime ASC

With Clickhouse as data source (plugin), I am trying to link logs related to spans in “Traces” panel. Below is the screenshot of how the trace panel looks now, we want that “Logs for this span” button to work.

DeepinScreenshot_select-area_20230822011025
DeepinScreenshot_select-area_202308220110252540×674 97.6 KB

Following is one of the log’s screen shot

DeepinScreenshot_select-area_20230822020743
DeepinScreenshot_select-area_202308220207431484×652 65.6 KB

Query used for it was

SELECT "SeverityText" as level,
TraceId AS traceID,
SpanId AS spanID,
ScopeName AS scopeName,
ServiceName AS serviceName,
Body as msg,
Timestamp AS timeStamp,
arrayMap(key -> map('key', key, 'value', ScopeAttributes[key]), mapKeys(ScopeAttributes)) AS tags,
arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags
FROM "otel"."otel_logs" WHERE SpanId=${query0:singlequote}
  • How are you trying to achieve it?

Trying to format the query and included the necessary fields as described in Trace Data Api Doc.

We thought logs field maps to that but it maps to “Events” shown in the spans.

Now we are trying to go through the Grafana docs and Github repo to find some cues.

We are actually unsure of how if Grafana going to link the logs to the spans, there is one way of just copying the spanId and pasting it in some variable to get to the logs, but we want an easier user journey.

  • What happened?

We couldn’t find anything around what field maps to that botton/link (Logs for this span, as show in the screen shot)

  • Did you follow any online instructions? If so, what is the URL?

We couldn’t find any on it – explaining how to make the button work for our case.

Our assumption : If it can be done for tempo (as a data source) there must be some formatting or query which can produce similar data frames as tempo does.