-
What Grafana version and what operating system are you using?
v8.3.5 -
What are you trying to achieve?
Search logs indexed from GCP DataFlow jobs via Pubsub. -
How are you trying to achieve it?
Following the guidance here Cloud setup GCP Logs | Grafana Loki documentation and using terraform to deploy the necessary GCP resources and using the following chart with this additional promtail configuration
promtail:
serviceAccount:
create: false
name: "loki-promtail"
config:
snippets:
extraScrapeConfigs: |
- job_name: gcplog
gcplog:
project_id: "psychic-sensor-230117"
subscription: "psychic-sensor-230117-log-subscription"
use_incoming_timestamp: false # default rewrite timestamps.
# labels:
# job: "gcplog"
relabel_configs:
- source_labels: ["__gcp_resource_labels_job_name"]
target_label: "job_name"
-
What happened?
Promtail successfully scrapes logs for Dataflow jobs (as configured in the log sink) and they are accessible in Grafana but only the labelresource_typeappears to be indexed. -
What did you expect to happen?
Based on the PR here Promtail: improve GCP labels by james-callahan · Pull Request #4628 · grafana/loki · GitHub and the contents of the log line
{
"insertId": "ezphqvb5a",
"labels": {
"dataflow.googleapis.com/job_id": "2022-07-20_18_50_03-16530679851809344259",
"dataflow.googleapis.com/job_name": "beamapp-seanharkins-0721014956-764459-h4z54jbx",
"dataflow.googleapis.com/log_type": "system",
"dataflow.googleapis.com/region": "us-west1"
},
"logName": "projects/psychic-sensor-230117/logs/dataflow.googleapis.com%2Fjob-message",
"receiveTimestamp": "2022-07-21T01:50:04.350309382Z",
"resource": {
"labels": {
"job_id": "2022-07-20_18_50_03-16530679851809344259",
"job_name": "beamapp-seanharkins-0721014956-764459-h4z54jbx",
"project_id": "psychic-sensor-230117",
"region": "us-west1",
"step_id": ""
},
"type": "dataflow_step"
},
"severity": "INFO",
"textPayload": "Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.",
"timestamp": "2022-07-21T01:50:03.511899054Z"
}
Based on my promtail configuration I would have expected to see the label job_name available in grafana.
-
Can you copy/paste the configuration(s) that you are having problems with?
See above. -
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
No.