Recording rule error: remote write failed: data source uid not specified and no default set (despite uid specified AND default set)

  • What Grafana version and what operating system are you using?

    • Grafana v12.3.0 (20051fb1fc) on Docker container, over Ubuntu 24.04.4 LTS
  • What are you trying to achieve?

    • I created a couple of recording rules, that are getting metrics from a datasource to build time-series data from it
  • How are you trying to achieve it?

    • I’m using provisioning to create the Recording rules from a YAML file:
    • 
      apiVersion: 1
      groups:
          - orgId: 1
            name: evalgroup-recordingrules-1m
            folder: alerts-folder
            interval: 1m
            rules:
              - uid: af5ti531c4ttsa
                title: Active Session Count
                data:
                  - refId: A
                    queryType: alerting
                    relativeTimeRange:
                      from: 600
                      to: 0
                    datasourceUid: samm-citrixodata-datasource
                    model:
                      count: true
                      datasource:
                          type: samm-citrixodata-datasource
                          uid: samm-citrixodata-datasource
                      entitySet:
                          entityType: Citrix.Monitor.DataService.V4.Session
                          name: Sessions
                      expression: ""
                      filterConditions:
                          - operator: eq
                            property:
                              name: ConnectionState
                              type: Edm.Int32
                            value: "5"
                      instant: true
                      intervalMs: 1000
                      maxDataPoints: 43200
                      properties:
                          - name: UserId
                            type: Edm.Int64
                      queryType: alerting
                      range: false
                      refId: A
                      timeProperty: null
                  - refId: B
                    datasourceUid: __expr__
                    model:
                      conditions:
                          - evaluator:
                              params:
                                  - 0
                                  - 0
                              type: gt
                            operator:
                              type: and
                            query:
                              params: []
                            reducer:
                              params: []
                              type: avg
                            type: query
                      datasource:
                          name: Expression
                          type: __expr__
                          uid: __expr__
                      expression: A
                      intervalMs: 1000
                      maxDataPoints: 43200
                      reducer: last
                      refId: B
                      type: reduce
                isPaused: false
                record:
                  metric: ctxdaassessiontimeseries
                  from: B
                  targetDatasourceUid: c3a5e98c-6e94-44b8-b894-ae05241051b7
              - uid: ff5tiqkkund34c
                title: Disconnected Session Count
                data:
                  - refId: A
                    relativeTimeRange:
                      from: 600
                      to: 0
                    datasourceUid: samm-citrixodata-datasource
                    model:
                      datasource:
                          type: samm-citrixodata-datasource
                          uid: samm-citrixodata-datasource
                      entitySet:
                          entityType: Citrix.Monitor.DataService.V4.Session
                          name: Sessions
                      filterConditions:
                          - operator: eq
                            property:
                              name: ConnectionState
                              type: Edm.Int32
                            value: "2"
                      hide: false
                      intervalMs: 1000
                      maxDataPoints: 43200
                      properties:
                          - name: UserId
                            type: Edm.Int64
                      refId: A
                      timeProperty: null
                  - refId: B
                    datasourceUid: __expr__
                    model:
                      conditions:
                          - evaluator:
                              params: []
                              type: gt
                            operator:
                              type: and
                            query:
                              params:
                                  - B
                            reducer:
                              params: []
                              type: last
                            type: query
                      datasource:
                          type: __expr__
                          uid: __expr__
                      expression: A
                      intervalMs: 1000
                      maxDataPoints: 43200
                      reducer: last
                      refId: B
                      type: reduce
                isPaused: false
                record:
                  metric: ctxdaasdisconsessiontimeseries
                  from: B
                  targetDatasourceUid: c3a5e98c-6e94-44b8-b894-ae05241051b7
      
      
  • What happened?

    • Getting errors on the provisioned recording rule: “remote write failed: data source uid not specified and no default set”
    • and also on grafana logs:
    • grafana-1  | logger=plugin.samm-citrixodata-datasource t=2026-02-16T23:05:20.12893042Z level=info msg="Plugin Request Completed" dsName=samm-citrixodata-datasource endpoint=queryData pluginId=samm-citrixodata-datasource statusSource=plugin uname=grafana_scheduler dsUid=samm-citrixodata-datasource duration=124.587563ms status=ok
      grafana-1  | logger=ngalert.scheduler rule_uid=af5ti531c4ttsa org_id=1 rule_uid=af5ti531c4ttsa org_id=1 now=2026-02-16T23:05:20Z fingerprint=f5ed2ceb38402bd0 attempt=1 t=2026-02-16T23:05:20.129499829Z level=error msg="Failed to evaluate rule" attempt=1 error="remote write failed: data source uid not specified and no default set"
      grafana-1  | logger=plugin.samm-citrixodata-datasource t=2026-02-16T23:05:21.186149337Z level=info msg="Plugin Request Completed" duration=102.077327ms status=ok statusSource=plugin dsUid=samm-citrixodata-datasource endpoint=queryData pluginId=samm-citrixodata-datasource uname=grafana_scheduler dsName=samm-citrixodata-datasource
      grafana-1  | logger=ngalert.scheduler rule_uid=af5ti531c4ttsa org_id=1 rule_uid=af5ti531c4ttsa org_id=1 now=2026-02-16T23:05:20Z fingerprint=f5ed2ceb38402bd0 attempt=2 t=2026-02-16T23:05:21.186749946Z level=error msg="Failed to evaluate rule" attempt=2 error="remote write failed: data source uid not specified and no default set"
      
  • What did you expect to happen?

  • Can you copy/paste the configuration(s) that you are having problems with?

    • Adding related section from my datasource provisioning file:
    • apiVersion: 1
      
      datasources:
        - name: samm-prometheus
          type: prometheus
          uid: c3a5e98c-6e94-44b8-b894-ae05241051b7
          url: http://mimir:8080/prometheus
          isDefault: true
          access: proxy
          editable: true
          jsonData:
            httpHeaderName1: X-Scope-OrgID
            httpMethod: POST
            manageAlerts: true
            prometheusType: Mimir
            prometheusVersion: 2.9.1
            cacheLevel: 'High'
            disableRecordingRules: false
            incrementalQueryOverlapWindow: 10m
            pdcInjected": false
          secureJsonData:
            httpHeaderValue1: <headervalue>
      
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

    • Errors already pasted before.
  • Did you follow any online instructions? If so, what is the URL?

First of all, apologies if I didn’t set the proper section. I didn’t found anything dedicated to Recording Rules, and Alerting is the best match for this. I am at a loss, since the datasource ID matches the recording rule definition.