BigQuery : DoiT International to Native Grafana Data Source Migration

Hi Grafana Team,

We are heavily using BigQuery with Grafana and currently rely on the DoiT International BigQuery plugin across our environment.

As part of our Grafana upgrade initiative, we are planning to migrate to the latest Grafana version and move from the DoiT International BigQuery plugin to the native Grafana BigQuery data source plugin.

While reviewing the documentation, I found the following migration guidance:

Open the dashboard containing queries from the DoiT International plugin.

Edit each panel and change the data source to Grafana BigQuery.

Save the dashboard.

URL for reference: Configure the Google BigQuery data source | Grafana Plugins documentation

This approach works for a small number of dashboards, but in our case we have hundreds of dashboards and each dashboard contains multiple panels. Performing this migration manually panel-by-panel would be extremely time-consuming and error-prone.

I would like to know if there is a recommended way to perform this migration at scale.

Some options we are considering:

  1. Updating the datasource references directly in the Grafana backend database.
  2. Exporting dashboard JSON, performing bulk replacements, and re-importing the dashboards.
  3. Using the Grafana HTTP API to automate the migration.
  4. Any Grafana-supported migration utility or script that can automatically convert dashboards from the DoiT plugin to the native BigQuery datasource.

I have already experimented with the following approaches:

  1. Updating the datasource references directly in the Grafana backend database.
  2. Exporting dashboard JSON, performing bulk replacements, and re-importing the dashboards.

However, neither approach worked 100%. While the datasource name was updated successfully, other plugin-specific attributes, version information and metadata were not fully converted, resulting in incomplete migrations.

Could you please advise on the best and safest migration strategy for a large Grafana deployment? Any documentation, scripts, examples, or recommended procedures would be greatly appreciated.

We would like to avoid manual updates wherever possible and ensure the migration is completed reliably.

Thanks in advance for your help.

I would go with 3.

Also remember just changing the datasource name is not sufficient. need to update the datasource uid.

for example this is what you see in the sqlite backend for infinity

This would be the approach I would recommend using python. DO NOT USE ON LIVE SYSTEM. Use on test/qa environment

pseudocode

get all dashboards

loop for each dashboard

get dashboard.uid

check to see if current dashboard uses the old BigQuery.

if so using dashboard api update datasource.name and datasource.uid

you get the drift.

The reason your previous attempts failed is that you were only updating datasource.name → you also need to update datasource.type and datasource.uid. Additionally, datasource references exist in 4 places inside a dashboard JSON → panels, query targets, template variables, and annotations all of which need to be updated.

One more important thing → the DoiT plugin is Angular-based and Grafana 13 has completely removed Angular support. If you are upgrading to Grafana 13+, the DoiT plugin will stop working entirely.
You can use this Python script

python

import requests
import copy
import logging
from datetime import datetime

GRAFANA_URL         = "http://your-grafana-host:3000"
API_KEY             = "your-grafana-admin-api-key"
OLD_DATASOURCE_TYPE = "doitintl-bigquery-datasource"
OLD_DATASOURCE_UID  = "your-old-uid"
NEW_DATASOURCE_NAME = "BigQuery"
NEW_DATASOURCE_TYPE = "grafana-bigquery-datasource"
NEW_DATASOURCE_UID  = "your-new-uid"
DRY_RUN = True


logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler(f"migration_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"),
        logging.StreamHandler()
    ]
)
log = logging.getLogger(__name__)
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def get_all_datasources():
    resp = requests.get(f"{GRAFANA_URL}/api/datasources", headers=HEADERS)
    resp.raise_for_status()
    return resp.json()

def get_all_dashboards():
    resp = requests.get(f"{GRAFANA_URL}/api/search", headers=HEADERS, params={"type": "dash-db", "limit": 5000})
    resp.raise_for_status()
    return resp.json()

def get_dashboard(uid):
    resp = requests.get(f"{GRAFANA_URL}/api/dashboards/uid/{uid}", headers=HEADERS)
    resp.raise_for_status()
    return resp.json()

def save_dashboard(dashboard_json, folder_uid):
    payload = {"dashboard": dashboard_json, "folderUid": folder_uid, "overwrite": True, "message": "Migrated from DoiT to native Grafana BigQuery"}
    resp = requests.post(f"{GRAFANA_URL}/api/dashboards/db", headers=HEADERS, json=payload)
    resp.raise_for_status()
    return resp.json()

def is_doit(ds):
    if not isinstance(ds, dict):
        return False
    return ds.get("type") == OLD_DATASOURCE_TYPE or ds.get("uid") == OLD_DATASOURCE_UID

def replace_ds(ds):
    updated = copy.deepcopy(ds)
    updated["type"] = NEW_DATASOURCE_TYPE
    updated["uid"]  = NEW_DATASOURCE_UID
    if "name" in updated:
        updated["name"] = NEW_DATASOURCE_NAME
    return updated

def migrate_panel(panel):
    panel = copy.deepcopy(panel)
    changed = False
    if is_doit(panel.get("datasource")):
        panel["datasource"] = replace_ds(panel["datasource"])
        changed = True
    for target in panel.get("targets", []):
        if is_doit(target.get("datasource")):
            target["datasource"] = replace_ds(target["datasource"])
            changed = True
    if "panels" in panel:
        new_panels = []
        for sub in panel["panels"]:
            updated_sub, sub_changed = migrate_panel(sub)
            new_panels.append(updated_sub)
            changed = changed or sub_changed
        panel["panels"] = new_panels
    return panel, changed

def migrate_dashboard(full_response):
    dashboard = copy.deepcopy(full_response["dashboard"])
    total_changed = 0
    new_panels = []
    for panel in dashboard.get("panels", []):
        updated, changed = migrate_panel(panel)
        new_panels.append(updated)
        if changed:
            total_changed += 1
    dashboard["panels"] = new_panels
    for var in dashboard.get("templating", {}).get("list", []):
        if is_doit(var.get("datasource")):
            var["datasource"] = replace_ds(var["datasource"])
            total_changed += 1
    for ann in dashboard.get("annotations", {}).get("list", []):
        if is_doit(ann.get("datasource")):
            ann["datasource"] = replace_ds(ann["datasource"])
            total_changed += 1
    return dashboard, total_changed

def run_migration():
    log.info(f"Migration started | DRY_RUN={DRY_RUN}")
    log.info("Available datasources:")
    for ds in get_all_datasources():
        log.info(f"  {ds['name']} | {ds['type']} | uid={ds['uid']}")
    dashboards = get_all_dashboards()
    log.info(f"Total dashboards: {len(dashboards)}")
    summary = {"migrated": 0, "skipped": 0, "errors": 0}
    for dash in dashboards:
        uid, title = dash["uid"], dash["title"]
        try:
            full_response = get_dashboard(uid)
            folder_uid = full_response.get("meta", {}).get("folderUid", "")
            updated, total_changed = migrate_dashboard(full_response)
            if total_changed == 0:
                log.info(f"  SKIP  | {title}")
                summary["skipped"] += 1
                continue
            log.info(f"  MATCH | {title} — {total_changed} reference(s) to update")
            if not DRY_RUN:
                updated.pop("id", None)
                save_dashboard(updated, folder_uid)
                log.info(f"  SAVED | {title}")
            else:
                log.info(f"  DRY   | {title} (no changes applied)")
            summary["migrated"] += 1
        except Exception as e:
            log.error(f"  ERROR | {title} (uid={uid}): {e}")
            summary["errors"] += 1
    log.info(f"Done | Migrated={summary['migrated']} Skipped={summary['skipped']} Errors={summary['errors']}")
    if DRY_RUN:
        log.info("DRY_RUN=True — set to False to apply changes")

if __name__ == "__main__":
    run_migration()

Before running:

Take a Grafana DB backup first
Run GET /api/datasources to confirm your exact DoiT type and uid and fill them in the config
Run with DRY_RUN = True first to verify output before applying changes
Test on QA before production

Hi @infofcc3 , That’s great and it is going to help me a lot. I will check it within the next one or two days and will come back with an update here without fail. I am grateful for all your help’s and support. Thanks a lot!