Hello,
I ask how you can import in a programmatic way dashboards to another environment.
Is related to this issue in Github [Grafana] 8.3.X Copy a Dashboard to another environment by copying it from Json Model · Issue #43440 · grafana/grafana · GitHub
I run a Powershell to Get all Dashboards and Folders and be able to import them in another enviroment(We used to do it for Grafana Version 4.6 and since the datasource was a simple string in the dashboard json, it give no problem).
I paste here the script
$user = 'xxxx'
$pass = 'xxxx'
$port = 8080
$server = 'xxxxxxx'
$outputPath = "$PSScriptRoot\output"
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"002_Dashboard"))";
$searchForAlarmsDashboards = $false;
rm -Path $outputPath -Recurse
New-Item -ItemType Directory -Force -Path $outputPath > $null;
New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
$folders = Invoke-RestMethod -Uri "http://$($server):$($port)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
$folders | % {
$folder = New-Object System.Object;
$folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
$folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
$folder | ConvertTo-Json | Out-File "$($outputFolderPath)\$($_.uid).folderjson";
}
$apiDashboardUri = "http://$($server):$($port)/api/dashboards";
$dashboards = Invoke-RestMethod -Uri "http://$($server):$($port)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
$dashboard = Invoke-RestMethod -Uri "$($apiDashboardUri)/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
$panelBody = New-Object System.Object;
$panelBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
$panelBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
$panelBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
$panelBody | ConvertTo-Json -Depth 100 | Out-File "$($outputDashboardPath)\$($_.uid).json";
}
A part of the panelBody has an uid that identifies the Datasource (That usually changes between the different enviroments).
This is the only solution that I see but I see that in the database api you can change the uid easily so I think that it is the only solution
$user = 'xxxxxx'
$pass = 'xxxxx'
$port = 8080
$server = 'xxxxxx'
$outputPath = "$PSScriptRoot\output"
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDbPath = "$([System.IO.Path]::Combine($outputPath,"002_DB"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"003_Dashboard"))";
$searchForAlarmsDashboards = $false;
$mainUrl = "http://$($server):$($port)"
rm -Path $outputPath -Recurse
New-Item -ItemType Directory -Force -Path $outputPath > $null;
New-Item -ItemType Directory -Force -Path $outputDbPath > $null;
New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
$folders = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
$folders | % {
$folder = New-Object System.Object;
$folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
$folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
$folder | ConvertTo-Json | Out-File "$($outputFolderPath)\$($_.uid).folder";
}
$dbs = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
$dbs | % {
$db = New-Object System.Object;
$db | Add-Member -NotePropertyName name -NotePropertyValue $_.name;
$db | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
$db | ConvertTo-Json | Out-File "$($outputDbPath)\$($_.name).db";
}
$dashboards = Invoke-RestMethod -Uri "$($mainUrl)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
$dashboard = Invoke-RestMethod -Uri "$($mainUrl)/api/dashboards/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
$panelBody = New-Object System.Object;
$panelBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
$panelBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
$panelBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
$panelBody | ConvertTo-Json -Depth 100 | Out-File "$($outputDashboardPath)\$($_.uid).json";
}
Finally this is the script that I use for BackUp the Dashboards and Folders from the master environment:
$user = 'xxxxxx';
$pass = 'xxxxxx';
$port = 8080;
$server = 'xxxx';
$outputPath = "$PSScriptRoot\output";
$outputFolderPath = "$([System.IO.Path]::Combine($outputPath,"001_Folder"))";
$outputDbPath = "$([System.IO.Path]::Combine($outputPath,"002_DB"))";
$outputDashboardPath = "$([System.IO.Path]::Combine($outputPath,"003_Dashboard"))";
$searchForAlarmsDashboards = $false;
$mainUrl = "http://$($server):$($port)";
rm -Path $outputPath -Recurse -ErrorAction SilentlyContinue;
New-Item -ItemType Directory -Force -Path $outputPath > $null;
if(-not $searchForAlarmsDashboards){
New-Item -ItemType Directory -Force -Path $outputDbPath > $null;
New-Item -ItemType Directory -Force -Path $outputFolderPath > $null;
}
$changelogStringBuilder = [System.Text.StringBuilder]::new();
$changelogStringBuilder.AppendLine("# $packageName") > $null;
$changelogStringBuilder.AppendLine("Package of all default grafana panels") > $null;
$changelogStringBuilder.AppendLine("") > $null;
$changelogStringBuilder.AppendLine("## [$nugetVersion] - $(Get-Date -format yyyy-MM-dd)") > $null;
$changelogStringBuilder.AppendLine("### Changed") > $null;
$changelogStringBuilder.AppendLine("- $nugetVersionReleaseNotes") > $null;
$changelogStringBuilder.AppendLine("### Added") > $null;
New-Item -ItemType Directory -Force -Path $outputDashboardPath > $null;
$Headers = @{
Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($user):$($pass)")))"
}
if(-not $searchForAlarmsDashboards){
Write-Host "Getting folders";
$folders = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
$folders | % {
$folder = New-Object System.Object;
$folder | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
$folder | Add-Member -NotePropertyName title -NotePropertyValue $_.title;
$folder | ConvertTo-Json | Out-File ([System.IO.Path]::Combine($outputFolderPath,"$($_.uid).folder"));
$changelogStringBuilder.AppendLine("- Folder $($_.title)") > $null;
}
Write-Host "Getting Dbs";
$dbs = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
$dbs | % {
$db = New-Object System.Object;
$db | Add-Member -NotePropertyName name -NotePropertyValue $_.name;
$db | Add-Member -NotePropertyName uid -NotePropertyValue $_.uid;
$db | ConvertTo-Json | Out-File ([System.IO.Path]::Combine($outputDbPath,"$($_.name).db"));
}
}
Write-Host "Getting Dashboards";
$dashboards = Invoke-RestMethod -Uri "$($mainUrl)/api/search/?type=dash-db" -Method Get -Headers $Headers -ContentType 'application/json';
$dashboards | ? {($searchForAlarmsDashboards -and ($_.tags.Contains("alarms"))) -or (-not $searchForAlarmsDashboards -and (-not $_.tags.Contains("alarms")))} | % {
$dashboard = Invoke-RestMethod -Uri "$($mainUrl)/api/dashboards/uid/$($_.uid)" -Method Get -Headers $Headers -ContentType "application/json";
$changelogStringBuilder.AppendLine("- Dashboard slug: $($dashboard.meta.slug), updated: $($dashboard.meta.updated), updatedBy: $($dashboard.meta.updatedBy), version: $($dashboard.meta.version) uid: $($_.uid)") > $null;
$dashboard | ConvertTo-Json -Depth 100 | Out-File ([System.IO.Path]::Combine($outputDashboardPath,"$($_.uid).json"));
}
Add-Content ([System.IO.Path]::Combine($outputPath,"changelog.txt")) $changelogStringBuilder.ToString();
And this is the script that I have for Importing them:
$inputPath = "$PSScriptRoot"
if(-not(Test-Path -Path $inputPath)){
Write-Error "This path: $inputPath not found"
return;
}
$Headers = @{
Authorization = "Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($GrafanaAdminUser):$($GrafanaAdminPassword)")))"
}
$mainUrl = "http://$($GrafanaNotificationChannelsServer):$($GrafanaHttpPort)";
$health = Invoke-RestMethod -Uri "$($mainUrl)/api/health" -Method Get -Headers $Headers -ContentType "application/json";
$versionGrafana = $null;
if([version]::TryParse($health.version, [ref] $versionGrafana)){
Write-Host "Version in Grafana is $versionGrafana";
$minVersionWithFolders = [version]::new(8,0,0);
if($versionGrafana.CompareTo($minVersionWithFolders) -ge 0){
Write-Host "Version in Grafana is $versionGrafana which is greater than $minVersionWithFolders";
#Reading Folder its necesary to create the dashboard in the correct folder
$foldersInGrafana = Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Get -Headers $Headers -ContentType 'application/json';
ls -Path "$inputPath\*.folder" -Recurse | % {
$folder = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
$folderInGrafana = $foldersInGrafana | ? {$_.title -eq $folder.title};
if($folderInGrafana -eq $null){
Write-Host "Creating folder $($folder.title)";
Invoke-RestMethod -Uri "$($mainUrl)/api/folders" -Method Post -Headers $Headers -ContentType "application/json" -Body ($folder | ConvertTo-Json);
}
else{
Write-Host "Folder already exist in grafana";
}
}
#Reading datasources its necesary to update uid of the datasource because its written in the dashboard
$dbsInGrafana = Invoke-RestMethod -Uri "$($mainUrl)/api/datasources" -Method Get -Headers $Headers -ContentType 'application/json';
ls -Path "$inputPath\*.db" -Recurse | % {
$db = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
$dbInGrafana = $dbsInGrafana | ? {$_.name -eq $db.name};
if($dbInGrafana -eq $null){
Write-Host "Db $($db.name) does not exist in Grafana";
}
else{
Write-Host "Db $($db.name) does exist in Grafana updating uid";
$dbInGrafana.uid = $db.uid;
Invoke-RestMethod -Uri "$($mainUrl)/api/datasources/$($dbInGrafana.id)" -Method Put -Headers $Headers -ContentType "application/json" -Body ($dbInGrafana | ConvertTo-Json);
}
}
}
else{
Write-Host "Version in Grafana is $versionGrafana which is lower than $minVersionWithFolders";
}
ls -Path "$inputPath\*.json" -Recurse | % {
$dashboard = [System.IO.File]::ReadAllText($_.FullName) | ConvertFrom-Json;
$dashboardBody = New-Object System.Object;
$dashboardBody | Add-Member -NotePropertyName dashboard -NotePropertyValue $dashboard.dashboard;
if($dashboard.meta.folderUid -ne $null){
$dashboardBody | Add-Member -NotePropertyName folderUid -NotePropertyValue $dashboard.meta.folderUid;
}
$dashboardBody | Add-Member -NotePropertyName overwrite -NotePropertyValue $true;
$dashboardBody.dashboard.id = $null; #Eliminamos los id, para poder importar.
Write-Host "Creating dashboard:$($dashboardBody.dashboard.title) Folder:$($dashboardBody.folderUid)";
Invoke-RestMethod -Method Post -Headers $Headers -ContentType "application/json" -Uri "$($mainUrl)/api/dashboards/db" -Body ($dashboardBody | ConvertTo-Json -Depth 100 );
Write-Host "Updating dashboard:$($dashboardBody.dashboard.title) Folder:$($dashboardBody.folderUid)";
## Invoke twice to make the alarms start
Invoke-RestMethod -Method Post -Headers $Headers -ContentType "application/json" -Uri "$($mainUrl)/api/dashboards/db" -Body ($dashboardBody | ConvertTo-Json -Depth 100 );
}
}
else{
write-Error "Not parsed version in Grafana";
}
I think that both are valid, but I need to know if this is the correct form of importing and exporting dashboards and folders.
Dbs are being created earlier in the process so when you import the dashboards you have to make sure that all the dbs have the same uid
Hi,
Can you help me ?
Will
Hi @wilfridrussodivito
So you have a Unauthorized response. Take into account that this was a question and it was never answered (It is not an official solution), if you want to just copy a dashboard to other environment and you have few environments this could not be the solution.
But the company I worked for had over 50 panels and over 20 platforms and we wanted a way to automatize it.
jgarciadelanoceda:
dashboards
Hi,
Many thanks.
Thanks:
Are you trying to move dashboards to a new server?
No. The goal it’s to export all folders (and all dashboards) from grafana in json file (for the dashboards) and import all this in Team Foundatiom server.
dashboard 1
dashboard 1
1 Like
Hello @yosiasz ,
The company I worked for had over 20 platforms were we did Monitoring using Grafana and InfluxDb, we wanted to have the same dashboards in all of them
1 Like
Here is what I wrote in python. The basic idea can be done in any other scripting language
import requests
import json
from dotenv import load_dotenv
load_dotenv()
import os
TOKEN=os.getenv('TOKEN')
URL=os.getenv('URL')
headers= {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': 'Bearer {}'.format(TOKEN)
}
r = requests.get(url = URL, headers = headers)
for d in r.json():
uid = d.get('uid')
title = d.get('title')
show original
yosiasz
December 2, 2022, 2:09pm
10
If you want to replicate the same structure it sounds like moving to me
In that case what I do is copy the data and conf folder to the secondary server and restart grafana - done
Thanks,https://yourgrafana/ with api or something else ?
Could you explain the solution a little bit more?
I think that this could be interesting for the rest of the community
yosiasz
December 2, 2022, 2:16pm
13
So we go 2 asks going here. One for TFS and the other for replicating same folder structure in another grafana instance. Lets not conflate those.
For tfs check this out
In the dashboard api, for a specitic dashboard, you can get folderId and folderGuid, you can find the folder name from these 2 values and create the folder accordingly and follower by saving the dashboard json to that folder programatically
Another instance of grafana
1 Like
Yes : i need to import the structure of Grafana (only folders and dashboards) in Team Foundation Server. For the moment i loop on Grafana’s folders and for each folder, i create it on Windows like this :.title .title -Path $outputFolderPath;
This was my original Idea @wilfridrussodivito check the dashboards importing scripts, because it is what it does (Although if you had already the same folderId with different Uid in the destination I think that the solution could have a problem)
I do not have access to the environments anymore so I am not updating the solution
It is also important to say that the databases in the destination environment were deployed in a previous step of the pipeline
yosiasz
December 2, 2022, 2:26pm
18
Importing to a different instance of grafana via script?
Yes
It was my only goal I had
yosiasz
December 2, 2022, 2:30pm
20
Why complicate our lives
Keep it simple. Copy data and conf folder to secondary instance. Couple of lines of code, very easy to maintain, wont break if underlying api changes
I would not recommend the complicated loop to json and import approach, way too complicated
