Best approach for multi-tenant / multiple orgs for external clients

Grafana MySQL
, , ,
1

We provide connectivity services and would like to use grafana to provide our external clients with one or more relatively simple dashboards to show equipment status and bandwidth utilization and so on.

Since these are external clients, we do not want any access to settings to be available to them, and clients should be completely isolated from each other.

Since there are also a few hundred of them, we would like to keep any manual updates or other tasks to organizations, user accounts and dashboards to an absolute minimum.

Our planned approach to do this is now as follows, any feedback and suggestions for improvements are very much welcome.

Our grafana is running an unlicensed Enterprise version, on latest stable repo installed yesterday (5/29/20). OS is CentOS 8.

Organizations
Looking at Provisioning | Grafana documentation it seems that provisioning of organizations is not possible.
As an alternative we now plan to create the organizations directly in the mysql database. (we do not use the default sqlite session database.)

INSERT INTO org (version, name, created, updated) VALUES ('1', '*$ClientName*', curdate(), curdate())

The Organization ID is auto-increment so we donā€™t need to worry about this. Most other columns are not mandatory, except for version (set to 1 here) and created and updated timestamps (here set to the current system date/time).

User Authentication
Because we want to reuse the credentials supplied to the client for other applications (e.g. ticketing), we have created client records in LDAP. Each client-company has its own OU and Security Group, and individual user accounts for each client are created in their respective OU and added to their Security Group.

To allow these accounts to then successfully log in to Grafana I believe this needs to be added in /etc/grafana/ldap.toml

We could create some scripting solution to insert multiple [[servers.group_mappings]] blocks looking something like this:

[[servers.group_mappings]]
group_dn = "CN=$ClientNameGroup1,OU=$ClientName1,DC=domain,DC=tld"
org_role = "Viewer"
org_id ="$ClientOrgID"

[[servers.group_mappings]]
group_dn = "CN=$ClientNameGroup2,OU=$ClientName2,DC=domain,DC=tld"
org_role = "Viewer"
org_id ="$ClientOrgID"

and so on and so on.

To retrieve the org ID from the MySQL DB itā€™s possible to use a command like this:

mysql -ugrafanauser -ppassword -sN grafana -e "select id from org where name='$ClientName';"

The output (because of -sN) is only the number (e.g. ā€˜2ā€™), so this can be directly stored in a script variable.

To avoid any potential issues with assigning the org ID Iā€™ll keep the default org ID empty without any datasource and with a text panel on the home dashboard asking the user to contact support.

Datasource Provisioning
It looks like that we have to create separate entries for each organization in /etc/grafana/provisioning/datasources/datasource.yaml, even if it points to the same datasource.

apiVersion: 1
datasources:
 - name: Graphite
   type: graphite
   access: proxy
   orgid: 2
   uid: $ClientName-DS
   url: http://graphitehost.domain.tld:8080

and so on and so on

Dashboard Provisioning
For the dashboard providers it is not yet clear to me what the best approach will be. The example provided in the provisioning page in the docs does not provide a file name, but just the directory path.So it seems to me that I might have to create a separate dashboard directory for each client, which will then have their own dashboard json files in them with each their own version numbering.

Of course these dashboard JSON files will be mostly identical.

I am currently in the process of setting this system up, but would very much like to hear experiences and advice from people with similar setups or plans.

If there is any interest I can convert this into a How-To when I have everything working and cleared up all unknowns.

2 Likes
2

Hi,
Would be interested in a how-to of your setup because i have the same problematic

Iā€™ve seen that an ansible module for grafana is available for datasource and dashboard provisioning
https://docs.ansible.com/ansible/latest/modules/grafana_datasource_module.html
https://docs.ansible.com/ansible/latest/modules/grafana_dashboard_module.html

Also for the dashboard, have you a datasource per client ?
Because i thinked of adding a hidden variable in the dashboard with a prometheus tag for each clients. (iā€™ve only one datasource for all clients)

3

Hi Zoinzibar,

Ultimately using the API seemed to be the only choice.Not only will the provisioning method above be deprecated in a future version, it also just simply doesnā€™t work very well. We had lots of errors about grafana not being able to delete old versions of a dashboard that didnā€™t exist yet and other annoying issues that just made it very impractical.

So after tossing all the work in my OP away and focusing on the API method, I came up with the following:

#! /bin/bash

# API Credentials
Username=Username
Password=Password

# Collect information from user:
printf "Please enter Customer Company Name (Long Format): "
read CompanyLongName
echo
CompanyShortName=$(echo $CompanyLongName | tr -d ' [:punct:]')
echo $CompanyLongName has been recorded, for internal purposes the short version of this wil be $CompanyShortName.
echo
printf "Enter the current bandwidth capacity of this client in bits per second (bps). (e.g. 20 Mbps is '20000000'): "
read CompanyCapacity
echo

echo Create Grafana Org...
Response=$(curl --silent  -X POST -H 'Content-Type: application/json' -d "{\"name\":\"$CompanyLongName\"}" http://$Username:$Password@community.grafana.com/api/orgs)
echo $Response | jq '.'
OrgID=${Response//[!0-9]/}

echo Create API Key...
OrgSwitchResponse=$(curl --silent -X POST http://$Username:$Password@community.grafana.com/api/user/using/$OrgID)
echo $OrgSwitchResponse | jq '.'
APIKeyResponse=$(curl --silent -X POST -H 'Content-Type: application/json' -d '{"name":"apikeycurl", "role": "Admin"}' http://$Username:$Password@community.grafana.com/api/auth/keys)
echo $APIKeyResponse | jq '.'
APIKey=$(echo $APIKeyResponse | cut -d',' -f 2 | cut -d':' -f 2 | tr -d '"{}')

echo Create Data Source...
curl --silent -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' -H "Authorization: Bearer $APIKey"  -d "{\"name\": \"$CompanyLongName\", \"type\":\"graphite\", \"url\":\"http://collectd.company.tld:8080\",\"access\":\"proxy\",\"basicAuth\":false,\"isDefault\":true}" http://grafana.staged-by-discourse.com/api/datasources | jq '.'

echo Create basic dashboard...
cp ./base.json ./$CompanyShortName.json
sed -i "s/VARCompanyLongName/$CompanyLongName/g" ./$CompanyShortName.json
sed -i "s/VARCompanyShortName/$CompanyShortName/g" ./$CompanyShortName.json
sed -i "s/VARCompanyCapacity/$CompanyCapacity/g" ./$CompanyShortName.json

echo Importing dashboard in Grafana...
DashboardImport=$(curl --silent -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer $APIKey" -d @$CompanyShortName.json http://grafana.staged-by-discourse.com/api/dashboards/db)
echo $DashboardImport | jq '.'

HomeDashboardID=$(echo $DashboardImport | cut -d':' -f 2 | cut -d',' -f 1)
curl -X PUT http://grafana.staged-by-discourse.com/api/user/preferences -H 'Content-Type: application/json' -d "{\"homeDashboardID\":$HomeDashboardID}" -H "Authorization: Bearer $APIKey"

echo creating ldap user mappings
echo >> /etc/grafana/ldap.toml
echo "[[servers.group_mappings]]" >> /etc/grafana/ldap.toml
echo "group_dn = \"CN=${CompanyShortName}_grp,OU=$CompanyLongName,OU=Clients,DC=company,DC=tld\"" >> /etc/grafana/ldap.toml
echo "org_role = \"Viewer\"" >> /etc/grafana/ldap.toml
echo "org_id = $OrgID" >> /etc/grafana/ldap.toml

echo Recording data...
echo $CompanyShortName,$CompanyLongName,$CompanyCapacity,$OrgID,$APIKey >> ClientList.csv

echo Done.

This script first takes the name of the customer (in human-readable / nice format), creates a shortened version of it, and it asks for the capacity of the service. This is injected in the base.json dashboard so that it becomes a ConstantLine value in our graphite backend and shows up in graphs and so on.

It then creates the organization, the API key and the data source. For the basic dashboard we have created a template dashboard, and exported the json code. Unlike with provisioned dashboards you can use any kind of exported json, that in itself would have been enough reason to drop the provisioned dashboards approach by the way.

In this base.json I have replaced all occurences of the long name, short name and capacity with some custom keywords so that I can use sed to replace these with the values for the current client.

The resulting dashboard is then imported to the org in grafana using the previously generated API key, and it is also made the home dashboard (something that is also not possible without using the API!).

It also creates an LDAP group mapping in the ldap.toml file. so that they clients can log in using their ldap accounts.

Finally it records the necessary variables into a csv file so you can reference these later and possibly reuse them (especially the API key).

2 Likes
4

Thanks for the quick reply !

Good to know that the API method work properly, iā€™ll dig into it

Youā€™ve made an amazing work ! The script is perfect and with the API itā€™s extremely configurable, also for my need
I just have to tweak the dashboard variables i want to import and thatā€™s all good

Just a question, at the start of your script you use the basic auth method for the API ? Canā€™t you create an API Token on your main org and use it to create other orgs ?

Thanks again,
Have a good day

5

Yeah I figured I could at least save some other people from the headache that I got from this and share it.

Each org needs itā€™s own API key, as changes such as changing the home dashboard is a preference change thatā€™s only valid for that org.

A username can still be a global admin so for anything that you do manually you can just log in as normal and change stuff.

So yes, I start with the basic Auth for the admin account to create the org, then I switch to using the API key when possible. I didnā€™t really want to administer the API keys for all these orgs either but this was what I found out to give the best results. This is also why you really want to keep the export of the API key at the end.

6

Ok, so i will just create a user for the API thanks

Update :
Iā€™ve tested the script and it work perfectly, it just need jq as dependency
Forr those interested iā€™ve added the following line at the end to create a user in the organization with basic auth

echo Creating organization user

UserCreation=$(curl --silent -X POST -H ā€˜Accept: application/jsonā€™ -H ā€˜Content-Type: application/jsonā€™ -d ā€œ{"name": "$CompanyLongName", "login":"$CompanyShortName", "password":"$CompanyShortName", "email":"$CompanyShortName@company.tld", "OrgId":$OrgID}ā€ http://$Username:$Password@community.grafana.com/api/admin/users | jq ā€˜.ā€™)

echo $UserCreation

7

Actually jq is used to format the json response to the output. My next step would be to remove the read commands and create the variables by using command line arguments ($1,$2 etc.)

At that point you donā€™t have to see the output anymore, which means you can remove some of the echo commands and you wonā€™t need jq anymore.

The script can then be kicked of by some task from a CRM application or something similar.

2 Likes