Users removed from Org after new login

We have setup SSO login using AzureAD and the actual login works great. However, we are having issues with the user access to Organizations. Using the Server Admin we are adding a user to an Organization which is accepted and we can even switch to that Organization using the signed in user. However, if the user logs out and then in again, the user is then removed from the Organization. We are unable to understand why this is happening.

I can also add that we are getting the correct role assigned to the logged in user as per the setup in AzureAD.

We are using the free version of Grafana. Both AzureAD and Basic Auth is enabled. Using the latest version of Grafana.

Could this be a bug? Or are we doing something wrong.

Any ideas are appreciated.