Azure Oauth gives Viewer role to whole organization. How can I restrict

I setup Grafana with Azure Oauth.

I created an App Registration and Enterprise Registration on Azure.
I assigned a specific AD group to the Enterprise Registration and gave it the Grafana Admin role.

Now I notice that all the users in my organization can use Azure Oauth to log into Grafana and with a Viewer role.

IS this a feature or a bug? How can I prevent this?