After connecting to Azure AD Oath i want only memebers of group abc able to connect

I was able to have Grafana connect to Azure AD so we have SSO now.
I only have an issue that it does not work without the sign up function on.
And then i wanted to limit the sign up with
“groupMembershipClaims”: “SecurityGroup, ApplicationGroup”
But this does not work.
Anyone in our organisation with an EntraID can still signup.

Turning the signup function off results in nobody being able to log on that did not log on once when it was on)

How do i limit the signup to only people in a certain Azure AD group?

IMHO you can do that on the Entra level:

Im new to this whole EntraID stuff.
I have added users and groups like the Grafana manual stated.
But it looks like i need to keep the signup function on if i want to be able to logon (for the first time).
So now when people use SSO and are not in the users or groups they get signed up as viewer.
What i would think is that i can turn signup off and let people log in by SSO only.

Do i then need to invite them or prepare something else for them to be able to log on with SSO?