Link SAML keycloak to AD roles

I am using grafana linux. I currently have LDAP set up and roles set in AD. Currently the LDAP side of things works perfectly linking permissions. My current issue is when a user logs in with SAML key cloak their permissions do not sync with AD. How do i go about making it so SAML syncs with AD. i ran some logs and this error shows up when i log in with SAML: evel: error msg= “Proxy request failed” err= dial tcp" connect: connection refused.
even though i get a connection refused i can still log in but it gives me the basic viewer permissions when logged in

{auth.proxy]
enabled = true
header_name = X-Forwarded-User
header_property = username
auto_sign_up = true
roles_header = X-Forwarded-Roles

[auth.saml]
enabled = true
**key stuf **
roles_values_none = none
assertion_attribute_role = role
role_values_viewer = Viewer
role_values_Editor = Editor
roles_values_Admin = Admin
assertion_attribute_role = role
assertion_attribute_name = name
assertion_attribute_login = username
assertion_attribute_email = email
assertion_attribute_groups = groups