You should be able to achieve this setup by creating a 'local' Group_RBACAdmins role for each group with the relevant authorizations such as
With these 'RBAC' roles the Group_Full_Access, Group_Read_Access etc roles can be created for each group by the group leaders. Additional roles and users can be added/created as desired (if the users already exist then permissions on the user object will need to be set to allow the Group_RBACAdmin access to these users)
Remember for server access the role must have Server.* (if full access is required) and the server object permissions must be set to allow access to the role.