AWS IAM Policy for restricting access to particular groups in AWS SSO

Question

I have multiple groups in AWS SSO as in image below. Individual groups contains users of individual customer.

AWS SSO Groups

I want to create a create a custom Policy in IAM. The policy should have the below definition –

a. Allow Add User in AWS SSO. (While adding user in AWS SSO, IAM user should be able to assign AWS SSO group to the AWS SSO user).

b. Edit User / Delete User in AWS SSO (Restricted Access. IAM user with this policy can perform these actions for users present in his/her customer group in AWS SSO). Eg. If IAM user is for ‘Test_Customer_1’, then IAM user should be able to edit/delete users in AWS SSO who are present in group ‘Users_Test_Customer_1’ only and not users of other groups.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sso-directory:ListGroupsForUser",
        "sso-directory:DisableUser",
        "sso-directory:EnableUser",
        "sso-directory:SearchGroups",
        "sso-directory:DeleteGroup",
        "sso-directory:AddMemberToGroup",
        "sso-directory:DescribeDirectory",
        "sso-directory:UpdateUser",
        "sso-directory:ListMembersInGroup",
        "sso-directory:CreateUser",
        "sso-directory:DescribeGroups",
        "sso-directory:SearchUsers",
        "sso:ListDirectoryAssociations",
        "sso-directory:RemoveMemberFromGroup",
        "sso-directory:DeleteUser",
        "sso-directory:DescribeUsers",
        "sso-directory:UpdateGroup",
        "sso-directory:CreateGroup"
      ],
      "Resource": "*"
    }
  ]
}

With the custom policy defined above, IAM users with this policy will be able to manipulate users in all groups present in AWS SSO.

Is there a way to update the policy to restrict IAM user to update AWS SSO users for a particular AWS SSO group and not all groups.