How can I improve my way of expanding a list of recipients (AD users and groups) into individual recipients in .NET?

Question

I'm writing a middleware in C# to run in a Windows Server. We have an internal email mass notification system that used to use Exchange Web Services but we've migrated to 365 and quickly discovered that there is a 10000/mailbox email limit per day. This app hit that limit day 1. We have an SMTP relay that can handle this no problem, but it's just that, a relay. Whatever comes in, goes out.
The middleware I'm writing receives an HTTP request from the email notification app with a list of recipients, builds a MailMessage object with the recipients and body, and sends it to the relay.

The receiving and sending parts are all set, but I ran into a problem:

If the list includes user A, and a distribution group that has user A as a member, user A will get the email twice, since I have no visibility into the group members. In our first trial some users were getting emails 14 times.

So, this led me down the path of how to resolve these emails to get the individual members to avoid sending them more than one copy of the message. For individual members, I'm using a hashset to store the recipients to avoid having duplicates. And for the groups, I'm checking against a list of AD groups I get in the background to see if they are a group or an individual, this works, but I'm having some SERIOUS issues with performance.

I came up with using AD to check each recipient to see if they are groups, and if they are, recursively extract the members, put them in a hashset, and use that to add to the message recipients. If the recipient is a group, these two methods are called. I first get the group by email then I get the members. I understand I'm mixing two libraries here, but it works. I'm sure it can be shortened, but I don't know if it will reduce the search time in half or more.

static GroupPrincipal FindGroupByEmail(string address)
{
    DirectoryEntry exchangeOU = new DirectoryEntry("LDAP://The specific OU where these groups are");
    DirectorySearcher searcher = new DirectorySearcher(exchangeOU);
    searcher.Filter = "(&(objectClass=group)(mail=" + address + "))";
    SearchResult result = searcher.FindOne();

    if (result != null)
    {
        DirectoryEntry groupEntry = result.GetDirectoryEntry();
        string groupDistName = groupEntry.Properties["distinguishedName"].Value.ToString();
        if (groupDistName != null)
        {
            GroupPrincipal group = GroupPrincipal.FindByIdentity(new PrincipalContext(ContextType.Domain), IdentityType.DistinguishedName, groupDistName);
            return group;
        }
    }
    return null;
}
static HashSet<string> GetMembers(GroupPrincipal group)
{
    HashSet<string> membersEmails = new HashSet<string>();

    foreach (Principal member in group.Members)
    {
        if (member is UserPrincipal user)
        {
        if (user.EmailAddress != null)
        {
            membersEmails.Add(user.EmailAddress.ToLower());
        }
        }
        else if (member is GroupPrincipal nestedGroup)
        {
            membersEmails.UnionWith(GetMembers(nestedGroup));
        }
    }
    return membersEmails;
}

This was incredibly slow. between 15 to 60 seconds per group depending on count of members. Some of these messages contain 25 groups with nested groups inside, so I end up having to wait 30 min for the resolution of the group members.

We also have 365 linked to our AD, but not all the groups are synced yet, so I can't reliably use it to query the groups.

I'm targeting .NET 4.7.2 which is what's available on the server but I'm working on getting .NET Core 7.0 installed if that makes things any easier.

There must be a better way of doing this, what else could one do to resolve these distribution groups?