Sitecore 8.0 programmatically adding contacts to a list

Question

I’m running into is with programmatically adding and removing contacts from a list in Sitecore. The code I’m using to add, is as follows:

public void AddContactsToList(Sitecore.ListManagement.ContentSearch.Model.ContactList contactList, List<Sitecore.ListManagement.ContentSearch.Model.ContactData> contactsToAdd)
{
    ContactListManager listManager = Sitecore.Configuration.Factory.CreateObject("contactListManager", false) as ContactListManager;

    using (new SecurityDisabler())
    {
        listManager.AssociateContacts(contactList, contactsToAdd);
    }
}

Essentially I have a ContentList instance and using a ContactListManager I call AssociateContacts to add contacts to the list. For removing contacts I have a similar method that calls RemoveContacts instead.

This all works fine until an email is sent, at which point I run into scenarios where the AssociateContacts method doesn’t error at all, and the log files look as normal, but a contact that was recently sent an email no longer gets added or removed from lists. After a few hours all is ok again.

I've tried a different solution described here (Sitecore 8 EXM add a contact to list from listmanager) but that seems to make things worse and I get the issue directly after adding a removing a contact. On the third attempt AddRecipient method errors with a TryLoadContact exception.