Isolating assemblies to load and execute plugin applications where the plugged-in object only use the assemblies close to it

Question

I am trying to load plugin applications at runtime and execute the code in isolation, with the option of unloading all assemblies and resources when the main application is done with the plugin.

The AppDomain gets the job done but it has a lot of baggage especially when it comes to transferring data to the secondary AppDomain. I have also found a few issues with keeping the AppDomain alive for a long time.

I have tried AssemblyLoadContext and for the most part, it gets the job done except for when the assembly exists in both the plugin and the main application.

static class PluginManager {
    public static IPlugin CreatePluginInstance(ILogger extensionLogger)
    {
      ...
      services.AddSingleton<ILogger>(logger => extensionLogger);
      return (IPlugin)ActivatorUtilities.CreateInstance(services.BuildServiceProvider(), {typeOfPlugin});
    }
}

The plugin class:

public class GoogleDocumentService: IPlugin
{
        public GoogleDocumentService(ILogger logger)
            : this(new PluginLogger(logger))
        {
        }

        public GoogleDocumentService(IPluginLogger logger)
        {
         ....
        }
}

Calling the PluginManager.CreatePluginInstance method when the assemblies are loaded in the default context returns a Plugin object.

When I call this method with the assemblies loaded to my custom load context, I get "System.InvalidOperationException: 'Multiple constructors accepting all given argument types have been found in type 'Plugin.GoogleDocumentService'. There should only be one applicable constructor.'"

This could be due to two different versions of the Logging assembly in the default and custom context. But the AssemblyLoadContext was built to aid in this exact usecase, so am I missing something here?