.Net Core app stuck hanging/loading on startup with out of process hosting model

Question

I have a 3.1 net core web api which I had to switch from in process to out of process hosting model because my hosting environment has only one application pool and I have two applications running in that pool. Because of the in process restriction related to the application pool I had to switch both apps to out of process.

The web api gets stuck/loading/hanging on startup both on my local environment (when running it from vs studio) and on the hosting environment. The only difference is that on the hosted environment, after a while, I get a "HTTP Error 502.5 ANCM Out-Of-Process Startup Failure" type error.

After doing some research I've found out that most of this errors are caused by some code in the "Program.cs" class. And indeed it was. After commenting out the following code portion it works.

    public static IHostBuilder CreateHostBuilder(string[] args, IConfiguration configuration) =>
        Host.CreateDefaultBuilder(args)
            //.ConfigureAppConfiguration(builder =>
            //{
            //    builder.Sources.Clear();
            //    builder.AddConfiguration(configuration);
            //})
            .UseSerilog()                
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });

What happens is that I clear the default configurations so that I can only insert my explicitly defined ones in the "Main" method (at least this is what I think I wanted to do at the time I wrote that code).

            var config = new ConfigurationBuilder()
            .AddJsonFile("appsettings.json")
            .AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json")
            .AddUserSecrets(Assembly.GetExecutingAssembly())
            .AddEnvironmentVariables()                
            .Build();

Somehow, clearing the default configurations causes the app to not work in the out of process hosting model and I really wanted to know why.