Why does Azure CloudConfigurationManager.GetSetting return null

Question

I have a cloud service web role project that runs locally in the emulator, but does not run when deployed. The error given is as follows:

[ArgumentNullException: Value cannot be null.
Parameter name: connectionString]
   Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(String connectionString) in e:\projects\azure-sdk-for-net\microsoft-azure-api\Services\Storage\Lib\Common\CloudStorageAccount.cs:344
   Candor.WindowsAzure.Storage.Table.CloudTableProxy`1.GetTable() in c:\Users\micha_000\Documents\GitHub\candor-common\Candor.WindowsAzure\Storage\Table\CloudTableProxy.cs:66
   Candor.WindowsAzure.Storage.Table.CloudTableProxy`1.Get(String partitionKey, String rowKey) in c:\Users\micha_000\Documents\GitHub\candor-common\Candor.WindowsAzure\Storage\Table\CloudTableProxy.cs:117
   Candor.WindowsAzure.Logging.Common.Table.CloudTableLogger.get_Configuration() +218
   Candor.WindowsAzure.Logging.Common.Table.CloudTableLogger.get_IsInfoEnabled() +9
   Common.Logging.Factory.AbstractLogger.Info(Object message) in c:\_oss\common-logging\src\Common.Logging.Core\Logging\Factory\AbstractLogger.cs:503
   Candor.Configuration.Provider.ProviderCollection`1.SetActiveProvider(T provider) in c:\Users\micha_000\Documents\GitHub\candor-common\Candor\Configuration\Provider\ProviderCollection.cs:169
   Candor.Configuration.Provider.ProviderResolver`1.AppendActive(T provider) in c:\Users\micha_000\Documents\GitHub\candor-common\Candor\Configuration\Provider\ProviderResolver.cs:77
   SHM.PublicMvcWeb.App_Start.ProviderBootstrapper.InitProviders() in c:\Users\micha_000\Documents\Git-Repos\shm-main\SHM.PublicMvcWeb\App_Start\ProviderBootstrapper.cs:23
   SHM.PublicMvcWeb.App_Start.ProviderBootstrapper.PostStartup() in c:\Users\micha_000\Documents\Git-Repos\shm-main\SHM.PublicMvcWeb\App_Start\ProviderBootstrapper.cs:18

You can see the code for the relevant lines shown in the stack trace on github.

https://github.com/michael-lang/candor-common/

The lines leading to the error in CloudTableProxy are:

            if (String.IsNullOrWhiteSpace(_connectionName))
                throw new InvalidOperationException("The Cloud ConnectionName has not been configured.");
            if (_account == null)
                _account = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting(_connectionName));

Since it does not throw the custom error "The cloud connectionName has not been configured.", that tells me that _connectionName passed into CloudConfigurationManager.GetSetting is not null. So CloudConfigurationManager is the call returning null for the supplied name. Given that this name does return a value when running locally, I am not sure why it isn't found when deployed to my cloud web role. If the connection name was entered as a typo, then it wouldn't work locally either. To be sure, here is my common logging configuration naming the connection to be used:

  <common>
    <logging>
      <factoryAdapter type="Candor.WindowsAzure.Logging.Common.Table.CloudTableLoggerFactoryAdapter, Candor.WindowsAzure.Logging.Common">
        <arg key="connectionName" value="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" />
      </factoryAdapter>
    </logging>
  </common>

I also have an entirely different solution with a cloud service web role using the same Candor.Common components, so I don't believe they are the problem.

I am using the same laptop to develop both solutions, and so both are using the Azure 2.2 Tools version. Inspecting the properties of the Cloud Service project in each solution verifies this.

I inspected every component and NuGet package reference to be sure there are no version mismatches through the solution. The same components are set to copy-local=true between the working solution and this non-working solution. They also have the same binding redirects. Although this was my biggest issue, now resolved, before running into this connection issue.

The non-working service deployment configuration:

<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="SHM.AzureService.PublicMvcWeb" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="4" osVersion="*" schemaVersion="2013-10.2.2">
  <Role name="SHM.PublicMvcWeb">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)" />
      <Setting name="DefaultTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)" />
      <Setting name="UserTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)" />
      <Setting name="UserSaltTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)" />
      <Setting name="UserAuditTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)" />
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

I also recently tried putting these same connection names in the web application's appsettings. This was in response to a comment 3 years ago that their code was executing before RoleEnvironment.OnStart. But this attempt still results in the same error.

Here is what the attempt looked like:

  <appSettings>
    <add key="webpages:Version" value="3.0.0.0" />
    <add key="webpages:Enabled" value="false" />
    <add key="ClientValidationEnabled" value="true" />
    <add key="UnobtrusiveJavaScriptEnabled" value="true" />
    <add key="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)"/>
    <add key="DefaultTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)"/>
    <add key="UserTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)"/>
    <add key="UserSaltTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)"/>
    <add key="UserAuditTableConnection" value="DefaultEndpointsProtocol=https;AccountName=(redacted);AccountKey=(redacted)"/>
  </appSettings>

This code in error does run on start of the application using WebActivator 2.0.6, wheras my working solution uses WebActivator 2.0.4. But the release notes of WebActivator only shows compilation change from Debug to "Retail" and a license change between the two versions, and those changes were a year ago.

In case timing was the issue, I also tried adding a Thread.Sleep(1000) just before the call in error, and it didn't work either, so I am going to remove it.

using System.Threading;
using Candor.Configuration.Provider;
using Candor.Security;
using Candor.Security.Cryptography;
using Candor.Security.Web;

[assembly: WebActivatorEx.PreApplicationStartMethod(typeof(StopHarassingMe.PublicMvcWeb.App_Start.ProviderBootstrapper), "PreStartup")]
[assembly: WebActivatorEx.PostApplicationStartMethod(typeof(StopHarassingMe.PublicMvcWeb.App_Start.ProviderBootstrapper), "PostStartup")]

namespace SHM.PublicMvcWeb.App_Start
{
    public class ProviderBootstrapper
    {
        public static void PreStartup()
        {
        }
        public static void PostStartup()
        {
            Thread.Sleep(1000); //let RoleEnvironment finish startup first or connectionstrings are not available.
            InitProviders();
        }

        private static void InitProviders()
        {
            ProviderResolver<HashProvider>.Configure()
                .AppendActive(new SHA2HashProvider("sha2") { IsObsolete = false, SaltModifier = "" });
        }
    }
}

This is the first provider configured in both the working and non-working solutions.

Here is a list of the packages used by the web application in error:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Antlr" version="3.5.0.2" targetFramework="net45" />
  <package id="Candor.Core" version="1.4.1.0" targetFramework="net45" />
  <package id="Candor.jQuery.AutoAsync" version="4.0.0.50129" targetFramework="net45" />
  <package id="Candor.Security" version="2.4.0.0" targetFramework="net451" />
  <package id="Candor.Security.AzureStorageProvider" version="2.3.2.0" targetFramework="net451" />
  <package id="Candor.Web.Mvc" version="1.0.3.0" targetFramework="net451" />
  <package id="Candor.Web.Mvc.ErrorHandler" version="1.0.0.0" targetFramework="net45" />
  <package id="Candor.Web.Mvc.Security" version="2.1.0.0" targetFramework="net45" />
  <package id="Candor.WindowsAzure" version="1.3.0.0" targetFramework="net451" />
  <package id="Candor.WindowsAzure.Logging.Common" version="1.1.1.0" targetFramework="net451" />
  <package id="Common.Logging" version="2.2.0" targetFramework="net45" />
  <package id="Common.Logging.Core" version="2.2.0" targetFramework="net45" />
  <package id="form2js" version="1.0.0.30224" targetFramework="net45" />
  <package id="jQuery" version="2.1.4" targetFramework="net451" />
  <package id="jQuery.UI.Combined" version="1.8.20.1" targetFramework="net45" />
  <package id="jQuery.Validation" version="1.13.1" targetFramework="net45" />
  <package id="MarkdownSharp" version="1.13.0.0" targetFramework="net451" />
  <package id="Microsoft.AspNet.Mvc" version="5.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.Razor" version="3.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.Web.Optimization" version="1.1.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi" version="5.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Core" version="5.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.HelpPage" version="5.2.3" targetFramework="net451" />
  <package id="Microsoft.AspNet.WebApi.WebHost" version="5.2.3" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebPages" version="3.2.3" targetFramework="net45" />
  <package id="Microsoft.Data.Edm" version="5.6.0" targetFramework="net451" />
  <package id="Microsoft.Data.OData" version="5.6.0" targetFramework="net451" />
  <package id="Microsoft.jQuery.Unobtrusive.Validation" version="3.2.3" targetFramework="net45" />
  <package id="Microsoft.Web.Infrastructure" version="1.0.0.0" targetFramework="net45" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.1.0" targetFramework="net45" />
  <package id="Modernizr" version="2.8.3" targetFramework="net45" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net45" />
  <package id="Respond" version="1.4.2" targetFramework="net45" />
  <package id="RestSharp" version="105.0.1" targetFramework="net451" />
  <package id="System.Spatial" version="5.6.0" targetFramework="net451" />
  <package id="T4MVC" version="3.7.4" targetFramework="net45" />
  <package id="T4MVCExtensions" version="3.7.4" targetFramework="net45" />
  <package id="Twilio" version="4.0.3" targetFramework="net451" />
  <package id="Twilio.Mvc" version="3.1.15" targetFramework="net451" />
  <package id="Twilio.TwiML" version="3.3.6" targetFramework="net451" />
  <package id="WebActivatorEx" version="2.0.6" targetFramework="net451" />
  <package id="WebGrease" version="1.6.0" targetFramework="net45" />
  <package id="WindowsAzure.Storage" version="2.1.0.3" targetFramework="net45" />
</packages>

There are a couple package differences between my working and non-working solution, but I didn't think these were applicable to the error. I went from MVC 5.0 to 5.2.3, upgraded javascript related packages, and Candor.WindowsAzure from 1.2.10 to 1.3. That may seem relevant, but the only code change was to replace the implementation of a batch update method.

WORKING SOLUTION PROJECT PACKAGES:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Antlr" version="3.5.0.2" targetFramework="net45" />
  <package id="Candor.Core" version="1.4.1.0" targetFramework="net451" />
  <package id="Candor.jQuery.AutoAsync" version="3.5.0.40210" targetFramework="net451" />
  <package id="Candor.Security" version="2.4.0.0" targetFramework="net451" />
  <package id="Candor.Security.AzureStorageProvider" version="2.3.2.0" targetFramework="net451" />
  <package id="Candor.Web.Mvc" version="1.0.3.0" targetFramework="net451" />
  <package id="Candor.Web.Mvc.ErrorHandler" version="1.0.0.0" targetFramework="net45" />
  <package id="Candor.Web.Mvc.Security" version="2.1.0.0" targetFramework="net45" />
  <package id="Candor.WindowsAzure" version="1.2.10.0" targetFramework="net451" />
  <package id="Candor.WindowsAzure.Logging.Common" version="1.1.1.0" targetFramework="net451" />
  <package id="Common.Logging" version="2.2.0" targetFramework="net451" />
  <package id="Common.Logging.Core" version="2.2.0" targetFramework="net451" />
  <package id="form2js" version="1.0.0.30224" targetFramework="net45" />
  <package id="jQuery" version="2.0.3" targetFramework="net45" />
  <package id="jQuery.UI.Combined" version="1.10.3" targetFramework="net45" />
  <package id="jQuery.Validation" version="1.11.1" targetFramework="net45" />
  <package id="MarkdownSharp" version="1.13.0.0" targetFramework="net451" />
  <package id="Microsoft.AspNet.Mvc" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.Razor" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.Web.Optimization" version="1.1.2" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Core" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.HelpPage" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.OData" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.WebHost" version="5.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebPages" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebPages.Data" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebPages.WebData" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.Data.Edm" version="5.6.0" targetFramework="net45" />
  <package id="Microsoft.Data.OData" version="5.6.0" targetFramework="net45" />
  <package id="Microsoft.jQuery.Unobtrusive.Ajax" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.jQuery.Unobtrusive.Validation" version="3.0.0" targetFramework="net45" />
  <package id="Microsoft.Web.Infrastructure" version="1.0.0.0" targetFramework="net45" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.1.0" targetFramework="net45" />
  <package id="Modernizr" version="2.7.1" targetFramework="net45" />
  <package id="Newtonsoft.Json" version="5.0.8" targetFramework="net45" />
  <package id="SlidesJS" version="3.0.4" targetFramework="net451" />
  <package id="System.Spatial" version="5.6.0" targetFramework="net45" />
  <package id="T4MVC" version="3.7.4" targetFramework="net45" />
  <package id="T4MVCExtensions" version="3.7.4" targetFramework="net45" />
  <package id="WebActivatorEx" version="2.0.4" targetFramework="net45" />
  <package id="WebGrease" version="1.6.0" targetFramework="net451" />
  <package id="WindowsAzure.Storage" version="2.1.0.3" targetFramework="net45" />
</packages>

Answer 1

I tried to deploy as a website just to see what the difference would be. The table connection strings were in appSettings from the earlier attempts to get past the timing issue, so it made sense to try the websites deployment option (not a cloud web role).

The error from this attempt was

[FileNotFoundException: Could not load file or assembly 'Microsoft.WindowsAzure.ServiceRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.]

[FileNotFoundException: Could not load file or assembly 'Microsoft.WindowsAzure.ServiceRuntime, Version=2.2.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.]
   Candor.WindowsAzure.Logging.Common.Table.CloudTableLogger.WriteInternal(LogLevel level, Object message, Exception exception) +0
   Common.Logging.Factory.AbstractLogger.Info(Object message) in c:\_oss\common-logging\src\Common.Logging.Core\Logging\Factory\AbstractLogger.cs:504
   Candor.Configuration.Provider.ProviderCollection`1.SetActiveProvider(T provider) in c:\Users\micha_000\Documents\GitHub\candor-common\Candor\Configuration\Provider\ProviderCollection.cs:166
   Candor.Configuration.Provider.ProviderResolver`1.AppendActive(T provider) in c:\Users\micha_000\Documents\GitHub\candor-common\Candor\Configuration\Provider\ProviderResolver.cs:75

This was odd, since the other working solution web role deployed to the cloud now, has that reference set as CopyLocal=false. But when I set it to true in this non-working solution, it started to work in the website deployment.

Next I redeployed the cloud web role with this component set as CopyLocal=True and it now works. I don't know why this is required in one project deployment and not the other, While both projects required the other azure components to be CopyLocal=true.

There is still one lingering problem with startup speed, but I'll leave that for another question, if I don't figure it out. For some reason this smaller project goes in a busy/recycle loop after it starts, but the page loads now and then. It is quite odd since it has fewer dependencies initialized on startup, that is providers instantiated that write a single log entry each.

Answer 2

Straight point to the question is - CloudConfigurationManager.GetSetting will read the given config key from Cloud Project settings if defined or else will fall back to actual project App Or Web config. So if you make sure that it exists at one of the places, it should work fine. As you confirmed that it is working locally, it should work exactly the same way even after deployment.

Other case that can happen is when you are having multiple config files for local and cloud deployment environments and that config could be missing from your cloud configuration file that you used to deploy into cloud.

Answer 3

In my case, I used SlowCheetah (NuGet package) to transform App.config files. In Debug it did not use the Transformation. Using the Debug settings in the "root"-config file solved my problem.