WebJobs dashboard broken for 2.0.0 beta C# SDK

Question

I am faced with a problem similar to this question in that the function list is showing duplicates and exclamation marks. In difference to that question, the About page is showing a long list of indexer failures, all of them with the message "The remote server returned an error: (403) Forbidden." with a stack trace showing that the error originates from the Azure Storage client.

The thing is, the webjob itself manages to connect to the storage account just fine. Also, if I copy the connection string from the AzureWebJobsDashboard setting into Azure Storage Explorer, it connects to the account just fine.

It occurred to me that this might have to do with the fact that I am using the beta version of the SDK and that it might be incompatible with the dashboard version that Azure is running, over which I have no control.

Unfortunately, I am unable to downgrade to a stable version, since we are using the NotificationHubs extension, which is based on the beta SDK.

Has anyone faced the same problem and come up with a solution that doesn't involve dropping the Notification Hubs support?

Update

It occurs to me that besides upgrading the WebJobs SDK version, I also went from having a single WebJob project in the solution to 2 WebJobs. Could the problem be related to this? I have placed both jobs in the webjobs-list.json file under Properties for the WebApi project:

{
  "$schema": "http://schemastore.org/schemas/json/webjobs-list.json",
  "WebJobs": [
    {
        "filePath": "../import/TeamString.Service.Import.csproj"
    },
    {
        "filePath": "../push-notifier/TeamString.Service.PushNotifier.csproj"
    }
  ]
}

Update 2

First of all, as mentioned previously, I have 2 continuous WebJobs deployed within the same App Service webapp:

1. Importer - this one only depends on the basic Microsoft.Azure.WebJobs NuGet package. No extensions and no service bus.
2. Push-Notifier - this one uses ServiceBus and Notification Hubs through WebJobs, so it depends on Microsoft.Azure.WebJobs.ServiceBus and Microsoft.Azure.WebJobs.Extensions.NotificationHubs.

I have now had some time to perform some additional experiments:

1. Updated WebJob NuGet packages to latest beta. This did not fix the constant indexer errors. Now running on the following versions:
   • WindowsAzure.Storage: 7.2.1
   • Microsoft.Web.WebJobs.Publish: 1.0.12
   • Microsoft.Azure.WebJobs.Core: 2.0.0-beta2-10491
   • Microsoft.Azure.WebJobs: 2.0.0-beta2-10491
   • Microsoft.Azure.WebJobs.ServiceBus: 2.0.0-beta2-10491
   • Microsoft.Azure.WebJobs.Extensions: 2.0.0-beta2
   • Microsoft.Azure.WebJobs.Script.Extensibility: 1.0.0-beta2-10690
   • Microsoft.Azure.WebJobs.Extensions.NotificationHubs: 1.0.0-beta2
2. Deleted the importer WebJob via the Azure Portal and stopped the push-notifier. Then deleted all WebJob log objects in the storage account. Result: indexer started spamming 404 Not Found errors instead of 403 Forbidden.
3. Started the push-notifier again and gave it some work to do. Result: indexer back to spamming 403 Forbidden.
4. Redeployed solution to get importer back. Then deleted push-notifier WebJob via portal and stopped importer. Deleted all WebJob log objects once again. Result: no indexer errors!?!
5. Started importer and gave it some work to do. Result: still no indexer errors!

So it's definitely something about the push-notifier WebJob that triggers this bug.

Answer 1

It occurred to me that this might have to do with the fact that I am using the beta version of the SDK and that it might be incompatible with the dashboard version that Azure is running

I use the WebJobs template to create a new web job project and install Microsoft.Azure.WebJobs.Extensions v2.0.0-beta2, and I find if I provide AzureWebJobsDashboard and AzureWebJobsStorage connection strings with correct values, the web job could run fine.

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.KeyVault.Core" version="1.0.0" targetFramework="net45" />
  <package id="Microsoft.Azure.WebJobs" version="2.0.0-beta2" targetFramework="net45" />
  <package id="Microsoft.Azure.WebJobs.Core" version="2.0.0-beta2" targetFramework="net45" />
  <package id="Microsoft.Azure.WebJobs.Extensions" version="2.0.0-beta2" targetFramework="net45" />
  <package id="Microsoft.Azure.WebJobs.Script.Extensibility" version="1.0.0-beta2" targetFramework="net45" />
  <package id="Microsoft.Data.Edm" version="5.6.4" targetFramework="net45" />
  <package id="Microsoft.Data.OData" version="5.6.4" targetFramework="net45" />
  <package id="Microsoft.Data.Services.Client" version="5.6.4" targetFramework="net45" />
  <package id="Microsoft.Tpl.Dataflow" version="4.5.24" targetFramework="net45" />
  <package id="Microsoft.Web.WebJobs.Publish" version="1.0.12" targetFramework="net45" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.3" targetFramework="net45" />
  <package id="ncrontab" version="3.2.0" targetFramework="net45" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net45" />
  <package id="System.Spatial" version="5.6.4" targetFramework="net45" />
  <package id="WindowsAzure.Storage" version="7.2.1" targetFramework="net45" />
</packages>

Please make sure you are setting a connection string named AzureWebJobsDashboard with the following format value DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY. And please check whether you specify another value for AzureWebJobsDashboard under Application settings blade in Azure portal (it could override existing settings).

Besides, please try to create a new web job project and install Microsoft.Azure.WebJobs.Extensions v2.0.0-beta2, without using NotificationHubs extension, to check whether NotificationHubs extension cause the issue.