Why is my webjob terminating without throwing an exception?

Question

My azure webjob appears to be terminating without throwing an exception and I'm lost.

My web job is run on-demand (or scheduled) and has a dependency on my web site DLL (and MVC app). It calls into it to do most of the work, which includes working with an entity frameworks database and making REST calls to several other sites. Most of the work is done asynchronously. Most of the code used to do this work is also called from other parts of the site without problem, and it goes without saying that the web job works flawlessly when run locally.

The web job terminates and doesn't seem to throw an exception when it does and it doesn't seem to be possible to debug a web that's not of the continuously run variety (?). Therefor, my debugging has mostly been of the Console.WriteLine variety. Because of that and the asynchronisity, I haven't been able to nail down exactly where it's crashing - I thought it was while accessing the database, but after mucking with it, the database access started working.. ugh. My next best guess it that it dies during an await or other async plumbing. It does, however, crash within two try/catch blocks that have finallys that log results to redis and azure storage. None of that happens. I can not figure out, or imagine, how this process is crashing without hitting any exception handlers.. ?

Anyone had this problem with an azure webjob? Any idea what I should be looking for or any tips for debugging this?

Thanks!

Answer 1

I figured it out! One of the many things happening asynchronously was the creation of a certificate. I traced it down to this:

signedCert = new X509Certificate2(cert, "notasecret", X509KeyStorageFlags.Exportable);

This code works fine when called from my azure website or my tests, but kills the webjob process completely without throwing an exception! For example, the WriteLine in the exception handler below never gets called:

X509Certificate2 signedCert;
try
{
    signedCert = new X509Certificate2(cert, "notasecret", X509KeyStorageFlags.Exportable);
}
catch (Exception ex)
{
    // We never get here! Argh!
    Console.WriteLine("Exception converting cert: " + ex);                
    throw;
}

Extremely time consuming and frustrating. Unlike the diagnosis, the fix is simple:

signedCert = new X509Certificate2(
    cert, 
    "notasecret", 
    X509KeyStorageFlags.Exportable | 
    X509KeyStorageFlags.MachineKeySet |
    X509KeyStorageFlags.PersistKeySet);