I have very strange issue, that MSFT support couldn't help to resolve. Let me start with describing use case.
I have a SaaS application. The application collects data from clients connecting to their databases. To link our infrastructure with client ones - we leverage Azure Hybrid Connections.
Problem:
Once in 1-3 month in Production (ONLY) we are seeing our application in "limbo" state: we are loosing connection to some clients (number is random each time, it can be 1-2, maybe 100). This happens upon adding new client (registering new Hybrid Connection) and, as a result, rebooting App Service. When this happens, a Hybrid Connection is shown healthy on both App Service and HCM (for those clients experiencing issues). From an application perspective, it looks like a database server is not available.
One more interesting thing we've caught recently. Our App Service instance hosts 2 applications: API and WebJobs. And fantastically, we see that the particular client's database is reachable from WebJobs app, but is not available from API (both applications use the same connection string, taken from a single config).
Does someone know the mechanics of this communication? Specifically from App Service side?
Unfortunately, MSFT don't log anything related to Hybrid Connections and couldn't explain how all that stuff works in deep.