mySQL connection times out on Windows Azure

Question

We have a Windows Azure Application (Lokad.CQRS architecture) that integrates with multiple 3rd party systems. One of these integrations takes place on mySQL. Everything was going good, but lately we've got one mySQL database that Windows Azure application can't connect to, resulting in timeout.

Here's the relevant stack trace:

System.TimeoutException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ---> 
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. --->
System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at MyNetworkStream.HandleOrRethrowException(Exception e)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32  offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.Open()
at MySql.Data.MySqlClient.Driver.Open()
at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
at MySql.Data.MySqlClient.MySqlConnection.Open()

Important highlights:

• target database is accessible from Windows Azure (used port forwarding to verify while connecting via the MySQL Workbench);
• integration with the target DB works properly from the Local Azure Dev Fabric;
• Problem on Windows Azure happens for booth pooled and non-pooled connections.
• TCP/IP, usual port, no SSL;
• connection timeout for this operation is large enough - 5 seconds and it continuously fails (but always succeeds from non-Azure machine located not far away from the NE DC)

We have Oracle, MS SQL, mySQL and PostgreSQL connectivity working from Lokad.CQRS over the Windows Azure in various setups and configurations (Linux/Windows-hosted database servers). But this specific mySQL case is just completely confusing me.

Any guesses on what might be causing the problem?

Answer 1

I just pinged you to ask you a bunch of questions as we'd like to trace this issue and help make sure we have no problems with oss apps running on Windows Azure. Let me know if you do not see it.

Jas Sandhu, @jassand Interoperability Strategy Team http://interoperabilitybridges.com http://blogs.msdn.com/interoperability

Answer 2

Solution:

For some reason Azure Production Fabric was adding too much latency to connection to the target database, it was always timing out. Same connection from the machine nearby in Europe never broke the connection timeout.

Increasing connection timeouts for this specific case (by 5x) did solve the case, without adding too much risk for the rest of the distributed system.

Probably the problem was in the network configuration at NE DC, faulty router or whatever.