It's been a nightmare learning and getting a working WCF to transfer "web crawling" data.
First it was GoDaddy's server creating a non-traceable error that closed my WCF connection every 15 seconds on the dot after the first connection/call, regardless if I closed the connection after each call. This was solved after a few weeks of exhausting every solution I possibly could when I finally contacted GoDaddy for the resolve. After a Server Migration, this problem disappeared.
After a few weeks of pulling out my hair over the 15 second connection drop, I get stuck with another non-traceable error... the "The underlying connection was closed: The connection was closed unexpectedly.". At this point, I about gave up WCF and went back to ASMX service's.
The error: "The underlying connection was closed: The connection was closed unexpectedly." is pretty generic and the solutions vary, making it difficult to diagnose/fix. Trace logs don't show jack and for some reason the same error would not occur on my local machine, only on the server. The Server Trace logs wouldn't even show the client call when I would receive this error. Like the call never made it to the WCF Service from the Client.... On top of that, the Client's trace log didn't show anything besides a "System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object)" erroring, which didn't help.
I've tried everything online as possible solutions or possible ways to provide a more meaningful trace but without luck. Anyone else run into this, without a Trace log that helps?
tl;dr: The main thing I wanted to get across if you get this error in WCF is to try the Magnifying glass icon / View as XML in the Intelli-Sense type debug window when it errors. It's possible the error will be exposed as it was for me.
At first I thought it was Serialization of the Class Array getting messed up before it hit the wire. The array returns a large number of objects, some nested, so it was very possible. I tested this theory using the Xml.Serializer to Serialize and Deserialize the returned Class Array into a XML document. This worked fine, no errors thrown on the client side. Rule serialization out.
Then I figured, well maybe I don't have one of the settings in my web.Config file set correctly. Nope, everything was maxed out with correct bindings, behaviors, limits, timeouts... all looked good. So I thought, well maybe since I am making multiple calls, it's a concurrency or threading problem. To test this, I only returned 1 set of results per request, instead of an array of the serialized class object. I ran into the error again! So it wasn't concurrency or threading problems.
However, returning a single class object led to the solution. I noticed one object kept failing every time... but this one class object is a container for several strings, integers, and lists(of anotherclass). There usually isn't more than 1000 sub-objects in the main class being returned. It became obvious however that there was an issue with something that is being returned in this object. Going through all the possibilities (usually between 500-1000) without knowing what to look for is kind of a needle-in-a-haystack problem.
Luckily, when the error popped up giving me the "The underlying connection was closed: The connection was closed unexpectedly." I hovered my mouse over the List(Of Class) being sent to the WCF service while the error was still displayed. This brought up a intelli-sense type of debugging window that lets you sift through each Class in the List(of class) while digging down to each sub-object in the class. Since I only returned 1 class in my List(of class), to help narrow things down, and assumed it wasn't the Integer fields causing the problem- I went straight for the String variables being returned. Copy & Pasted them into Notepad, didn't really see anything wrong that I noticed right off the bat, although a few sub-objects raised suspicion. Then by curiosity, I noticed a small magnifying glass with a drop-down arrow next to each of the return Strings inside this intelli-sense type debug window. I clicked on the magnifying glass for each of my suspicions and noticed an option to View the string in XML format. I clicked this and long-and-behold, I got an error on one of the strings that made sense! The error pointed out an un-escaped & symbol mingled with some text. I didn't realize this was a no-no, so I escaped the following 5 characters &, <, >, ", '. I tried again and got a new error, "Only one top level element is allowed in an XML document. Error processing resource"!@!
After doing a little research, it appeared there was something still funky with this object. So I tried wrapping possible problem strings in "
<![CDATA["&"]]>". Got the same "Only one top level element is allowed in an XML document. Error processing resource" error! Turns out there was some CSS slipping through the cracks in the format of {color:#000000;}. For some reason the brackets were causing problems. I'm assuming the reason is because the format of CSS looks a lot like JSON and the Client was seeing it as such. After removing the brackets/CSS, the "Only one top level element is allowed in an XML document. Error processing resource" disappeared and the object successfully transferred.Sorry for the long descriptive answer but I didn't see anyone else with these problems, that didn't throw useful trace information. Hopefully this helps someone else! (I posted here, answering my own question because while it was a simple solution, it was extremely frustrating to solve.)