I was working with a old ASP website that relied heavily on WCF. We would see errors firing on the client side but never on the server side. We started seeing these errors logged about 20-30 times a day:
Number -2147012866: The connection with the server was terminated abnormally
Perfectly descriptive asp error. I tried correlating the times of the errors to other system logs, event logs, performance logs, IIS recycles, db performance, etc. I could find anything that correlated. That basically let me to the belief that the WCF service wasn’t erroring out and it completed the operation normally.
After about 2 days of research, digging the bowls of the internet, and some deduction I narrowed it down to HTTP keep alives. You might ask how the heck does that matter?
First an explanation on what an HTTP Keep Alive is:
Most Web browsers request that the server keep the client connection open while the server sends multiple elements (.htm files and .gif or .jpeg files) to the client. Keeping the client connection open in this way is referred to as an HTTP Keep-Alive. Keep-Alive is an HTTP 1.1 specification that improves server performance by not having the normal handshake for every object. This connection is open for a certain amount of seconds after the first request, then closed on the server side.
When the ASP pages were calling the WCF service it was opening the connection, the server kept that connection open for 120 seconds (IIS6 default). The next time a call was made it would use that same open connection. Here is where my deduction came in… My theory was that executing a request at the same moment the server is closing the connection causes this “terminated abnormally” message.
In our code we use MSXML2.ServerXMLHTTP as the helper library that does all our SOAP calls for us. We used it because the regular XMLHTTP wasn’t threadsafe. It works great with one exception… it has no option to turn off the use of keep alives. After digging the internet and reading through some really crappy documentation on Microsoft’s part I came across a different library that comes with windows: WinHttp.WinHttpRequest.5.1.
The WinHttp is threadsafe and has almost the same interface as the MSMXL2 library but allows you a little more flexibility. Specifically:
xmlhttp.Option(WinHttpRequestOption_EnableHttp1_1) = False
Basically that tells this soap request to use the HTTP 1.0 protocol. The HTTP 1.0 protocol doesn’t support keep alives. In essence turning it off.
Now you’re thinking… Wouldn’t turning off keep alives hurt performance? Your right it does. It takes about 3-5 milliseconds to do the http handshake. Add that onto every soap call our pages make it adds up. But would you rather have failed connections or 3 milliseconds longer? Yes, I’ll take reliable messaging.
I couldn’t replicate the initial issue in the test environment at all. After moving it to production… wouldn’t ya know… no errors all day. Bingo!
After realizing errors wern’t showing up in the ASP pages I started digging through our .NET site logs. Look what I found about 10-15 times a day:
System.ServiceModel.CommunicationException: An error (The request was aborted: The request was canceled.) occurred while transmitting data over the HTTP channel. ---> System.Net.WebException: The request was aborted: The request was canceled. at System.Net.HttpWebRequest.GetResponse()
Doesn’t that look familiar? Yup. After googleing this error I came across something similar to the keep alive issue. The fix for the ASP.net pages is a little different. With WCF you have to change the binding from “BasicHttpBinding” to “CustomBinding”. Once you have that you need to set these flags on the CustomBinding:
After digging in a little bit more about WCF and keep alives, Microsoft even recommends turning this off in a load-balance scenario. I wish they would have found that earlier.