January 4, 2008
I've been chasing a problem with Apache for the past several months now that I finally was able to solve last night. After upgrading to Apache 2.2.6 and installing ColdFusion 8, I started having intermittent problems with images not loading on pages, very slow page loads, or page requests simply hanging. At first, I thought the problem was a ColdFusion Apache connector problem, because I could only make it happen when I was running Apache with ColdFusion. After a long and tedius testing and troubleshooting process, I was finally able to narrow the issue to Apache itself, and not the JRun connector for ColdFusion.
I caught a big break when I saw the following error message in my logs:
[warn] (OS 995)The I/O operation has been aborted because of either a thread exit or an application request. : winnt_accept: Asynchronous AcceptEx failed.
I had seen the same error before a few months back, but Googling the error didn't produce anything useful in past attempts. Last night, for whatever reason, I hit pay dirt. This blog entry put me on the right track. I'm not sure why I didn't find it before as it was posted in 2006, but that really isn't important.
The blog post suggests that the error is caused by a conflict between one or more programs (anti-virus, firewall, virtualization, or vpn) and AcceptEx(), and goes on to say that disabling it using the win32DisableAcceptEx directive should solve the problem. Curious, I googled win32DisableAcceptEx and found this in the Apache docs:
[error] (730038)An operation was attempted on something that is not a socket.: winnt_accept: AcceptEx failed. Attempting to recover.
you should use this directive to disable the use of AcceptEx().
AcceptEx() is a Microsoft WinSock v2 API that provides some performance improvements over the use of the BSD style accept() API in certain circumstances. Some popular Windows products, typically virus scanning or virtual private network packages, have bugs that interfere with the proper operation of AcceptEx(). If you encounter an error condition like:
Here's the relevant section from my Apache 2.2.6 httpd.conf file that needed to be changed:
# memory-mapping or the sendfile syscall is used to deliver
# files. This usually improves server performance, but must
# be turned off when serving from networked-mounted
# filesystems or if support for these functions is otherwise
# broken on your system.
EnableMMAP off #<-- uncomment this line
EnableSendfile off #<-- uncomment this line
Win32DisableAcceptEx #this line had to be added
# EnableMMAP and EnableSendfile: On systems that support it,
Although the Apache documentation doesn't mention the EnableMMAP and EnableSendfile directives, I found that I had to disable them as well (as per the first blog post I found). To do so, you'll notice that I simply had to uncomment the EnableMMAP off line as well as the EnableSendfile off line. The win32DisableAcceptEx line is a little less obvious. It isn't included anywhere in the default httpd.conf file, but it was listed in the Apache docs, so I just added the directive after the EnableSendfile off.
After a quick restart of the Apache service, I ran through my battery of tests across all of my instances and wouldn't you know it, problem solved!