In the meantime, about the last comment you made, I still wonder if I’m missing how we can tell how many requests are in that state of “waiting on the server socket”, in other words, that have not yet “come out of serversocket.accept()”. It’s those concepts you’ve raised which I’m most interested in.

I have a sense that we don’t often measure that like we might and it could be important. Or have I missed something you said that clarified that.

Yes. You are right about handler threads. Though the definition for this set applies to all the threadpools in JRun, I was mainly talking about the web threads – which actually mean WebService incase of the internal web server or jRunProxyService in case of the connector.

As far as I know, Schedulerservice is used by the Metrics service of the Jrun. Thats the service which collects various metric information that you can see using cfstat.

I’d like to ask another point of clarification: you made the comment that the default setting of 1000 for maxhandlerthreads may be too large, because when a server’s traffic finally approaches using that that many, the OS may simply not be able to provide them. Do I have that right?

Yes. There are two factors here – 1) OS will not be able to give you that much thread. This is too high a number and this is just a part of total number of threads in the JVM. 2)Even if it does, the VM might not scale upto that. There will be too much of thread contention and scheduling and the processor will be busy more in that rather than doing any meaningful processing.

Now how do you find the optimum number? – you have to run a load test and keep increasing the load and the numbers and arrive at some optimum.

And since you said the total is those threads running + those queued + those “waiting on the server socket”, how do we monitor each? I’ll assume jrpp.totalth is the total currently used, and jrpp.busyth is those active. Is jrpp.delayTh those queued? What then are those you describe as “waiting on the server socket”? Is that jrpp.delayTh or jrpp.delayRq?

You can refer to this document for jrpp metrics – http://livedocs.adobe.com/jrun/4/JRun_Administrators_Guide/netmon2.htm. This should answer most of your questions. jrpp.delayTh gives you the count of threads which have accepted the request but are queued because of high load. This is different from threads waiting on the server socket. If you see the description of min Handler threads in my post above, I have mentioned clearly that when a thread accepts the request i.e comes out of serversocket.accept(), it goes through a throttle and if the total no of threads processing the request has reached the activeHandler count, then it will be queued.

In that case, how do we best monitor that, and how do we mitigate it?

In the case of monitoring, if on Windows, are we talking about monitoring the threadcount counter in the process object for the jrun instance? And would this be the same value we’d see if we enabled Jrun metrics to watch jrpp.totalTh?

And since you said the total is those threads running + those queued + those “waiting on the server socket”, how do we monitor each? I’ll assume jrpp.totalth is the total currently used, and jrpp.busyth is those active. Is jrpp.delayTh those queued? What then are those you describe as “waiting on the server socket”? Is that jrpp.delayTh or jrpp.delayRq?

Also, can we monitor these through perfmon?

Finally, as for how to mitigate the problem, one may wonder what they can do when they do hit the error “unable to create new native threads”. If we lower the max, that will increase the chance of people getting “server unavailable” errors, right?

But then maybe one could handle more total threads by using the -Xss jvm argument to lower the amount of stack space used per thread, as discussed in the first comment at http://www.talkingtree.com/blog/index.cfm/2005/3/11/NewNativeThread. What do you think of that?

FInally, what if one is getting the “Unable to create new native thread” error, but analysis shows that there’s not a problem at the time with there being too many running or queued requests? How then might we still get this error? It would seem surprising for the requests “waiting on the server socket” would be that high.

But maybe this points to something else not said in your entry (and hinted at in my last comment). What if the inability to create a new thread is due to the total thread count being more than just the threads from the jrun.servlet.jrpp.JRunProxyService?

Maybe it’s also requests coming in on the jrun.servlet.http.WebService, if the internal server is enabled. And maybe it’s also requests/threads running per the jrunx.scheduler.SchedulerService. I see that the Jrun metrics let us monitor them, too, but what sort of things happen on those threads?

And what about CFTHREADs? and CFREPORT threads? And though only Enterprise lets you configure thread counts for them, what about flash remoting, web service, and http-based CFC method calls?

Finally, for others facing this problem, what if additional threads are being grabbed by things other than JRun in the JVM. Consider if you’re running something like FusionReactor or SeeFusion, since those have their own web server that they deploy within the CF instance.

If we could accurately measure what CF thinks it’s using (across all 3 thread types), and what the jrun process reports is being used to the OS, we might then be able to infer the difference to be associated with them. Or am I off base?

Of course, the CF8 Server Monitor also uses threads, but it’s using Flash Remoting I think.

I realize this is a long couple of comments, but this is all very interesting stuff that’s often been hard to understand.

One thing you’ve not clarified, though, is which set of handlerthread entries you were referring to. In the jrun.xml, there are 3 such sets:

- jrunx.scheduler.SchedulerService
- jrun.servlet.http.WebService
- jrun.servlet.jrpp.JRunProxyService

Since you refer to the activeHandlerThreads being connected to the Admin setting for simultaneous requests, and I find that that gets set for the latter two above, I’ll assume you mean them. And the difference for them is that the WebService is for requests from the internal web server, while the JRunProxyService is for those from an external web server (such as IIS or Apache). Can you confirm?

And what about the SchedulerService? Where does it come into play for this discussion? You’re right that so much of this stuff has always been very much a black art with confusing recommendations, within and without the CF community.

Finally, it seems worth clarifying for some readers where this jrun.xml is found. In a standalone deployment (on Windows, for CF8), it would be in C:\ColdFusion8\runtime\servers\coldfusion\SERVER-INF\, and in the multiserver mode, it would be C:\JRun4\servers\[instancename]\SERVER-INF.