Posts Tagged java

DNS lookup Caching in ColdFusion/Java

Two days back, I got an interesting question from our support team. The Customer in this case was using CFLdap which connected to a particular LDAP server and things were working fine. The problem came when they replaced this LDAP server with another LDAP server and assigned the same dns name to the new ldap server. Ideally any connection made henceforth should have worked with the new ldap server but actually that did not happen. ColdFusion started throwing error and it did not work until the ColdFusion server is restarted. Ever seen something similar?

It is obvious that it happend because the IP address was being cached for the host name as a result of which ColdFusion was still trying to connect to the old IP address even though the host name now pointed to a different IP address. This caching also applies for all other network protocol tags such as CFHTTP, CFFTP, CFFEED etc and is not limited to CFLDAP. It is actually the JVM that does this caching. When JVM is requested to resolve a host name, it does the host name to IP address resolution and caches the result. This result will be used for all subsequent lookups. It also caches the negative results. By that I mean, if the dns reolution fails, it caches the failed result for a certain period so that any lookup for that hostname in that period will not result into another resolution on network and will immediately return the failed result.

For more detail on this caching, check out the Javadoc for InetAddress class.

As per this doc, there are two parameters that control this caching

networkaddress.cache.ttl (default: -1) : Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup.A value of -1 indicates “cache forever”.

networkaddress.cache.negative.ttl (default: 10) : Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups.A value of 0 indicates “never cache”. A value of -1 indicates “cache forever”.

Now where do you specify this settying? You can specify this setting in <Java_home>/jre/lib/security/java.security file. For standalone ColdFusion server it will be in <ColdFusion _dir>/runtime/jre/lib/security/java.security file.

As you see, by default networkaddress.cache.ttl caches the result for ever and hence it is configured for best performance. Any change to this mean drop in performance. If you don’t want to cache the resolved IP address for ever, as is the case here, you would need to change networkaddress.cache.ttl value to 60 seconds or 300 seconds or any value you feel suitable. You would not want to set it to 0 as that would mean “never cache” the result which might affect the performance significantly.

In which case you would want to change the value for networkaddress.cache.negative.ttl? That would be mostly in case when you want to cache the negative result for a longer time and in turn improving the performance. For example, if you are trying to connect to a hostname which can not be resolved to any ip address, and that happens very frequently, each of the call (as long as they are not in the same 10 sec window) would become very slow. Increasing this value would increase the performance but again you would not want to cache the negative result for ever.

After you change this setting, you will have to restart the ColdFusion server for this change to take effect.

Tags: , ,

Thread dumps…

Many a times our customers complain of Coldfusion Server(CF) hanging, becoming non-responsive, taking 100% cpu or some request taking too much time. In order to understand the reasons why this happens, only way is to analyze the “Thread dumps”. Not many of our CF users are java geeks and therefore when we ask them to send a thread dump, they have a very little clue about it. I hope they wouldn’t be clueless anymore ;)

What exactly is “Thread dump”
“Thread dump” basically gives you information on what each of the thread in the VM is doing at any given point of time. This makes “Thread dump” an excellent debugging tool. It can tell you the states of each of the thread in the VM, where exactly each thread is in the execution path at that point, which thread is waiting and where is it waiting and lot more. It also shows you the stack of each thread and can help you track the execution path.

How do I take a Thread dump?
For any Java VM, if you are running it from a command prompt, you can get the thread dump by pressing Ctrl+Break (for windows) or Ctrl+\ (for unix machines) on the terminal running the server.
In case you are running the VM in background or as services, you can send a “SIGQUIT” signal to the process to get the thread dump. To send this signal to VM on unix machine, you can use “kill -SIGQUIT [pid]” where ‘pid’ is the process id of the server process. This will send the signal to the VM to dump the thread stack on the error stream (On Sun JVM) or to a new file (IBM’s JVM).

You can also use the Stacktrace tool to get the thread dump. It is a nice java webstart application that you can run on the machine running the server. All you need is to specify the “process id” and it will nicely show the threaddump in its own window.

To be more specific for CF, if the standalone CF server or CF on J2EE server has a terminal, use Ctrl+Break or Ctrl+\ depending on the platform. If its running as background process or as services, use Stacktrace tool I mentioned earlier.

Here is how the thread dump would look.


Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode):

"web-4" prio=5 tid=0x28999418 nid=0x1460 runnable [0x2fbaf000..0x2fbafd18]
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:374)
at java.io.File.getCanonicalPath(File.java:531)
at java.io.FilePermission$1.run(FilePermission.java:218)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:212)
at java.io.FilePermission.(FilePermission.java:264)
at java.lang.SecurityManager.checkRead(SecurityManager.java:871)
at java.io.File.exists(File.java:700)
at jrunx.resource.FileResource.exists(FileResource.java:98)
at jrunx.resource.FileResource.getURL(FileResource.java:140)
at jrun.servlet.JRunServletContext.getResource(JRunServletContext.java:192)
at jrun.servlet.file.FileServlet.service(FileServlet.java:148)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"web-3" prio=5 tid=0x2814e470 nid=0xd70 in Object.wait() [0x2faaf000..0x2faafd98]
at java.lang.Object.wait(Native Method)
- waiting on (a jrunx.scheduler.JSemaphore)
at jrunx.scheduler.JSemaphore.acquire(JSemaphore.java:74)
- locked (a jrunx.scheduler.JSemaphore)
at jrun.servlet.network.NetworkService.accept(NetworkService.java:348)
at jrun.servlet.http.WebService.createRunnable(WebService.java:104)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable(ThreadPool.java:285)
at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:347)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable(ThreadPool.java:239)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"web-2" prio=5 tid=0x271954c0 nid=0x170c in Object.wait() [0x2ef6e000..0x2ef6fa18]
at java.lang.Object.wait(Native Method)
- waiting on (a java.awt.image.PixelGrabber)
at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:254)
- locked (a java.awt.image.PixelGrabber)
at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:209)
at com.lowagie.text.Image.getInstance(Unknown Source)
at com.lowagie.text.Image.getInstance(Unknown Source)
at com.lowagie.text.pdf.PdfGraphics2D.addAltText(Unknown Source)
at ice.pilots.html4.MacromediaCSSExtension.addonImageProcessing(MacromediaCSSExtension.java:186)
at ice.pilots.html4.ObjectBox.paintChunk(OEAB)
at ice.pilots.html4.InlineBox.paintChildren(OEAB)
at ice.pilots.html4.InlineBox.paintChunk(OEAB)
at ice.pilots.html4.InlineBox.paintChildren(OEAB)
at ice.pilots.html4.BlockBox.paint(OEAB)
at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
at ice.pilots.html4.BlockBox.paint(OEAB)
at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
at ice.pilots.html4.BlockBox.paint(OEAB)
at ice.pilots.html4.CSSLayout.paint(OEAB)
- locked (a java.lang.Object)
at ice.pilots.html4.ThePrinter.printPage(Unknown Source)
at coldfusion.document.DocumentSection.process(DocumentSection.java:230)
at coldfusion.document.DocumentSection.print(DocumentSection.java:108)
at coldfusion.document.DocumentExporter.export(DocumentExporter.java:237)
at coldfusion.document.DocumentFrame.exportContent(DocumentFrame.java:118)
at coldfusion.document.DocumentProcessor.processContent(DocumentProcessor.java:130)
at coldfusion.document.DocumentProcessor.ProcessContent(DocumentProcessor.java:59)
at coldfusion.tagext.lang.DocumentTag.processContent(DocumentTag.java:1218)
at coldfusion.tagext.lang.DocumentTag.access$100(DocumentTag.java:84)
at coldfusion.tagext.lang.DocumentTag$3.run(DocumentTag.java:1179)
at java.security.AccessController.doPrivileged(Native Method)
at coldfusion.tagext.lang.DocumentTag.doAfterBody(DocumentTag.java:1174)
at cfcmyk2ecfm937582361.runPage(E:\CFusionMX7\wwwroot\testfolder\doctest\cmyk.cfm:3)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:152)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:349)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:209)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:51)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69)
at coldfusion.filter.ClientScopePersistenceFilter .invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:115)
at coldfusion.CfmServlet.service(CfmServlet.java:107)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:78)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
.
.
.

If you anayze this thread dump, you can see that “web-2″ thread is handling request for “cmyk.cfm” and is waiting for the images to be loaded i.e (actually at Image.getInstance()). At this point, cfdocument has sent another http request for the image file that it has to render and that request is handled by another thread “web-4″ which is running and is currently checking if the image file requested exists on the file system. Isn’t it very powerful ?

You can checkout a very nice article Thread dump and stack traces written by Rajiv (my ex-colleague and mentor in Pramati) which I feel is a must read for Java developers.

Tags: , ,