Eric Scheie - 17/Jul/07 12:00 AM BTW, I noticed this problem on 1.2.23 as well. I upgraded to 1.2.30 hoping that it would solve the memory problems.

jason - 17/Jul/07 12:03 AM Hi Eric, I have had a look through the log files, and unfortunately they do not provide much help. However, you should be able to get the JVM to dump a heap snapshot on out of memory by passing the following option: -XX:+HeapDumpOnOutOfMemoryError This is a JVM option that will need to be added to wrapper.conf. When doing this, it is probably better to turn the heap size back down so that the dump itself is not so large. If you can get this to work, then the JVM will tell you the file it dumps the heap to on error. I will send instructions via email for uploading this file to us.

Eric Scheie - 17/Jul/07 12:34 AM I just finished uploading the dump to you. The file is named java_pid1742.hprof. I changed the heap size back to the default 512 as you recommended. No builds ran from the time I started pulse to the time It threw the out of memory error. Please let me know if I can provide any more debug information. Thanks for your assistance! -Eric

jason - 17/Jul/07 02:24 PM Hi Eric, Thanks for the dump. With this info and a lot of investigation, I think I may have found the problem. There appears to be a strange interaction between 'remember me' processing when logging in and HTTP sessions. In particular, if 'remember me' is turned on, when you hit Pulse from a web browser without a session cookie, you never get a session cookie. On the Pulse side, however, it is creating a new session every time and trying to set the cookie, but too late (actually, it is a library we use that sets it up too late). This means that every single request is generating a new session, and these sessions are hanging around until they timeout (after 30 mins). The sessions themselves hold information about the logged in user, which it turns out indirectly references a lot of other configuration information, leading to heap exhaustion after some number of HTTP requests. The solution is at least two-fold: 1) To fix remember me processing so that it does not interfere with session management. 2) To ensure that the session does not reference so much data anyway, as it is wasteful (there are other, less extreme, cases where a session can hang around for a while) In your heap dump there were about 1500 sessions. This indicates that about that many requests were made to Pulse from the time you started to heap exhaustion (note that viewing one page leads to multiple requests, so this may only be around 100 page hits depending on client side caching). Another thing to check out is if this is expected, or is there something hitting the web UI more frequently than you anticipate. Once the bug is fixed this level of usage should be fine in any case. I have a fix for 1) which I am testing and the fix for 2) is more straightforward. I will put up a new build as soon as both are addressed.

jason - 17/Jul/07 11:29 PM Fixed as suggested in change 3512.

Eric Scheie - 18/Jul/07 12:03 AM I just installed the 1.2.31 release and at first glance it seems to have resolved the issue. Its holding steady as far as memory consumption with myself and someone else trying to stress test it. Thanks for the quick turnaround as always!!! If only every software company could be as agile as you prove to be.