There are a lot of causes for OOE in Java. Here is my list of causes

• Exhausted heap. Try to increase heap size with -Xmx.
• Too small PermSize. You can increase this memory pool by adding -XX:MaxPermSize=value to the JVMs command line. If you use JSPs or some similar technology you should set this to at least 128MB. (see http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html for details)
• No more free filehandles (mostly a Windows problem) or other system resources like sockets etc.
• Exhausted virtual memory of the OS: if you set the heap size on a 32Bit OS to a too large value the size of the Java process can become larger than the maximum size of a process. So even when plenty of free RAM is available your Java application cannot get more memory from the OS (resulting in OOE). The BEA JRockit JVM has an automatic fail-safe mechanism to prevent such conditions.

Generally: newer Java versions are much better in telling you what the problem is. So if you encounter such errors try to run your application with the latest Java version (if possible). Especially Java6 is much better in this. See http://java.sun.com/developer/technicalArticles/J2SE/monitoring/ for details.