Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Bug #834 (closed)

Opened 12 years ago

Closed 11 years ago

JBoss PermGen Errors

Reported by: cxallan Owned by: cxallan
Priority: critical Cc: atarkowska
Sprint: n.a.
Total Remaining Time: n.a.

Description

JBoss is throwing java.lang.OutOfMemoryError: PermGen space exceptions:

15:12:51,890 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/codegen/CodeStream
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:508)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:584)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.generateCode(CompilationUnitDeclaration.java:214)
        at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:602)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:411)
        at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:405)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:308)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:286)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:273)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:566)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)
15:12:55,949 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/codegen/CodeStream
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:508)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:584)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.generateCode(CompilationUnitDeclaration.java:214)
        at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:602)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:411)
        at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:405)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:308)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:286)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:273)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:566)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)
15:51:43,148 ERROR [STDERR] Exception in thread "RMI TCP Connection(idle)"
15:51:43,151 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space
15:51:57,866 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@723e94a6
java.lang.OutOfMemoryError: PermGen space
16:21:54,278 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@53b69ca8
java.lang.OutOfMemoryError: PermGen space
16:22:24,906 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@6264c005
java.lang.OutOfMemoryError: PermGen space
16:22:48,294 ERROR [STDERR] Exception in thread "WorkerThread#0[10.34.2.145:57702]"
16:22:48,294 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space

Change History (7)

comment:1 Changed 12 years ago by cxallan

  • Cc josh removed

comment:2 Changed 12 years ago by cxallan

Context from the JBoss server.log:

2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:18:56>
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:19:06>
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:21:06,274 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:21:06>
2007-11-12 16:21:06,274 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:21:06,275 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:21:06,275 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:21:16,279 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:21:16>
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:21:54,278 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@53b69ca8
java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:22:24,906 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@6264c005
java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:22:45,285 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
2007-11-12 16:22:48,294 ERROR [STDERR] Exception in thread "WorkerThread#0[10.34.2.145:57702]" 
2007-11-12 16:22:48,294 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:23:16>
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:23:26>
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:25:00,446 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
2007-11-12 16:25:26,324 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:25:26>
2007-11-12 16:25:26,324 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:25:26,325 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:25:26,325 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:25:36,329 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:25:36>

comment:3 Changed 12 years ago by jmoore

For anyone experiencing this, it would probably be useful to run JBoss with the following:

JAVA_OPTS=-XX:-HeapDumpOnOutOfMemoryError $JBOSS_HOME/bin/run.sh

comment:4 Changed 12 years ago by cxallan

  • Cc atarkowska added
  • Owner changed from jmoore to callan
  • Status changed from new to assigned

Looking at the JVM's permanent generation usage (explanation on what the columns represent is available ​here (java.sun.com)):

$ jstat -gcutil 13242
 S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
 0.00  97.83  47.87  82.41  99.12    327    4.487    33   25.707   30.194

The important column for us is P which tells us the permanent generation space utilization is at 99.12%. Not exactly ideal and just begging to be pushed beyond the maximum allowable space.

Checking the actual numbers on permanent generation maximum allowable space we achieve by examining further output from jstat:

$ bin/jstat -gcpermcapacity 13242
 PGCMN      PGCMX       PGC         PC      YGC   FGC    FGCT     GCT   
 21248.0    86016.0    82240.0    82240.0   328    33   25.707   30.207

So, it would appear that our solution lies in increasing the size of the permanent generation using the -XX:MaxPermSize JVM arguments as documented ​here (java.sun.com).

The default permanent generation size for Java 5.0 and newer VMs is:

  • 64MB on 32-bit systems
  • 64MB * 30% (83.20MB) on 64-bit systems

comment:5 Changed 12 years ago by cxallan

Now testing with the following:

JAVA_OPTS="-XX:MaxPermSize=128M" $JBOSS_HOME/bin/run.sh

comment:6 Changed 12 years ago by cxallan

  • Milestone set to 3.0-Beta2.3

comment:7 Changed 11 years ago by jmoore

  • Milestone set to OMERO-Beta4
  • Resolution set to fixed
  • Status changed from assigned to closed

As of #1106 (OmeroBlitz migration) JBoss is no longer used. The max perm options, though, may need to be added to OmeroBlitz's startup eventually.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.76276 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!