Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #7272 (closed)

Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

Bug: /tmp/files on mage EMDB

Reported by: wmoore Owned by: wmoore
Priority: critical Milestone: OMERO-4.4
Component: Web Version: n.a.
Keywords: n.a. Cc: atarkowska, cxallan, jamoore
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: 2011-11-29 (3)

Description

Seems that various file downloads etc are not cleaning up files in /tmp. Found large numbers of files there. Cleaned up with:

jboss@mage /tmp $ ls | less            # lots of files!
jboss@mage /tmp $ rm *emanFilter.jpg
jboss@mage /tmp $ rm *projection.jpg
jboss@mage /tmp $ rm *download
jboss@mage /tmp $ rm *dataset_stack.mrc

Also, large numbers of sessionid* files

jboss@mage /tmp $ rm sessionid*
-bash: /bin/rm: Argument list too long
jboss@mage /tmp $ rm sessionid0*
jboss@mage /tmp $ rm sessionida*
# etc.....!

Change History (5)

comment:1 Changed 8 years ago by wmoore

  • Status changed from new to accepted

comment:2 Changed 8 years ago by wmoore

NB: when mage runs out of space, postgres crashes (needs a restart)

2011-11-10 08:12:25,552 INFO  [        ome.services.util.2011-11-10 18:20:25,535 INFO  [ome.services.sessions.state.SessionCache] (3-thread-5) Synchronizing session cache. Count = 2
2011-11-10 18:20:25,564 INFO  [        ome.services.util.ServiceHandler] (3-thread-5)  Executor.doWork -- ome.services.sessions.SessionManagerImpl.reload(fc271748-e649-48f9-8ea7-54e18968c996)
2011-11-10 18:20:25,564 INFO  [        ome.services.util.ServiceHandler] (3-thread-5)  Args:    [null, InternalSF@351984520]
2011-11-10 18:20:25,568 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Found location in errorTimes: -1
2011-11-10 18:20:25,568 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Removing 0 from errorTimes
2011-11-10 18:20:25,568 WARN  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Registering error with list: Current size: 0
2011-11-10 18:20:25,568 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Sleeping for 0 then retry: 1
2011-11-10 18:20:25,569 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Found location in errorTimes: -1
2011-11-10 18:20:25,569 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Removing 0 from errorTimes
2011-11-10 18:20:25,569 WARN  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Registering error with list: Current size: 1
2011-11-10 18:20:25,569 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Sleeping for 1000 then retry: 2
2011-11-10 18:20:26,572 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Found location in errorTimes: -1
2011-11-10 18:20:26,572 INFO  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Removing 0 from errorTimes
2011-11-10 18:20:26,572 WARN  [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Registering error with list: Current size: 2
2011-11-10 18:20:26,572 ERROR [ome.services.db.SelfCorrectingDataSource] (3-thread-5) Failed to acquire connection after retries=3
java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource 62ea0d04-0f3d-4102-88a9-9e60d570fff0 with 0 connection(s) (0 still available)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:201)
        at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:83)
        at ome.services.db.SelfCorrectingDataSource.call(SelfCorrectingDataSource.java:102)
        at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:75)
        at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:59)
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:354)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:103)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at $Proxy64.doWork(Unknown Source)
        at ome.services.util.Executor$Impl.execute(Executor.java:371)
        at ome.services.sessions.SessionManagerImpl.reload(SessionManagerImpl.java:886)
        at ome.services.sessions.state.SessionCache.reload(SessionCache.java:628)
        at ome.services.sessions.state.SessionCache.doUpdate(SessionCache.java:580)
        at sun.reflect.GeneratedMethodAccessor805.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
        at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: unable to connect to non-XA resource org.postgresql.Driver
        at bitronix.tm.resource.jdbc.lrc.LrcXADataSource.getXAConnection(LrcXADataSource.java:106)
        at bitronix.tm.resource.jdbc.PoolingDataSource.createPooledConnection(PoolingDataSource.java:274)
        at bitronix.tm.resource.common.XAPool.createPooledObject(XAPool.java:283)
        at bitronix.tm.resource.common.XAPool.grow(XAPool.java:400)
        at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:379)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:123)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:197)
        ... 29 more
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:123)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:124)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
        at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:386)
        at org.postgresql.Driver.connect(Driver.java:260)
        at bitronix.tm.resource.jdbc.lrc.LrcXADataSource.getXAConnection(LrcXADataSource.java:102)
        ... 36 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:525)
        at java.net.Socket.connect(Socket.java:475)
        at java.net.Socket.<init>(Socket.java:372)
        at java.net.Socket.<init>(Socket.java:186)
        at org.postgresql.core.PGStream.<init>(PGStream.java:62)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:77)
        ... 43 more
2011-11-10 18:20:26,601 INFO  [                 org.perf4j.TimingLogger] (3-thread-5) start[1320949225564] time[1037] tag[omero.call.exception]
2011-11-10 18:20:26,601 INFO  [        ome.services.util.ServiceHandler] (3-thread-5)  Excp:    ome.conditions.DatabaseBusyException: Cannot acquire connection
2011-11-10 18:20:26,601 WARN  [ome.services.sessions.state.SessionCache] (3-thread-5) 1error(s) on reload of fc271748-e649-48f9-8ea7-54e18968c996
ome.conditions.DatabaseBusyException: Cannot acquire connection
        at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:91)
        at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:59)
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:354)
        at org.springframework.transaction.interceptor

comment:3 Changed 8 years ago by wmoore

Code like this was creating a new file for every session

    tempdir = settings.FILE_UPLOAD_TEMP_DIR
    tempJpg = os.path.join(tempdir, ('%s.emanFilter.jpg' % (conn._sessionUuid))).replace('\\','/')
    em.write_image(tempJpg)
    originalFile_data = FileWrapper(file(tempJpg))
    rsp = HttpResponse(originalFile_data)
    rsp['Content-Type'] = "image/jpg"
    return rsp

comment:4 Changed 8 years ago by wmoore

  • Resolution set to fixed
  • Status changed from accepted to closed

Fixed in this commit htts://github.com/will-moore/openmicroscopy/commit/c8ba957d7b91522d7433c62dc4cf043e56a6b590

comment:5 Changed 8 years ago by wmoore

Updated mage. Looking at /var/log/OMEROweb.log with debugging ON, I see these lines for the various temp files mentioned above

Mon, 21 Nov 2011 11:51:44 webemdb      DEBUG    EMAN2 xyz projection: {'name': '/tmp/tmpR3FJUO.jpg', 'size': 1223}
Mon, 21 Nov 2011 11:47:23 webemdb      DEBUG    EMAN2 filter jpeg: {'name': '/tmp/tmpqqoZmw.jpeg', 'size': 671}
Mon, 21 Nov 2011 11:48:40 webemdb      DEBUG    download file: {'size': 5114, 'name': '/tmp/tmpOrR6Nm.fileAnn', 'fileName': '80_1080.gif'}
Mon, 21 Nov 2011 11:52:30 webemdb      DEBUG    EMAN2 dataset download: {'name': '/tmp/tmpadRO5M.mrc', 'size': 166912}

Then, looking in /tmp/ there are no tmp* files

jboss@mage ~ $ ls /tmp/tmp*
ls: cannot access /tmp/tmp*: No such file or directory
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.78304 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!