importer-testengine build blocks forever

[jamoore]

I've been killing them periodically, but at the moment, they're giving us nearly no feedback.

[jmoore]

​http://hudson.openmicroscopy.org.uk/job/omero-importer-testengine/270/console

from a "kill -QUIT 19000" (visible at the link above) looks to be stuck here:

"main" prio=10 tid=0x000000004011b800 nid=0x4a3c in Object.wait() [0x000000004022a000..0x000000004022aed0]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab4865a98> (a IceInternal.Outgoing)
	at java.lang.Object.wait(Object.java:485)
	at IceInternal.Outgoing.invoke(Outgoing.java:109)
	- locked <0x00002aaab4865a98> (a IceInternal.Outgoing)
	at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1132)
	at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
	at omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
	at ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:2962)
	at ome.formats.OMEROMetadataStoreClient$$FastClassByCGLIB$$c2d47132.invoke(<generated>)
	at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
	at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at ome.formats.test.util.IniWritingInterceptor.invoke(IniWritingInterceptor.java:73)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
	at ome.formats.OMEROMetadataStoreClient$$EnhancerByCGLIB$$1b3355fc.setPlane(<generated>)
	at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:476)
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:325)
	at ome.formats.test.util.TestEngine.processDirectory(TestEngine.java:214)
	at ome.formats.test.util.TestEngine.run(TestEngine.java:128)
	at ome.formats.test.util.TestEngine.main(TestEngine.java:503)

[jmoore]

Finally the communicator died:

2009-10-14 07:58:38,199 17549658   [      main] ERROR          ome.formats.test.util.TestEngine  - Ice.CommunicatorDestroyedException
Ice.CommunicatorDestroyedException
	at IceInternal.OutgoingConnectionFactory.findConnectionByEndpoint(OutgoingConnectionFactory.java:461)
	at IceInternal.OutgoingConnectionFactory.create(OutgoingConnectionFactory.java:257)
	at IceInternal.RoutableReference.createConnection(RoutableReference.java:848)
	at IceInternal.RoutableReference$1.setEndpoints(RoutableReference.java:484)
	at IceInternal.RouterInfo.getClientEndpoints(RouterInfo.java:92)
	at IceInternal.RoutableReference.getConnection(RoutableReference.java:476)
	at IceInternal.ConnectRequestHandler.connect(ConnectRequestHandler.java:41)
	at Ice._ObjectDelM.setup(_ObjectDelM.java:275)
	at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:1007)
	at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946)
	at omero.api.MetadataStorePrxHelper.createRoot(MetadataStorePrxHelper.java:46)
	at omero.api.MetadataStorePrxHelper.createRoot(MetadataStorePrxHelper.java:20)
	at ome.formats.OMEROMetadataStoreClient.createRoot(OMEROMetadataStoreClient.java:475)
	at ome.formats.OMEROMetadataStoreClient$$FastClassByCGLIB$$c2d47132.invoke(<generated>)
	at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
	at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at ome.formats.test.util.IniWritingInterceptor.invoke(IniWritingInterceptor.java:73)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
	at ome.formats.OMEROMetadataStoreClient$$EnhancerByCGLIB$$1b3355fc.createRoot(<generated>)
	at loci.formats.FormatReader.initFile(FormatReader.java:142)
	at loci.formats.in.MinimalTiffReader.initFile(MinimalTiffReader.java:213)
	at loci.formats.in.BaseTiffReader.initFile(BaseTiffReader.java:589)
	at loci.formats.in.NikonReader.initFile(NikonReader.java:434)

but two threads are dead-locked:

toring value for DJC_D2X_DOF_0002.nef: key=addDataset, value=: java.lang.String(mrc), java.lang.String(), omero.model.Project(omero.model.ProjectI@1259b924), 
Exception in thread "main" Ice.CommunicatorDestroyedException
	at IceInternal.OutgoingConnectionFactory.findConnectionByEndpoint(OutgoingConnectionFactory.java:461)
	at IceInternal.OutgoingConnectionFactory.create(OutgoingConnectionFactory.java:257)
	at IceInternal.RoutableReference.createConnection(RoutableReference.java:848)
	at IceInternal.RoutableReference$1.setEndpoints(RoutableReference.java:484)
	at IceInternal.RouterInfo.getClientEndpoints(RouterInfo.java:92)
	at IceInternal.RoutableReference.getConnection(RoutableReference.java:476)
	at IceInternal.ConnectRequestHandler.connect(ConnectRequestHandler.java:41)
	at Ice._ObjectDelM.setup(_ObjectDelM.java:275)
	at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:1007)
	at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946)
	at omero.api.IUpdatePrxHelper.saveAndReturnObject(IUpdatePrxHelper.java:320)
	at omero.api.IUpdatePrxHelper.saveAndReturnObject(IUpdatePrxHelper.java:294)
	at ome.formats.OMEROMetadataStoreClient.addDataset(OMEROMetadataStoreClient.java:2828)
	at ome.formats.OMEROMetadataStoreClient$$FastClassByCGLIB$$c2d47132.invoke(<generated>)
	at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
	at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at ome.formats.test.util.IniWritingInterceptor.invoke(IniWritingInterceptor.java:73)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
	at ome.formats.OMEROMetadataStoreClient$$EnhancerByCGLIB$$1b3355fc.addDataset(<generated>)
	at ome.formats.test.util.TestEngine.run(TestEngine.java:126)
	at ome.formats.test.util.TestEngine.main(TestEngine.java:503)

and

"pool-1-thread-1" prio=10 tid=0x00002aab3c00b800 nid=0x4a64 in Object.wait() [0x0000000041a42000..0x0000000041a42a10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab42b9eb8> (a IceInternal.IncomingConnectionFactory)
	at java.lang.Object.wait(Object.java:485)
	at IceInternal.IncomingConnectionFactory.waitUntilFinished(IncomingConnectionFactory.java:87)
	- locked <0x00002aaab42b9eb8> (a IceInternal.IncomingConnectionFactory)
	at Ice.ObjectAdapterI.waitForDeactivate(ObjectAdapterI.java:281)
	at IceInternal.ObjectAdapterFactory.waitForShutdown(ObjectAdapterFactory.java:99)
	at IceInternal.ObjectAdapterFactory.destroy(ObjectAdapterFactory.java:125)
	at IceInternal.Instance.destroy(Instance.java:864)
	at Ice.CommunicatorI.destroy(CommunicatorI.java:17)
	at omero.client.closeSession(client.java:662)
	at ome.formats.OMEROMetadataStoreClient.logout(OMEROMetadataStoreClient.java:451)
	at ome.formats.importer.util.ClientKeepAlive.run(ClientKeepAlive.java:41)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:619)

[bwzloranger]

Running now.