id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc,drp_resources,i_links,o_links,remaining_time,sprint 10464,"Bug: HDF5 isn't threadsafe by default (Tables openTable() / addData() sometimes returns None, occasionally Ice::ObjectNotExistException)",spli,spli,"Another mysterious intermittent bug. Occasionally `openTable(ofile)` returns `None` in the following (I know `name` isn't required for `getObject` if the `id` is given, but that should be irrelevant): {{{ attrs = {'id': long(tableId)} if tableName: attrs['name'] = tableName ofile = conn.getObject(""OriginalFile"", attributes = attrs) table = res.openTable(ofile) }}} Retrying it immediately succeeds (`table` is a valid handle). Looking through `Tables-0.log` for the table OriginalFile ID (1688): {{{ 2013-02-24 20:34:41,226 INFO [ omero.tables.TablesI] (Dummy- 3 ) getTable: 1688 {'omero.session.uuid': '138230fc-0ceb-4736-bab2-1d15b4c5f4d 4', 'omero.client.uuid': '32cb3552-6cc3-4526-be4d-27a51304a7f6'} 2013-02-24 20:34:41,257 INFO [ omero.remote] (Dummy- 11 ) Meth: TablesI.getTable 2013-02-24 20:34:41,258 INFO [ omero.tables.TablesI] (Dummy- 11 ) getTable: 1688 {'omero.session.uuid': '138230fc-0ceb-4736-bab2-1d15b4c5f4d 4', 'omero.client.uuid': '32cb3552-6cc3-4526-be4d-27a51304a7f6'} 2013-02-24 20:34:41,276 INFO [ omero.tables.HdfStorage] (Dummy- 11 ) Size: 0 - Attaching Table-FDDF8540-1E9F-404B-BABF-C87FDDA6335E to /Users/s imon/var/omero.data/Files/Dir-001/1688 2013-02-24 20:34:41,276 INFO [ omero.remote] (Dummy- 11 ) Rslt: FDDF8540-1E9F-404B-BABF-C87FDDA6335E -t @ Tables-0.TablesAdapter 2013-02-24 20:34:41,277 INFO [ omero.tables.HdfStorage] (Dummy- 3 ) Size: 1 - Attaching Table-CDD126B0-9BA6-49E5-939F-5AF655DB335E to /Users/s imon/var/omero.data/Files/Dir-001/1688 2013-02-24 20:34:41,278 INFO [ omero.remote] (Dummy- 3 ) Rslt: CDD126B0-9BA6-49E5-939F-5AF655DB335E -t @ Tables-0.TablesAdapter }}} Now it looks like it's been opened twice, so the first `openTable()` call should've worked. Corresponding section of `Blitz-0.log`: {{{ 2013-02-24 20:34:36,173 INFO [ ome.services.blitz.fire.Registry] (l.Serv er-4) Found 2 table services(s) : [Tables-0 -t @ Tables-0.TablesAdapter, Tables -t:tcp -h 192.168.1.84 -p 65353:tcp -h 192.168.56.1 -p 65353] 2013-02-24 20:34:41,225 INFO [ ome.services.util.ServiceHandler] (l.Server-2) Executor.doWork -- ome.services.blitz.impl.SharedResourcesI.checkOriginalFilePermissions(1688) 2013-02-24 20:34:41,226 INFO [ ome.services.util.ServiceHandler] (l.Server-2) Args: [null, InternalSF@50258881] 2013-02-24 20:34:41,227 INFO [ ome.security.basic.EventHandler] (l.Server-2) Auth: user=2,group=3,event=null(User),sess=138230fc-0ceb-4736-bab2-1d15b4c5f4d4 2013-02-24 20:34:41,239 INFO [ org.perf4j.TimingLogger] (l.Server-2) start[1361738081226] time[13] tag[omero.call.success.ome.services.blitz.impl.SharedResourcesI$4.doWork] 2013-02-24 20:34:41,239 INFO [ ome.services.util.ServiceHandler] (l.Server-2) Rslt: ome.model.core.OriginalFile:Id_1688 2013-02-24 20:34:41,240 INFO [ ome.services.blitz.fire.Registry] (l.Server-2) Found 2 table services(s) : [Tables-0 -t @ Tables-0.TablesAdapter, Tables -t:tcp -h 192.168.1.84 -p 65353:tcp -h 192.168.56.1 -p 65353] 2013-02-24 20:34:41,268 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Meth: interface ome.api.IAdmin.canUpdate 2013-02-24 20:34:41,268 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Args: [ome.model.core.OriginalFile:Id_1688] 2013-02-24 20:34:41,272 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Meth: interface ome.api.IAdmin.canUpdate 2013-02-24 20:34:41,272 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Args: [ome.model.core.OriginalFile:Id_1688] 2013-02-24 20:34:41,273 INFO [ ome.security.basic.EventHandler] (l.Server-5) Auth: user=2,group=3,event=null(User),sess=138230fc-0ceb-4736-bab2-1d15b4c5f4d4 2013-02-24 20:34:41,274 INFO [ ome.security.basic.EventHandler] (l.Server-3) Auth: user=2,group=3,event=null(User),sess=138230fc-0ceb-4736-bab2-1d15b4c5f4d4 2013-02-24 20:34:41,275 INFO [ org.perf4j.TimingLogger] (l.Server-5) start[1361738081272] time[3] tag[omero.call.success.ome.logic.AdminImpl.canUpdate] 2013-02-24 20:34:41,275 INFO [ ome.services.util.ServiceHandler] (l.Server-5) Rslt: true 2013-02-24 20:34:41,277 INFO [ org.perf4j.TimingLogger] (l.Server-3) start[1361738081268] time[9] tag[omero.call.success.ome.logic.AdminImpl.canUpdate] 2013-02-24 20:34:41,277 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Rslt: true }}} This error seems to only occur when opening multiple tables, but is frequent enough to warrant using this workaround: {{{ def openRetry(ofile, n): for i in xrange(n): t = self.res.openTable(ofile) if t: return t print 'Failed to open table %d (attempt %d)' % \ (ofile.getId().val, i + 1) raise Exception('Failed to open table %d' % ofile.getId().val) }}} On rare occasions an `Ice::ObjectNotExistException` is thrown, this is on a local server which should rule out network issues, the Table definitely exists and is accessble by the current user. `Tables-0.log`: {{{ 2013-02-24 00:32:27,853 INFO [ omero.remote] (Dummy- 11 ) Meth: TablesI.getTable 2013-02-24 00:32:27,853 INFO [ omero.tables.TablesI] (Dummy- 11 ) getTable: 1603 {'omero.session.uuid': 'b88e52e2-cea5-4619-9355-0a08026a9a2 d', 'omero.client.uuid': '4d5818ed-1dbe-4719-9ad4-9caccbd52fef'} 2013-02-24 00:32:27,905 INFO [ omero.remote] (Dummy- 11 ) Excp: exception ::Ice::ObjectNotExistException { id = { name = b88e52e2-cea5-4619-9355-0a08026a9a2d c 2013-02-24 00:32:27,906 WARNI [ omero.remote] (Dummy- 11 ) raised a non-ServerError (): exception ::Ice::ObjectNotExistException { id = { name = b88e52e2-cea5-4619-9355-0a08026a9a2d category = session-4d5818ed-1dbe-4719-9ad4-9caccbd52fef } facet = operation = getAdminService } }}} ",task,closed,critical,5.0.3,Services,4.4.8,fixed,,python-team@… analysis@…,,,,0,