Bug: TB reused after exception in omero.gateway

[jamoore]

A TB seems to be reused after an exception several times, causing odd Hibernate exceptions. Most likely the instance should be closed after the first such exception.

Possibly related to #5827

See attached log.

[Aleksandra Tarkowska <aleksandrat@…>]

(In [35055af8404d33a62a03ae5fc5e95afb0e50c1b7/ome.git] on branch develop) this closes thumbnail service if anything failed, see #5828

[jmoore]

Moving to 4.3.1 for handling

[jmoore]

I'm still not sure what caused the internal exception to be thrown, and in fact, that issue may have been fixed by various ConcurrencyException fixes. However, looking at wrapped, I think I know what's causing this to keep throwing:

    def wrapped (*args, **kwargs): #pragma: no cover
        try:
            return inner(*args, **kwargs)
        except Ice.MemoryLimitException:
            logger.debug("MemoryLimitException! abort, abort...")
            raise
        except omero.SecurityViolation:
            logger.debug("SecurityViolation, bailing out")
            raise
        except omero.ApiUsageException:
            logger.debug("ApiUsageException, bailing out")
            raise
        except omero.ConcurrencyException:
            logger.debug("ConcurrencyException, bailing out")
            raise
        except Ice.UnknownException:
            logger.debug("UnknownException, bailing out")
            raise
        except Ice.ConnectionLostException:
            logger.debug("ConnectionLostException, bailing out")
            raise
        except Ice.CommunicatorDestroyedException:
            logger.debug("CommunicatorDestroyedException, bailing out")
            raise
        except Ice.Exception, x:
            logger.debug('wrapped ' + f.func_name)
            logger.debug(x.__dict__)
            if hasattr(x, 'serverExceptionClass') and x.serverExceptionClass == 'ome.conditions.InternalException':
                if x.message.find('java.lang.NullPointerException') > 0:
                    logger.debug("NullPointerException, bailing out")
                    raise
                elif x.message.find('Session is dirty') >= 0:
                    logger.debug("Session is dirty, bailing out")
                    raise
                else:
                    logger.debug(x.message)
            logger.debug("exception caught, first time we back off for 10 secs")
            logger.debug(traceback.format_exc())
            #time.sleep(10)
            return inner(*args, **kwargs)
    return wrapped

The handler for Ice.Exception will pick up just about every exception. The one thrown in this case is an serverExceptionClass but it doesn't contain either of those strings but rather "Hibernate session is not re-entrant". Since the exception doesn't match, the inner call at the bottom tries again when it definitely should not. We should most likely fail hard unless this is a case where we know we can retry.

Note: the logging of backing off for 10 seconds should also be adjusted.

[wmoore]

So, do we simply remove the final

return inner(*args, **kwargs)

or are there times when we do want to retry? Does waiting 10 seconds increase our chance of success in any cases?

[wmoore]

Not going to make big changes to exception handling in 4.3.1, pushing.

[wmoore]

Carlos - When we discussed retrying within Exception handling, Josh thought that retrying should be the exception whereas you had reason to prefer retrying as the norm. This behavior also relates to #5827, so can I pass both these tickets to you?

[cneves]

This should have been resolved by #6478, can you double check it?

[wmoore]

I believe that this issue has gone away with the extensive re-working of safeCallWrap. See #6478