Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #5828 (closed)

Opened 13 years ago

Closed 13 years ago

Bug: TB reused after exception in omero.gateway

Reported by: jamoore Owned by: wmoore
Priority: major Milestone: OMERO-Beta4.3.2
Component: OmeroPy Version: n.a.
Keywords: n.a. Cc: python-team@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description

A TB seems to be reused after an exception several times, causing odd Hibernate exceptions. Most likely the instance should be closed after the first such exception.

Possibly related to #5827

See attached log.

Attachments (1)

TB.log (51.8 KB) - added by jmoore 13 years ago.

Download all attachments as: .zip

Change History (10)

Changed 13 years ago by jmoore

comment:1 Changed 13 years ago by Aleksandra Tarkowska <aleksandrat@…>

(In [35055af8404d33a62a03ae5fc5e95afb0e50c1b7/ome.git] on branch develop) this closes thumbnail service if anything failed, see #5828

comment:2 Changed 13 years ago by jmoore

  • Milestone changed from OMERO-Beta4.3 to OMERO-Beta4.3.1

Moving to 4.3.1 for handling

comment:3 Changed 13 years ago by jmoore

I'm still not sure what caused the internal exception to be thrown, and in fact, that issue may have been fixed by various ConcurrencyException fixes. However, looking at wrapped, I think I know what's causing this to keep throwing:

    def wrapped (*args, **kwargs): #pragma: no cover
        try:
            return inner(*args, **kwargs)
        except Ice.MemoryLimitException:
            logger.debug("MemoryLimitException! abort, abort...")
            raise
        except omero.SecurityViolation:
            logger.debug("SecurityViolation, bailing out")
            raise
        except omero.ApiUsageException:
            logger.debug("ApiUsageException, bailing out")
            raise
        except omero.ConcurrencyException:
            logger.debug("ConcurrencyException, bailing out")
            raise
        except Ice.UnknownException:
            logger.debug("UnknownException, bailing out")
            raise
        except Ice.ConnectionLostException:
            logger.debug("ConnectionLostException, bailing out")
            raise
        except Ice.CommunicatorDestroyedException:
            logger.debug("CommunicatorDestroyedException, bailing out")
            raise
        except Ice.Exception, x:
            logger.debug('wrapped ' + f.func_name)
            logger.debug(x.__dict__)
            if hasattr(x, 'serverExceptionClass') and x.serverExceptionClass == 'ome.conditions.InternalException':
                if x.message.find('java.lang.NullPointerException') > 0:
                    logger.debug("NullPointerException, bailing out")
                    raise
                elif x.message.find('Session is dirty') >= 0:
                    logger.debug("Session is dirty, bailing out")
                    raise
                else:
                    logger.debug(x.message)
            logger.debug("exception caught, first time we back off for 10 secs")
            logger.debug(traceback.format_exc())
            #time.sleep(10)
            return inner(*args, **kwargs)
    return wrapped

The handler for Ice.Exception will pick up just about every exception. The one thrown in this case is an serverExceptionClass but it doesn't contain either of those strings but rather "Hibernate session is not re-entrant". Since the exception doesn't match, the inner call at the bottom tries again when it definitely should not. We should most likely fail hard unless this is a case where we know we can retry.

Note: the logging of backing off for 10 seconds should also be adjusted.

comment:4 Changed 13 years ago by wmoore

So, do we simply remove the final

return inner(*args, **kwargs)

or are there times when we do want to retry? Does waiting 10 seconds increase our chance of success in any cases?

comment:5 Changed 13 years ago by wmoore

  • Milestone changed from OMERO-Beta4.3.1 to OMERO-Beta4.3.2
  • Sprint 2011-07-07 (1) deleted

Not going to make big changes to exception handling in 4.3.1, pushing.

comment:6 Changed 13 years ago by wmoore

  • Sprint set to 2011-07-21 (1)

comment:7 Changed 13 years ago by wmoore

  • Owner changed from wmoore to cneves-x
  • Sprint 2011-07-21 (1) deleted

Carlos - When we discussed retrying within Exception handling, Josh thought that retrying should be the exception whereas you had reason to prefer retrying as the norm. This behavior also relates to #5827, so can I pass both these tickets to you?

comment:8 Changed 13 years ago by cneves

  • Owner changed from cneves-x to wmoore

This should have been resolved by #6478, can you double check it?

comment:9 Changed 13 years ago by wmoore

  • Resolution set to fixed
  • Status changed from new to closed

I believe that this issue has gone away with the extensive re-working of safeCallWrap. See #6478

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.90092 sec.)

We're Hiring!