Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #5177 (closed)

Opened 8 years ago

Closed 4 years ago

gatewaytest too many open files

Reported by: wmoore Owned by: cneves
Priority: major Milestone: Unscheduled
Component: OmeroPy Version: OMERO-5.2.0
Keywords: n.a. Cc: omero-team@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: n.a.

Description

Under standard configuration, gatewaytests often fail with too many open files.

256 limit is not enough. 1k is OK.

Change History (14)

comment:1 Changed 8 years ago by jmoore

If we could figure out where the leak is, rather than just modifying the ulimit, that'd be great. It may just be something in the test framework that's not calling close, which is simple to fix and would explain things. But if an Ice.Communicator is being kept alive somewhere deeper in the gateway, it behooves us to find out where.

This may be related to #2007.

(NB: I haven't found the email/ticket yet, but there was one case

comment:2 Changed 8 years ago by wmoore

The only other time I have seen a problem with too many open files was when some bot requested a ton of webemdb pages very quickly. In both cases it may be as Carlos suggested: Not a 'leak' of file handles, but simply they are not getting released fast enough when being created very quickly.

comment:3 Changed 8 years ago by jmoore

  • Milestone changed from Unscheduled to OMERO-Beta4.4
  • Priority changed from minor to major

Moving this into 4.4 with the hope that we find time to find the open file issues with gateway.

comment:4 Changed 7 years ago by jmoore

  • Owner changed from cneves-x to jmoore
  • Remaining Time set to 0.25
  • Sprint set to 2012-04-24 (13)
  • Status changed from new to accepted

Currently have a couple of commits for this. in-progress.

comment:5 Changed 7 years ago by jmoore

  • Owner jmoore deleted
  • Remaining Time changed from 0.25 to 0.5

Situation is improved but still requires a ulimit -n ... call. I temporarily had one commit in place which counted all !omero.client and BlitzGateway creations. For a single file like test/gatewaytest/chmod.py there are 300 of these objects created. My commits on the 2874-chmod branch (PR 139) try to guarantee that more of these objects are cleaned up, but it'd be better if we could refactor the tests to re-use, for example, the root connection. I'm passing this back for anyone else who wants to try to take this the whole way.

commit 03d2b659e420edb6bd56909e48b959e6c473d2c5
Author: jmoore <josh@glencoesoftware.com>
Date:   Thu Apr 19 09:31:29 2012 +0200

    Introduce Counter base class for printing instances (See #5177)

diff --git a/components/tools/OmeroPy/src/omero/clients.py b/components/tools/OmeroPy/src/omero/clients.py
index 6112555..e45df45 100644
--- a/components/tools/OmeroPy/src/omero/clients.py
+++ b/components/tools/OmeroPy/src/omero/clients.py
@@ -26,7 +26,28 @@ import omero_ext.uuid as uuid # see ticket:3774
 IceImport.load("Glacier2_Router_ice")
 import Glacier2
 
-class BaseClient(object):
+class Counter(object):
+
+    COUNTS = dict()
+
+    def __init__(self):
+        key = self.__class__.__name__
+        try:
+            Counter.COUNTS[key] += 1
+        except:
+            Counter.COUNTS[key] = 1
+
+            def onexit():
+                logging.basicConfig(level=logging.ERROR)
+                log = logging.getLogger(key)
+                count = Counter.COUNTS.get(key, 0)
+                log.error("Created %s objects", count)
+
+            import atexit
+            atexit.register(onexit)
+
+
+class BaseClient(Counter):
     """
     Central client-side blitz entry point, and should be in sync with OmeroJava's omero.client
     and OmeroCpp's omero::client.
@@ -78,6 +99,7 @@ class BaseClient(object):
 
         Equivalent to all OmeroJava and OmeroCpp constructors
         """
+        super(BaseClient, self).__init__()
 
         # Setting all protected values to prevent AttributeError
         self.__agent = "OMERO.py" #: See setAgent
diff --git a/components/tools/OmeroPy/src/omero/gateway/__init__.py b/components/tools/OmeroPy/src/omero/gateway/__init__.py
index fbadbf6..ce975f5 100644
--- a/components/tools/OmeroPy/src/omero/gateway/__init__.py
+++ b/components/tools/OmeroPy/src/omero/gateway/__init__.py
@@ -1189,7 +1189,7 @@ class NoProxies (object):
     def __getitem__ (self, k):
         raise Ice.ConnectionLostException
 
-class _BlitzGateway (object):
+class _BlitzGateway (omero.clients.Counter):
     """
     Connection wrapper. Handles connecting and keeping the session alive, creation of various services,
     context switching, security privilidges etc.  

comment:6 Changed 7 years ago by jburel

  • Sprint changed from 2012-04-24 (13) to 2012-05-08 (14)

Moved from sprint 2012-04-24 (13)

comment:7 Changed 7 years ago by jburel

  • Sprint changed from 2012-05-08 (14) to 2012-05-22 (15)

Moved from sprint 2012-05-08 (14)

comment:8 Changed 7 years ago by jmoore <josh@…>

(In [09e735190bc730cd6bdc933447477bfb23d8e1aa/ome.git] on branch develop) Partial fixes for "too many open files" (See #5177)

These two changes already reduced the number of files
left open, but are not complete. Before this change
the chmod.py test wouldn't complete with ulimit -n
1024.

comment:9 Changed 7 years ago by jmoore <josh@…>

(In [ad2b38a5e2ee0fdebc029d39939e0a229f0723a6/ome.git] on branch develop) try/finally blocks for closing resources (See #5177)

Wherever a BlitzGateway? or an omero.client object is created,
there must also be a guaranteed block which calls seppuku
or die respectively.

comment:10 Changed 7 years ago by jburel

  • Sprint changed from 2012-05-22 (15) to 2012-06-05 (16)

Moved from sprint 2012-05-22 (15)

comment:11 Changed 7 years ago by jburel

  • Sprint changed from 2012-06-05 (16) to 2012-06-19 (17)

Moved from sprint 2012-06-05 (16)

comment:12 Changed 7 years ago by wmoore

  • Milestone changed from OMERO-Beta4.4 to Future
  • Owner set to cneves-x
  • Sprint 2012-06-19 (17) deleted

We've discussed the intention to look at this in more detail later...

comment:13 Changed 4 years ago by jamoore

  • Milestone changed from Future to Unscheduled

comment:14 Changed 4 years ago by sbesson

  • Remaining Time changed from 0.5 to 0
  • Resolution set to duplicate
  • Status changed from accepted to closed
  • Version set to OMERO-5.2.0
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.131059 sec.)

We're Hiring!