Bug: bin/omero sessions list fails on "too many files"

[jamoore]

@develop ~/git/dist $ bin/omero sessions list
Traceback (most recent call last):
  File "bin/omero", line 123, in <module>
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 1184, in argv
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 734, in invoke
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 803, in onecmd
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 883, in execute
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/plugins/sessions.py", line 408, in list
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/util/sessions.py", line 127, in remove
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/path.py", line 602, in text
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/path.py", line 559, in open
IOError: [Errno 24] Too many open files: path('/Users/moore/omero/sessions/localhost/8052_web/._LASTSESS_')

[jmoore]

CC'ing Sebastien in case we need to overall any of the other session code.

[jmoore]

I haven't been able to reproduce even with:

$ ulimit -n 1024
$ for x in `echo {0..100}`; do bin/omero -C -w ome login root@localhost; done
...
$ bin/omero sessions list

It's pretty clear, though, that the closeSession call should be in a try/finally. Adding that and closing this ticket.

[jmoore]

Opened PR - ​https://github.com/openmicroscopy/openmicroscopy/pull/825

[jmoore]

Moved to ​https://github.com/openmicroscopy/openmicroscopy/pull/826

[jmoore <josh@…>]

(In [93e5108102484a91a1b00eee8631a6c192768b64/ome.git] on branch develop) try/finally after store.attach to close sessions (Fix #9331)

[pwalczysko]

I have a similar problem when testing ​https://github.com/openmicroscopy/openmicroscopy/pull/2272. Reopening this ticket, as the output suggests "bin/omero session file" to be a similar culprit as the bin/omero session list noted here ?

[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ bin/omero admin restart
Waiting on shutdown. Use CTRL-C to exit
.No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cd ..
[petr@ls25165 ~/Work/omero5-server]$ ./OMERO-CURRENT/bin/omero login --sudo=admin member-one-ra@localhost
Password for admin:
Created session 60537fd4-b07d-404f-aa98-5bfbd3edb14a (member-one-ra@localhost:4064). Current group: read-annotate-1
[petr@ls25165 ~/Work/omero5-server]$ cd OMERO-CURRENT
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cat $(bin/omero sessions file)
omero.sess=60537fd4-b07d-404f-aa98-5bfbd3edb14a
omero.user=member-one-ra
omero.host=localhost
omero.sudo=admin
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ bin/omero logout
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cd ..
[petr@ls25165 ~/Work/omero5-server]$ ./OMERO-CURRENT/bin/omero login --sudo=root member-one-rw@localhost
Password for root:
InternalException: Failed to connect: Ice.SyscallException:
Too many open files
[petr@ls25165 ~/Work/omero5-server]$ 

[jamoore]

moving to active milestone

[jamoore]

I don't see me getting back to this for 5.0.2, too much else to get out the door.

[bpindelski]

The issue Petr is seeing isn't directly related to the Python interpreter consuming all available file handles. The fact that an Ice.SyscallException? is thrown means that probably somewhere in the login-with-sudo code path a resource leak is happening (maybe a similar issue as in ​http://trac.openmicroscopy.org.uk/ome/ticket/5177?).

[jamoore]

What I had thought was happening, which may either be wrong or only part of the truth, is that after much testing on a shared system like gretzky a large number of sessions are left "active" in ~/omero/sessions. Then when Petr comes along and tries to test (and these sessions have all timed out), he gets the nice error. What src/omero/util/sessions.py is trying to do is actively close each of these sessions.

[pwalczysko]

Replying to jamoore:

What I had thought was happening, which may either be wrong or only part of the truth, is that after much testing on a shared system like gretzky a large number of sessions are left "active" in ~/omero/sessions. Then when Petr comes along and tries to test (and these sessions have all timed out), he gets the nice error. What src/omero/util/sessions.py is trying to do is actively close each of these sessions.

Not sure if this is the right remark, but please note that the output pasted by me above is from my localhost (on my Mac), certainly not on gretzky or any shared machine.

[bpindelski]

I've managed to reproduce it locally:

1. Set ulimit -n 256
2. Use for x in `echo {0..300}`; do bin/omero -C -w omero login root@localhost; done
3. Stop the server
4. Call bin/omero sessions list

It fails with

(omero)[bpindelski dist ((cab81c6...))]$ bin/omero sessions list
Traceback (most recent call last):
  File "bin/omero", line 125, in <module>
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 1242, in argv
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 791, in invoke
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 860, in onecmd
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 940, in execute
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/plugins/sessions.py", line 536, in list
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/util/sessions.py", line 144, in remove
  File "/usr/local/dev/openmicroscopy/dist/lib/python/path.py", line 614, in text
  File "/usr/local/dev/openmicroscopy/dist/lib/python/path.py", line 571, in open
IOError: [Errno 24] Too many open files: path(u'/Users/bpindelski/omero/sessions/localhost/root/._LASTSESS_')

[bpindelski]

I did a re-run of Petr's steps with a dev_5_0 build and can't hit the "Too many open files" error. My previous comment does still apply, but I'm not sure that ulimit -n 256 is a normal setting for an operating system. Bottom line - I can't reproduce Petr's error.

[bpindelski]

After discussing the issue with Josh, we do still need some form of graceful failure when bin/omero sessions list tries to close all invalid sessions (after bin/omero admin stop has been issued). Proposed solutions:

• better cleanup (no concrete idea how to do that yet),
• limiting how many sessions are cleaned in one go (maybe base the limit on the system's ulimit -n value; psutil might be helpful here),
• catch the IOException and skip cleaning while displaying a message.

[bpindelski]

Initial work on ​https://github.com/bpindelski/openmicroscopy/tree/9331_filehandles.

[bpindelski]

PR now open: ​https://github.com/openmicroscopy/openmicroscopy/pull/2635.

[Josh Moore <josh@…>]

(In [c00ceec4244adbb7ee1b20032b03fedd01bbd246/ome.git] on branch develop) Merge pull request #2644 from bpindelski/rebased/develop/9331_filehandles

Avoid file handle exhaustion during sessions listing (see #9331). (rebased onto develop)