Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #9331 (closed)

Opened 12 years ago

Closed 10 years ago

Last modified 10 years ago

Bug: bin/omero sessions list fails on "too many files"

Reported by: jamoore Owned by: bpindelski
Priority: major Milestone: 5.0.3
Component: OmeroPy Version: 5.0.1
Keywords: n.a. Cc: ux@…, sbesson, spli, jamoore
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: n.a.

Description

@develop ~/git/dist $ bin/omero sessions list
Traceback (most recent call last):
  File "bin/omero", line 123, in <module>
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 1184, in argv
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 734, in invoke
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 803, in onecmd
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/cli.py", line 883, in execute
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/plugins/sessions.py", line 408, in list
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero/util/sessions.py", line 127, in remove
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/path.py", line 602, in text
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/path.py", line 559, in open
IOError: [Errno 24] Too many open files: path('/Users/moore/omero/sessions/localhost/8052_web/._LASTSESS_')

Change History (21)

comment:1 Changed 11 years ago by jmoore

  • Cc sbesson added

CC'ing Sebastien in case we need to overall any of the other session code.

comment:2 Changed 11 years ago by jmoore

  • Remaining Time set to 0.1
  • Sprint set to Bugs Fixing

comment:3 Changed 11 years ago by jmoore

I haven't been able to reproduce even with:

$ ulimit -n 1024
$ for x in `echo {0..100}`; do bin/omero -C -w ome login root@localhost; done
...
$ bin/omero sessions list

It's pretty clear, though, that the closeSession call should be in a try/finally. Adding that and closing this ticket.

comment:4 Changed 11 years ago by jmoore

  • Remaining Time changed from 0.1 to 0
  • Resolution set to fixed
  • Status changed from new to closed

comment:6 Changed 11 years ago by jmoore <josh@…>

(In [93e5108102484a91a1b00eee8631a6c192768b64/ome.git] on branch develop) try/finally after store.attach to close sessions (Fix #9331)

comment:7 Changed 10 years ago by pwalczysko

I have a similar problem when testing https://github.com/openmicroscopy/openmicroscopy/pull/2272. Reopening this ticket, as the output suggests "bin/omero session file" to be a similar culprit as the bin/omero session list noted here ?

[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ bin/omero admin restart
Waiting on shutdown. Use CTRL-C to exit
.No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cd ..
[petr@ls25165 ~/Work/omero5-server]$ ./OMERO-CURRENT/bin/omero login --sudo=admin member-one-ra@localhost
Password for admin:
Created session 60537fd4-b07d-404f-aa98-5bfbd3edb14a (member-one-ra@localhost:4064). Current group: read-annotate-1
[petr@ls25165 ~/Work/omero5-server]$ cd OMERO-CURRENT
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cat $(bin/omero sessions file)
omero.sess=60537fd4-b07d-404f-aa98-5bfbd3edb14a
omero.user=member-one-ra
omero.host=localhost
omero.sudo=admin
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ bin/omero logout
[petr@ls25165 ~/Work/omero5-server/OMERO-CURRENT]$ cd ..
[petr@ls25165 ~/Work/omero5-server]$ ./OMERO-CURRENT/bin/omero login --sudo=root member-one-rw@localhost
Password for root:
InternalException: Failed to connect: Ice.SyscallException:
Too many open files
[petr@ls25165 ~/Work/omero5-server]$ 

comment:8 Changed 10 years ago by pwalczysko

  • Cc ux@… spli added
  • Milestone changed from OMERO-4.4.7 to 5.0.1
  • Resolution fixed deleted
  • Sprint Bugs Fixing deleted
  • Status changed from closed to reopened
  • Version set to 5.0.1

comment:9 Changed 10 years ago by jamoore

  • Milestone changed from 5.0.1 to 5.0.2

moving to active milestone

comment:10 Changed 10 years ago by jamoore

  • Cc jamoore added
  • Milestone changed from 5.0.2 to 5.0.3
  • Owner jamoore deleted
  • Status changed from reopened to accepted

I don't see me getting back to this for 5.0.2, too much else to get out the door.

comment:11 Changed 10 years ago by bpindelski

The issue Petr is seeing isn't directly related to the Python interpreter consuming all available file handles. The fact that an Ice.SyscallException? is thrown means that probably somewhere in the login-with-sudo code path a resource leak is happening (maybe a similar issue as in http://trac.openmicroscopy.org.uk/ome/ticket/5177?).

comment:12 follow-up: Changed 10 years ago by jamoore

What I had thought was happening, which may either be wrong or only part of the truth, is that after much testing on a shared system like gretzky a large number of sessions are left "active" in ~/omero/sessions. Then when Petr comes along and tries to test (and these sessions have all timed out), he gets the nice error. What src/omero/util/sessions.py is trying to do is actively close each of these sessions.

comment:13 in reply to: ↑ 12 Changed 10 years ago by pwalczysko

Replying to jamoore:

What I had thought was happening, which may either be wrong or only part of the truth, is that after much testing on a shared system like gretzky a large number of sessions are left "active" in ~/omero/sessions. Then when Petr comes along and tries to test (and these sessions have all timed out), he gets the nice error. What src/omero/util/sessions.py is trying to do is actively close each of these sessions.

Not sure if this is the right remark, but please note that the output pasted by me above is from my localhost (on my Mac), certainly not on gretzky or any shared machine.

comment:14 Changed 10 years ago by bpindelski

I've managed to reproduce it locally:

  1. Set ulimit -n 256
  2. Use for x in `echo {0..300}`; do bin/omero -C -w omero login root@localhost; done
  3. Stop the server
  4. Call bin/omero sessions list


It fails with

(omero)[bpindelski dist ((cab81c6...))]$ bin/omero sessions list
Traceback (most recent call last):
  File "bin/omero", line 125, in <module>
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 1242, in argv
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 791, in invoke
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 860, in onecmd
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/cli.py", line 940, in execute
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/plugins/sessions.py", line 536, in list
  File "/usr/local/dev/openmicroscopy/dist/lib/python/omero/util/sessions.py", line 144, in remove
  File "/usr/local/dev/openmicroscopy/dist/lib/python/path.py", line 614, in text
  File "/usr/local/dev/openmicroscopy/dist/lib/python/path.py", line 571, in open
IOError: [Errno 24] Too many open files: path(u'/Users/bpindelski/omero/sessions/localhost/root/._LASTSESS_')

comment:15 Changed 10 years ago by bpindelski

I did a re-run of Petr's steps with a dev_5_0 build and can't hit the "Too many open files" error. My previous comment does still apply, but I'm not sure that ulimit -n 256 is a normal setting for an operating system. Bottom line - I can't reproduce Petr's error.

comment:16 Changed 10 years ago by bpindelski

After discussing the issue with Josh, we do still need some form of graceful failure when bin/omero sessions list tries to close all invalid sessions (after bin/omero admin stop has been issued). Proposed solutions:

  • better cleanup (no concrete idea how to do that yet),
  • limiting how many sessions are cleaned in one go (maybe base the limit on the system's ulimit -n value; psutil might be helpful here),
  • catch the IOException and skip cleaning while displaying a message.
Last edited 10 years ago by bpindelski (previous) (diff)

comment:17 Changed 10 years ago by bpindelski

  • Owner set to bpindelski

comment:20 Changed 10 years ago by bpindelski

  • Resolution set to fixed
  • Status changed from accepted to closed

comment:21 Changed 10 years ago by Josh Moore <josh@…>

(In [c00ceec4244adbb7ee1b20032b03fedd01bbd246/ome.git] on branch develop) Merge pull request #2644 from bpindelski/rebased/develop/9331_filehandles

Avoid file handle exhaustion during sessions listing (see #9331). (rebased onto develop)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.64926 sec.)

We're Hiring!