Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #7325 (closed)

Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

DOC: bin/omero admin restart - failed

Reported by: wmoore Owned by: jamoore
Priority: major Milestone: OMERO-4.4
Component: Deployment Version: n.a.
Keywords: n.a. Cc: cxallan, jburel
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: 2012-01-17 (6)

Description (last modified by jmoore)

After opening my laptop...

See comments for diagnostics and $ ps aux

jrs-macbookpro-25107:OMERO will$ omero admin restart
error: node `master' couldn't be reached:
the node is not active
Was the server already stopped?
Waiting on shutdown. Use CTRL-C to exit
.............................
Failed to shutdown some components after 5 minutes
jrs-macbookpro-25107:OMERO will$ omero admin start
Server already running
jrs-macbookpro-25107:OMERO will$ omero admin diagnostics

See separate but perhaps related issues below for which the solution may be to include retries on certain types of failures.

Change History (10)

comment:1 Changed 8 years ago by wmoore 

jrs-macbookpro-25107:OMERO will$ ps auxw
USER       PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
will     73605   7.2  4.4  4306468 369220   ??  S     3:33pm  14:46.81 /Applications/Firefox.app/Contents/MacOS/firefox -foreground
will     73638   1.7  0.9   641356  71620   ??  S     3:45pm   1:26.52 /Applications/Firefox.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container /Library/Internet Plug-Ins/Flash Player.plugin -greomni /Applicat
_windowserver    70   1.6  3.8  3494576 318352   ??  Ss   14Oct11 334:49.04 /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreGraphics.framework/Resources/WindowServer -daemon
will     73777   0.7  3.5  3182704 289444   ??  S     9:32pm   1:19.62 /usr/bin/java -Xmx256M -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -
will     73781   0.4  4.9  3191024 413692   ??  S     9:32pm   1:21.22 /usr/bin/java -Xmx256M -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0
will       185   0.3  1.0  2885704  85716   ??  R    14Oct11  11:37.40 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_81940
will     73774   0.3  5.5  3526992 463028   ??  S     9:32pm   1:42.65 /usr/bin/java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name
will     74179   0.2  0.6  2561660  53564 s000  S+   11:20pm   1:19.52 /Users/will/apps/OMERO.libs/bin/python manage.py runserver 0.0.0.0:8000
will     73801   0.1  0.1  2446780   8256   ??  Ss    9:32pm   0:05.58 postgres: omero omero 127.0.0.1(61326) idle    
will     73783   0.1  0.5  2553748  38024   ??  S     9:32pm   0:14.57 python lib/python/runTables.py --Ice.Config=/Users/will/Desktop/OMERO/dist/var/master/servers/Tables-0/config/config
will     73800   0.1  0.1  2447548   9724   ??  Ss    9:32pm   0:04.04 postgres: omero omero 127.0.0.1(61323) idle    
will      8408   0.1  0.2   568068  17636   ??  S    19Oct11  35:33.43 /Applications/Snapz Pro X/Snapz Pro X.app/Contents/MacOS/Snapz Pro X -psn_0_1511793
will     73782   0.1  0.5  2553772  38644   ??  S     9:32pm   0:14.61 python lib/python/runProcessor.py --Ice.Config=/Users/will/Desktop/OMERO/dist/var/master/servers/Processor-0/config/config
will     70005   0.0  0.4  2470688  33916 s000  S+   Wed08am   0:00.54 python manage.py runserver 0.0.0.0:8000
will     67695   0.0  0.0  2435548   1132 s004  S    Tue09am   0:00.03 -bash
root     67694   0.0  0.0  2436216   1644 s004  Ss   Tue09am   0:00.03 login -pf will
will     67687   0.0  0.0  2434976    604 s002  S+   Tue09am   0:00.03 more
will     66020   0.0  0.0  2435548   1132 s003  S+   Mon09pm   0:00.08 -bash
root     66019   0.0  0.0  2436216   1644 s003  Ss   Mon09pm   0:00.02 login -pf will
will     63942   0.0  0.0  2435548   1096 s005  S+   Mon09am   0:00.01 -bash
root     63941   0.0  0.0  2436216   1644 s005  Ss   Mon09am   0:00.02 login -pf will
will     63877   0.0  4.5   924692 377068   ??  S    Mon09am  15:56.86 /Applications/TextMate.app/Contents/MacOS/TextMate -psn_0_6608461
will     61264   0.0  0.7  2590216  58636   ??  SNs  Fri09pm   0:32.71 /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker MDSImporterWorker com.apple.Spotlight.ImporterWorke
will     61195   0.0  0.3  2836852  21980   ??  S    Fri09pm   0:19.52 /System/Library/CoreServices/SystemUIServer.app/Contents/MacOS/SystemUIServer
will     58467   0.0  0.2  2830332  15580   ??  S    Fri12pm   0:03.97 /Applications/Stickies.app/Contents/MacOS/Stickies -psn_0_6161888
will     57767   0.0  0.0  2435548   1104 s000  S    Fri09am   0:00.01 -bash
root     57766   0.0  0.0  2436216   1644 s000  Ss   Fri09am   0:00.01 login -pf will
will     40085   0.0  1.3   692384 111228   ??  S     9Nov11  26:09.98 /Applications/Adium.app/Contents/MacOS/Adium -psn_0_4809878
will     38850   0.0  0.1  2447652   7076   ??  Ss    7Nov11   0:00.05 postgres: will omero [local] idle    
will     38848   0.0  0.0  2436880   1992 s002  S+    7Nov11   0:00.04 psql omero
_www     28053   0.0  0.0  2438072    584   ??  S    31Oct11   0:00.00 /usr/sbin/httpd -D FOREGROUND
_www     28044   0.0  0.0  2438072   1464   ??  S    31Oct11   0:00.01 /usr/sbin/httpd -D FOREGROUND
root     28040   0.0  0.0  2438072   1996   ??  Ss   31Oct11   0:19.30 /usr/sbin/httpd -D FOREGROUND
will     16596   0.0  0.0  2435548    772 s002  S    25Oct11   0:00.19 -bash
root     16595   0.0  0.0  2436216    752 s002  Ss   25Oct11   0:00.01 login -pf will
will      7864   0.0  0.0  2435548    968 s001  S    19Oct11   0:03.86 -bash
root      7863   0.0  0.0  2436216    752 s001  Ss   19Oct11   0:00.01 login -pf will
will       682   0.0  2.3  4147296 189304   ??  S    14Oct11  45:49.43 /Applications/Mail.app/Contents/MacOS/Mail -psn_0_114716
will       336   0.0  0.0  2485208   1568   ??  S    14Oct11   0:04.60 /usr/bin/ssh-agent -l
will       284   0.0  0.2  2529632  12900   ??  S    14Oct11   1:31.89 /System/Library/Services/AppleSpell.service/Contents/MacOS/AppleSpell -psn_0_106522
will       167   0.0  0.0  2441760    404   ??  Ss   14Oct11   0:59.11 postgres: stats collector process       
will       166   0.0  0.0  2445860   3900   ??  Ss   14Oct11   0:40.29 postgres: autovacuum launcher process       
will       165   0.0  0.0  2445728    384   ??  Ss   14Oct11   2:30.23 postgres: wal writer process       
will       164   0.0  0.0  2445728   3436   ??  Ss   14Oct11   3:31.57 postgres: writer process       
root       155   0.0  0.0   427400   2804   ??  S    14Oct11   0:12.09 /Library/KeyAccess/KeyAccess.app/Contents/SharedSupport/kass.app/Contents/MacOS/kass -psn_0_65552
will       151   0.0  0.0  2771432   1788   ??  S    14Oct11   0:05.67 /System/Library/CoreServices/RemoteManagement/AppleVNCServer.bundle/Contents/MacOS/AppleVNCServer
will       145   0.0  0.0  2445728   1364   ??  S    14Oct11   0:09.42 /usr/local/Cellar/postgresql/9.0.4/bin/postgres -D /usr/local/var/postgres -r /usr/local/var/postgres/server.log
will       143   0.0  0.1   431464   5240   ??  S    14Oct11   2:08.26 /Library/KeyAccess/KeyAccess.app/Contents/MacOS/KeyAccess
will       142   0.0  0.0   464856   1932   ??  S    14Oct11   0:42.65 /Library/Printers/Canon/UFR2/Utilities/UFR II BackGrounder.app/Contents/MacOS/UFR II BackGrounder
will       141   0.0  0.0  2809256   3200   ??  S    14Oct11   0:05.47 /System/Library/CoreServices/AirPort Base Station Agent.app/Contents/MacOS/AirPort Base Station Agent -launchd -allowquit
will       136   0.0  0.1  2790592   7180   ??  S    14Oct11   1:20.89 /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/MacOS/ARDAgent
will       133   0.0  0.1  2798916   7740   ??  S    14Oct11   0:10.15 /usr/libexec/UserEventAgent -l Aqua
will       126   0.0  0.2  2754492  20016   ??  S    14Oct11   0:59.14 /System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/Support/fontd
will       124   0.0  0.0  2436088    552   ??  S    14Oct11   0:00.02 /usr/sbin/pboard
will       122   0.0  1.4  3060200 116524   ??  S    14Oct11   9:47.25 /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder
will       120   0.0  0.3  2887940  27276   ??  S    14Oct11  33:22.42 /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
will       116   0.0  0.0  2511668   1364   ??  Ss   14Oct11   0:36.72 /sbin/launchd
_coreaudiod    99   0.0  0.1  2517712   8480   ??  Ss   14Oct11   0:23.40 /usr/sbin/coreaudiod
root        85   0.0  0.0  2444508    864   ??  Ss   14Oct11   0:00.07 /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/cvmsServ
root        52   0.0  0.0   160460   3656   ??  Ss   14Oct11   0:27.35 /Library/KeyAccess/KeyAccess.app/Contents/SharedSupport/karl.app/Contents/MacOS/karl -c
root        49   0.0  0.0  2461692    820   ??  Ss   14Oct11   0:03.70 autofsd
root        43   0.0  0.0  2434864    496   ??  Ss   14Oct11   0:00.11 /sbin/dynamic_pager -F /private/var/vm/swapfile
root        41   0.0  0.1  2510044   5188   ??  Ss   14Oct11   1:52.84 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/Support/fseventsd
root        40   0.0  0.0  2485172   1868   ??  Ss   14Oct11  31:20.65 /usr/libexec/hidd
root        38   0.0  0.0  2486364    984   ??  Ss   14Oct11   0:09.28 /usr/sbin/KernelEventAgent
will        37   0.0  0.1  2850012   9908   ??  Ss   14Oct11   2:12.81 /System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow console
root        36   0.0  2.1  2921032 173580   ??  Ss   14Oct11  21:49.53 /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Support/mds
root        34   0.0  0.0  2444888    688   ??  Ss   14Oct11   0:00.01 /System/Library/CoreServices/RemoteManagement/AppleVNCServer.bundle/Contents/Support/RFBRegisterMDNS
root        30   0.0  0.0  2466304    752   ??  Ss   14Oct11   0:04.35 /sbin/SystemStarter
_usbmuxd    29   0.0  0.0  2474744   1232   ??  Ss   14Oct11   0:35.67 /System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/Resources/usbmuxd -launchd
root        26   0.0  0.0  2435292    852   ??  Ss   14Oct11   1:00.91 /usr/sbin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift
root        24   0.0  0.5  2566212  45940   ??  Ss   14Oct11   1:19.55 /System/Library/CoreServices/coreservicesd
root        22   0.0  0.1  2515260   4688   ??  Ss   14Oct11   0:08.01 /usr/sbin/securityd -i
_mdnsresponder    18   0.0  0.0  2516888   3036   ??  Ss   14Oct11   0:46.69 /usr/sbin/mDNSResponder -launchd
daemon      17   0.0  0.0  2508260   2776   ??  Ss   14Oct11   0:37.59 /usr/sbin/distnoted
root        16   0.0  0.0  2522832   3256   ??  Ss   14Oct11   0:22.75 /usr/sbin/blued
root        15   0.0  0.0  2512552    924   ??  Ss   14Oct11   0:07.14 /usr/sbin/syslogd
root        14   0.0  0.0  2526000   4100   ??  Ss   14Oct11   1:37.81 /usr/libexec/configd
root        13   0.0  0.0  2508408   1924   ??  Ss   14Oct11   0:05.98 /usr/sbin/diskarbitrationd
root        12   0.0  0.0  2467152    624   ??  Ss   14Oct11   0:11.27 /usr/sbin/notifyd
root        11   0.0  0.1  2517180   7468   ??  Rs   14Oct11   1:43.04 /usr/sbin/DirectoryService
root        10   0.0  0.2  2514496  12784   ??  Ss   14Oct11   0:20.96 /usr/libexec/kextd
root         1   0.0  0.0  2512136   1496   ??  Ss   14Oct11  30:24.66 /sbin/launchd
root     74715   0.0  0.0  2434868    564 s001  R+    8:57am   0:00.00 ps auxw
root      8431   0.0  0.0        0      0   ??  Z    19Oct11   0:00.00 (tee)
will     74711   0.0  0.1  2446116   5236   ??  Ss    8:56am   0:00.01 postgres: omero omero 127.0.0.1(64478) idle    
will     74679   0.0  1.4  3733348 119016   ??  S     8:52am   0:03.71 /Users/will/Desktop/OMERO.clients-Beta4.3.3.mac/OMERO.insight.app/Contents/MacOS/JavaApplicationStub -psn_0_7604032
_spotlight 73883   0.0  0.1  2525764   6376   ??  SNs   9:58pm   0:00.30 /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker MDSImporterWorker com.apple.Spotlight.ImporterWorke
will     73815   0.0  0.1  2811012   9320   ??  S     9:35pm   0:01.84 /Applications/Preview.app/Contents/MacOS/Preview -psn_0_7558965
will     73780   0.0  0.1  2517296   5924   ??  S     9:32pm   0:00.26 icebox --Ice.Config=/Users/will/Desktop/OMERO/dist/var/master/servers/OMERO.IceStorm/config/config
will     73779   0.0  0.1  2508896   4580   ??  S     9:32pm   0:00.72 glacier2router --Ice.Config=/Users/will/Desktop/OMERO/dist/var/master/servers/OMERO.Glacier2/config/config
will     73776   0.0  0.1  2481960  12148   ??  S     9:32pm   0:01.29 python lib/python/fsServerFS.py --Ice.Config=/Users/will/Desktop/OMERO/dist/var/master/servers/FileServer/config/config
will     73772   0.0  0.1  2526880   8416   ??  S     9:32pm   0:00.85 icegridnode --daemon --pidfile /Users/will/Desktop/OMERO/dist/var/master/master.pid --nochdir --Ice.Config=/Users/will/Desktop/OMERO/dist/etc/internal.cfg,/Use
will     72441   0.0  0.0  2438800   3696 s004  S+   10:40am   0:00.02 python
will     72146   0.0  0.3  2836016  25680   ??  S     9:29am   0:01.07 /Applications/iCal.app/Contents/MacOS/iCal -psn_0_7362309
will     71056   0.0  1.3  3969772 109708   ??  S    Wed03pm   0:30.53 /System/Library/PrivateFrameworks/WebKit2.framework/WebProcess.app/Contents/MacOS/WebProcess /System/Library/PrivateFrameworks/WebKit2.framework/WebKit2 -type 
will     71054   0.0  0.6  2903456  49332   ??  S    Wed03pm   0:39.86 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_7227108

comment:2 Changed 8 years ago by wmoore

  • Owner set to jmoore 
================================================================================
OMERO Diagnostics 4.3.3
================================================================================
        
Commands:   java -version                  1.6.0     (/usr/bin/java)
Commands:   python -V                      2.6.1     (/Users/will/apps/OMERO.libs/bin/python -- 2 others)
Commands:   icegridnode --version          3.3.1     (/Users/will/apps/OMERO.libs/bin/icegridnode)
Commands:   icegridadmin --version         3.3.1     (/Users/will/apps/OMERO.libs/bin/icegridadmin)
Commands:   psql --version                 9.0.4     (/usr/local/bin/psql)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 73774, enabled)
Server:     DropBox                        inactive (disabled)
Server:     FileServer                     active (pid = 73776, enabled)
Server:     Indexer-0                      active (pid = 73777, enabled)
Server:     MonitorServer                  inactive (disabled)
Server:     OMERO.Glacier2                 active (pid = 73779, enabled)
Server:     OMERO.IceStorm                 active (pid = 73780, enabled)
Server:     PixelData-0                    active (pid = 73781, enabled)
Server:     Processor-0                    active (pid = 73782, enabled)
Server:     Tables-0                       active (pid = 73783, enabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /Users/will/Desktop/OMERO/dist/var/log exists

Log files:  Blitz-0.log                    29.0 MB       errors=55   warnings=541 
Log files:  DropBox.log                    12.0 KB       errors=10   warnings=0   
Log files:  FileServer.log                 3.0 KB       
Log files:  Indexer-0.log                  42.0 KB      
Log files:  MonitorServer.log              8.0 KB        errors=10   warnings=0   
Log files:  OMEROweb.log                   1.0 MB        errors=218  warnings=1   
Log files:  PixelData-0.log                25.0 KB       errors=1    warnings=0   
Log files:  Processor-0.log                874.0 KB      errors=0    warnings=329 
Log files:  Tables-0.log                   224.0 KB      errors=0    warnings=319 
Log files:  TestDropBox.log                n/a
Log files:  master.err                     0.0 KB       
Log files:  master.out                     0.0 KB       
Log files:  Total size                     32.07 MB

Parsing Blitz-0.log:[line:30] => Server restarted <= 
Parsing Blitz-0.log:[line:50225] => Server restarted <= 
Parsing Blitz-0.log:[line:53714] => Server restarted <= 
Parsing Blitz-0.log:[line:64630] => Server restarted <= 
Parsing Blitz-0.log:[line:79985] => Server restarted <= 
Parsing Blitz-0.log:[line:85793] => Server restarted <= 
Parsing Blitz-0.log:[line:96678] => Server restarted <= 
Parsing Blitz-0.log:[line:100911] => Server restarted <= 
Parsing Blitz-0.log:[line:170752] => Server restarted <= 
Parsing Blitz-0.log:[line:172077] => Server restarted <= 

Environment:OMERO_HOME=/Users/will/Desktop/OMERO/dist 
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:PATH=/Users/will/apps/OMERO.libs/Cellar/zeroc-ice33/bin:/usr/local/bin:/Users/will/apps/OMERO.libs/bin:/usr/local/lib/node_modules:/Users/will/Desktop/OMERO/dist/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/Applications/gitflow/ 
Environment:ICE_HOME=/Users/will/apps/OMERO.libs/Cellar/zeroc-ice33 
Environment:LD_LIBRARY_PATH=(unset)        
Environment:DYLD_LIBRARY_PATH=:/Users/will/apps/OMERO.libs/lib 

OMERO data dir: '/OMERO'	Exists? True	Is writable? True
OMERO.web status... Traceback (most recent call last):
  File "/Users/will/Desktop/OMERO/dist/bin/omero", line 123, in <module>
    rv = omero.cli.argv()
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/cli.py", line 1172, in argv
    cli.invoke(args[1:])
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/cli.py", line 722, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/cli.py", line 791, in onecmd
    self.execute(line, previous_args)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/cli.py", line 871, in execute
    args.func(args)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/plugins/prefs.py", line 67, in open_and_close_config
    return func(*args, **kwargs)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/plugins/admin.py", line 780, in diagnostics
    WebControl().status(args)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/plugins/web.py", line 391, in status
    import omeroweb.settings as settings
ImportError: No module named omeroweb.settings
jrs-macbookpro-25107:OMERO will$ omero config get
omero.web.application_server=development
omero.web.debug=True

comment:3 Changed 8 years ago by wmoore

Another restart problem. This is some work at home with a dodgy internect connection - probably caused the need to restart with IP address change? Then tried restarting the next morning at work (last 2 restarts). 

jrs-macbookpro-25107:OMERO will$ omero admin restart
error: node `master' couldn't be reached:
the node is not active
Was the server already stopped?
Waiting on shutdown. Use CTRL-C to exit
.............................
Failed to shutdown some components after 5 minutes
jrs-macbookpro-25107:OMERO will$ omero admin stop
Waiting on shutdown. Use CTRL-C to exit
.jrs-macbookpro-25107:OMERO will$ omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
jrs-macbookpro-25107:OMERO will$ omero config set omero.sessions.timeout 20000
jrs-macbookpro-25107:OMERO will$ omero admin restart
Waiting on shutdown. Use CTRL-C to exit
.No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
jrs-macbookpro-25107:OMERO will$ omero admin restart
Waiting on shutdown. Use CTRL-C to exit
.No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
jrs-macbookpro-25107:OMERO will$ tail -f dist/var/log/Blitz-0.log 
2011-12-07 23:07:39,004 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Args:	[select distinct obj from Experimenter as obj left outer join fetch obj.groupExperimenterMap as map left outer join fetch map.parent g where obj.id in (:ids), PARAMS:ids=ArrayList(1) ]
2011-12-07 23:07:39,005 INFO  [         ome.security.basic.EventHandler] (l.Server-9)  Auth:	user=2,group=3,event=null(User),sess=5ef34d52-1c6f-4020-b3d6-8567e386ac98
2011-12-07 23:07:39,009 INFO  [                 org.perf4j.TimingLogger] (l.Server-9) start[1323299259004] time[5] tag[omero.call.success.ome.logic.QueryImpl.findByQuery]
2011-12-07 23:07:39,009 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Rslt:	ome.model.meta.Experimenter:Id_2
2011-12-07 23:07:39,013 INFO  [e.services.sessions.SessionContext$Count] (l.Server-5) -Reference count: 14831435-ff56-46b6-97b3-1f7db16e0847=0
2011-12-07 23:07:39,014 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Meth:	interface ome.api.IAdmin.lookupGroup
2011-12-07 23:07:39,014 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Args:	[user]
2011-12-07 23:07:39,015 INFO  [         ome.security.basic.EventHandler] (l.Server-0)  Auth:	user=2,group=3,event=null(User),sess=5ef34d52-1c6f-4020-b3d6-8567e386ac98
2011-12-07 23:07:39,039 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1323299259014] time[25] tag[omero.call.success.ome.logic.AdminImpl.lookupGroup]
2011-12-07 23:07:39,039 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Rslt:	ome.model.meta.ExperimenterGroup:Id_1
2011-12-07 23:08:30,035 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Executor.doWork -- ome.services.sessions.SessionManagerImpl.createSession
2011-12-07 23:08:30,035 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Args:	[null, InternalSF@1225998403]
2011-12-07 23:08:30,036 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Executor.doWork -- ome.services.sessions.SessionManagerImpl.createSession
2011-12-07 23:08:30,036 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Args:	[null, InternalSF@1225998403]
2011-12-07 23:08:30,046 INFO  [         ome.security.basic.EventHandler] (l.Server-1)  Auth:	user=0,group=0,event=51131(Sessions),sess=8ef97cbc-3a22-43e7-a3b6-4905dd252829
2011-12-07 23:08:30,046 INFO  [         ome.security.basic.EventHandler] (l.Server-2)  Auth:	user=0,group=0,event=51132(Sessions),sess=8ef97cbc-3a22-43e7-a3b6-4905dd252829
2011-12-07 23:08:30,057 INFO  [       ome.security.basic.CurrentDetails] (l.Server-1) Adding log:INSERT,class ome.model.meta.Session,14271
2011-12-07 23:08:30,057 INFO  [       ome.security.basic.CurrentDetails] (l.Server-2) Adding log:INSERT,class ome.model.meta.Session,14272
2011-12-07 23:08:30,067 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1323299310035] time[32] tag[omero.call.success.ome.services.sessions.SessionManagerImpl$2.doWork]
2011-12-07 23:08:30,067 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Rslt:	(ome.model.meta.Experimenter:Id_0, ome.model.meta.ExperimenterGroup:Id_0, [0, 1], ... 4 more)
2011-12-07 23:08:30,067 INFO  [                 org.perf4j.TimingLogger] (l.Server-2) start[1323299310036] time[31] tag[omero.call.success.ome.services.sessions.SessionManagerImpl$2.doWork]
2011-12-07 23:08:30,067 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Rslt:	(ome.model.meta.Experimenter:Id_0, ome.model.meta.ExperimenterGroup:Id_0, [0, 1], ... 4 more)
2011-12-07 23:08:30,068 INFO  [ ome.services.blitz.fire.SessionManagerI] (l.Server-1) Created session ServiceFactoryI(session-4c1a3040-c157-4aec-aa5a-3d75e05ada8e/e015e67d-5b49-4302-82c4-14a8fc08ba77) for user root (agent=Python service)
2011-12-07 23:08:30,068 INFO  [ ome.services.blitz.fire.SessionManagerI] (l.Server-2) Created session ServiceFactoryI(session-ee18ac57-d52b-4e94-a55d-f62509fed456/55102c50-27c8-4d03-8b15-276da0e99186) for user root (agent=Python service)
2011-12-07 23:08:30,069 INFO  [ ome.services.blitz.impl.ServiceFactoryI] (l.Server-0) Added servant to adapter: 55102c50-27c8-4d03-8b15-276da0e99186/ee18ac57-d52b-4e94-a55d-f62509fed456omero.grid.SharedResources(omero.grid._SharedResourcesTie@49e3c996)
2011-12-07 23:08:53,833 INFO  [ ome.services.blitz.impl.ServiceFactoryI] (l.Server-7) Added servant to adapter: 5ef34d52-1c6f-4020-b3d6-8567e386ac98/f8e4838e-517e-4c8e-8053-d0bf21a69da9omero.api.IConfig(omero.api._IConfigTie@1833b96c)
2011-12-07 23:09:00,018 INFO  [ ome.services.blitz.fire.SessionManagerI] (3-thread-1) Performing requestHeartbeats
^C
jrs-macbookpro-25107:OMERO will$ git status
# On branch ajax_helper
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   components/tools/OmeroWeb/omeroweb/common/static/common/javascript/popup.js
#	modified:   components/tools/OmeroWeb/omeroweb/webclient/templates/webclient/annotations/metadata_general.html
#
no changes added to commit (use "git add" and/or "git commit -a")
jrs-macbookpro-25107:OMERO will$ omero admin restart
Waiting on shutdown. Use CTRL-C to exit
......No descriptor given. Using etc/grid/default.xml
icegridnode: failure occurred in daemon:
service caught unhandled Ice exception:
FileUtil.cpp:477: IceUtil::FileLockedException:
could not lock file: `var/registry/__Freeze/lock'
syscall exception: Resource temporarily unavailable
jrs-macbookpro-25107:OMERO will$ omero admin restart
Server not running
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
jrs-macbookpro-25107:OMERO will$

comment:4 Changed 8 years ago by wmoore

master.err for restart above

jrs-macbookpro-25107:OMERO will$ cat dist/var/log/master.err 
-! 12/2/11 09:26:37:548 warning: Blitz-0-Ice.ThreadPool.Server-6: dispatch exception:
   identity: session-aba9b137-b6ac-4baf-84f9-d6e254edc562/a4a6d8ac-fd16-4306-bc09-d846d39fb82e
   facet: 
   operation: destroy
   remote host: 10.12.2.172 remote port: 61057
   Ice.ObjectAdapterDeactivatedException
       name = "BlitzAdapter"
   	at Ice.ObjectAdapterI.checkForDeactivation(ObjectAdapterI.java:1121)
   	at Ice.ObjectAdapterI.findFacet(ObjectAdapterI.java:505)
   	at Ice.ObjectAdapterI.find(ObjectAdapterI.java:499)
   	at ome.services.blitz.impl.ServiceFactoryI.unregisterServant(ServiceFactoryI.java:1005)
   	at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:801)
   	at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:703)
   	at Glacier2._SessionDisp.___destroy(_SessionDisp.java:104)
   	at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1412)
   	at IceInternal.Incoming.invoke(Incoming.java:159)
   	at Ice.ConnectionI.invokeAll(ConnectionI.java:2357)
   	at Ice.ConnectionI.dispatch(ConnectionI.java:1208)
   	at Ice.ConnectionI.message(ConnectionI.java:1163)
   	at IceInternal.ThreadPool.run(ThreadPool.java:302)
   	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
   	at java.lang.Thread.run(Thread.java:680)
   
-! 12/2/11 09:26:37:548 warning: Blitz-0-Ice.ThreadPool.Server-5: dispatch exception:
   identity: session-d1004155-2312-4e2d-9042-a3d79f83a18a/59d84cf8-ee11-44a3-9b02-0391ccc4cca8
   facet: 
   operation: destroy
   remote host: 10.12.2.172 remote port: 61054
   Ice.ObjectAdapterDeactivatedException
       name = "BlitzAdapter"
   	at Ice.ObjectAdapterI.checkForDeactivation(ObjectAdapterI.java:1121)
   	at Ice.ObjectAdapterI.findFacet(ObjectAdapterI.java:505)
   	at Ice.ObjectAdapterI.find(ObjectAdapterI.java:499)
   	at ome.services.blitz.impl.ServiceFactoryI.unregisterServant(ServiceFactoryI.java:1005)
   	at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:801)
   	at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:703)
   	at Glacier2._SessionDisp.___destroy(_SessionDisp.java:104)
   	at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1412)
   	at IceInternal.Incoming.invoke(Incoming.java:159)
   	at Ice.ConnectionI.invokeAll(ConnectionI.java:2357)
   	at Ice.ConnectionI.dispatch(ConnectionI.java:1208)
   	at Ice.ConnectionI.message(ConnectionI.java:1163)
   	at IceInternal.ThreadPool.run(ThreadPool.java:302)
   	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
   	at java.lang.Thread.run(Thread.java:680)
   
!! 12/03/11 21:01:41.518 icegridnode: error: service caught unhandled Ice exception:
   FileUtil.cpp:477: IceUtil::FileLockedException:
   could not lock file: `var/registry/__Freeze/lock'
   syscall exception: Resource temporarily unavailable
-! 12/04/11 08:08:59.944 OMERO.Glacier2: warning: dispatch exception: Network.cpp:1178: Ice::ConnectFailedException:
   connect failed: No route to host
   identity: session-71ca903c-cd65-49b5-a224-b60676209eec/8322950c-2c74-4bda-9810-f9398c25d47d
   facet: 
   operation: keepAllAlive
   remote host: 127.0.0.1 remote port: 51739
-! 12/04/11 08:08:59.955 OMERO.Glacier2: warning: dispatch exception: Network.cpp:1178: Ice::ConnectFailedException:
   connect failed: No route to host
   identity: session-63dbf2f0-9c1e-4425-b624-d09f08e0b1ea/8322950c-2c74-4bda-9810-f9398c25d47d
   facet: 
   operation: keepAllAlive
   remote host: 127.0.0.1 remote port: 51742
-! 12/04/11 08:10:04.957 OMERO.Glacier2: warning: dispatch exception: Network.cpp:1184: Ice::SocketException:
   socket exception: Host is down
   identity: session-71ca903c-cd65-49b5-a224-b60676209eec/8322950c-2c74-4bda-9810-f9398c25d47d
   facet: 
   operation: keepAllAlive
   remote host: 127.0.0.1 remote port: 51739
-! 12/04/11 08:10:04.958 OMERO.Glacier2: warning: dispatch exception: Network.cpp:1184: Ice::SocketException:
   socket exception: Host is down
   identity: session-63dbf2f0-9c1e-4425-b624-d09f08e0b1ea/8322950c-2c74-4bda-9810-f9398c25d47d
   facet: 
   operation: keepAllAlive
   remote host: 127.0.0.1 remote port: 51742
!! 12/08/11 09:43:45.229 icegridnode: error: service caught unhandled Ice exception:
   FileUtil.cpp:477: IceUtil::FileLockedException:
   could not lock file: `var/registry/__Freeze/lock'
   syscall exception: Resource temporarily unavailable

comment:5 Changed 8 years ago by jmoore

  • Description modified (diff)

comment:6 Changed 8 years ago by jmoore

  • Cc cxallan added

Working through this, Will, the only way that this print out can take place is if "ping node" passes but "shutdown node" fails. I could capture the "node is not active" message (not terribly robust) and then what? Do we offer to the user how to kill the process?

  The process "5825" cannot be stopped. Use "kill 5825" ...

Thoughts? Adding Chris to the conversation.

comment:7 Changed 8 years ago by jmoore

  • Cc jburel added
  • Sprint set to 2012-01-17 (6)
  • Status changed from new to accepted
  • Summary changed from Bug: bin/omero admin restart - failed to DOC: bin/omero admin restart - failed

Like #5576, this is a developer only issue. Adding to FAQ as solution for both of these.

comment:8 Changed 8 years ago by jmoore

  • Resolution set to fixed
  • Status changed from accepted to closed

Added to ​troubleshooting.

comment:9 Changed 8 years ago by jmoore <josh@…>

(In [b04ee7b866e2b94e33bd9ec66581e3bb5b3b7c43/ome.git] on branch develop) Failing test shutdown (See #7325)

comment:10 Changed 8 years ago by jmoore <josh@…>

(In [d84045cfcd7a5155dec4b7aea1ee8a25f64577cc/ome.git] on branch develop) Try adding restart wait (See #7325)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.83976 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!