Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #2485 (closed)

Opened 12 years ago

Closed 9 years ago

Last modified 9 years ago

Bug: Mage/4064 refuses outside connections

Reported by: jamoore Owned by: jamoore
Priority: critical Milestone: OMERO-4.4
Component: Deployment Version: n.a.
Keywords: n.a. Cc: omero-team@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: 2012-06-19 (17)

Description (last modified by jmoore)

The exception thrown is ConnectTimeoutException which occurs as soon as Blitz attempts to access the proxy (e.g. prx.ice_id()).

This is due to missing bidirectional communication configuration in both the client and the server. Chapter 38 of the ZeroC manual explains both how to do this and its limitations.

The primary limitation for our usage is that the "fixed proxies" which get created are only of use in process. Since OMERO clients don't connect directly to IceStorm via Glacier2, there's no way to reuse the bidirectional connection.

Instead we will initially attempt to use a wrapper, routing the storm communication through the Blitz server and down to the client.

Similar configuration will be needed for all callbacks which come from the client:

components/blitz/resources $ git grep -E '\(.*[a-zA-Z][*].*\)'
omero/API.ice:      void subscribe(string topicName, Object* prx) throws ServerError;
omero/API.ice:      bool keepAlive(ServiceInterface* proxy) throws ServerError;  // SAFE; PASSED OUT OF SERVER
omero/Repositories.ice:            void transfer(string srcPath, Repository* target, string targetPath) // SAFE; PASSSED OUT OF SERVER
omero/Scripts.ice:            void registerCallback(ProcessCallback* cb) throws omero::ServerError;
omero/Scripts.ice:            void unregisterCallback(ProcessCallback* cb) throws omero::ServerError;
omero/Scripts.ice:            void requestRunning(ProcessorCallback* cb);
omero/Scripts.ice:            omero::RMap getResults(Process* proc) throws ServerError; // SAFE; PASSED OUT OF SERVER
omero/SharedResources.ice:                addProcessor(omero::grid::Processor* proc)
omero/SharedResources.ice:                removeProcessor(omero::grid::Processor* proc)

See:

Change History (25)

comment:1 Changed 12 years ago by jmoore

(In [7188]) see #2485 - tracking down server-side timeout

comment:2 Changed 12 years ago by jmoore

(In [7190]) see #2485 - calling del on exceptions during createSession

comment:3 Changed 11 years ago by cxallan

  • Sprint changed from 2010-05-27 (10) to 2010-06-10 (11)

comment:4 Changed 11 years ago by jmoore

  • Description modified (diff)

comment:5 Changed 11 years ago by jmoore

  • Description modified (diff)

comment:6 Changed 11 years ago by jmoore

(In [7279]) Preventing login exceptions. See #2485

comment:7 Changed 11 years ago by jmoore

  • Milestone changed from OMERO-Beta4.2 to Unscheduled
  • Priority changed from critical to blocker
  • Sprint 2010-06-10 (11) deleted

This issue is going to require a significant rework and/or perhaps a patch to Ice. The current limitation is that if the client cannot be seen by the server, then many if not all callbacks will fail. (e.g ClientCallback, ProcessCallback, etc.) Pushing to unscheduled but as "blocker" so that this gets addressed as soon as possible.

comment:8 Changed 11 years ago by jmoore

(In [7452]) Disabling setCallback for 4.2.0 (See #2485, #2558)

comment:10 Changed 11 years ago by jmoore

  • Milestone changed from Unscheduled to OMERO-Beta4.2.1

comment:11 Changed 11 years ago by jmoore

  • Milestone changed from OMERO-Beta4.2.1 to Unscheduled

Moved to 4.2.1 because of a suspected workaround ("createObjectAdapterWithRouter") This, however, was shown to have further Ice-issues. First, Ice.stringToIdentity() on the output of routerPrx.getCategoryForClient() periodically throws an exception requiring one to create the Identity manually. Second, periodic close hangs. Holding off.

comment:12 Changed 10 years ago by jmoore

  • Milestone changed from Unscheduled to OMERO-Beta4.3.2

From http://www.zeroc.com/forums/help-center/5418-bidir-links-cross-nat.html

A server can freely return proxies to a bidirectional client. The only limitation is that all of the proxies must resolve to the same connection; in general, this means the proxies should all refer to servants in the same object adapter. (The goal is usually for the client to have a single connection to the server, and using multiple object adapters in the server would require the client to create multiple connections.) The server must create the proxy for every callback object in the client. As shown in our Ice/bidir example, the client passes the identity of its callback object to the server, and the server creates the proxy that it uses to send callback requests. The server should not be requesting proxies from the client.

This is without Glacier2, but it may be a workaround for the issues that we've run into to

comment:13 Changed 10 years ago by jmoore

  • Milestone changed from OMERO-Beta4.3.2 to OME-5.0

Again, not something we can do in a point release. Pushing.

comment:14 Changed 10 years ago by jmoore

  • Cc omero-team@… added; bwzloranger cxallan jburel removed
  • Remaining Time set to 2.5
  • Summary changed from Mage/4064 refuses outside connections to Bug: Mage/4064 refuses outside connections

Bumping this issue after the sprint6 cleanup since other issues were tangentially related (e.g. #3191). Also marking as a bug so it won't slip through the cracks.

comment:15 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:16 Changed 10 years ago by jmoore

  • Remaining Time changed from 2.5 to 1.5
  • Sprint set to 2012-04-24 (13)
  • Status changed from new to accepted

I've finally got a working demo and can now proceed to modify the OMERO callbacks.

comment:17 Changed 10 years ago by jmoore

  • Remaining Time changed from 1.5 to 0.75

ProcessCallback (scripts) and CmdCallback (graphs) now have proper callbacks working on branch https://github.com/joshmoore/openmicroscopy/tree/2485-callbacks

I've tested both of those callbacks being passed in with Ice 3.3 via SSH port-forwarding and everything works exactly as expected. Other callback scenarios need to also be tested and possibly modified.

comment:18 Changed 10 years ago by jburel

  • Sprint changed from 2012-04-24 (13) to 2012-05-08 (14)

Moved from sprint 2012-04-24 (13)

comment:19 Changed 10 years ago by jburel

  • Sprint changed from 2012-05-08 (14) to 2012-05-22 (15)

Moved from sprint 2012-05-08 (14)

comment:20 Changed 10 years ago by jmoore

  • Remaining Time changed from 0.75 to 0.5
  • Sprint changed from 2012-05-22 (15) to 2012-06-05 (16)

This requires a bit more testing for processors/scripts, but should be quite straight-forward. One thing to remember is that the blocking methods should check one extra time via a poll before throwing LockTimeouts. With that, this should be good to go. Pushing to the next sprint.

comment:21 Changed 10 years ago by jmoore <josh@…>

(In [a6dc3afddb9c6a8edba73be3a068f97e907e8d30/ome.git] on branch develop) Fix modulus logic in HandleI (Fix #8845, See #2485)

If there were too few steps, then an arithmetic exception
was being thrown leading to no callback reaching the client
within the tests expected 10 seconds.

We may should make the timeout try again with a poll
to be sure it wasn't a network or other issue.

comment:22 Changed 9 years ago by jburel

  • Sprint changed from 2012-06-05 (16) to 2012-06-19 (17)

Moved from sprint 2012-06-05 (16)

comment:23 Changed 9 years ago by jmoore

  • Priority changed from blocker to critical
  • Remaining Time changed from 0.5 to 0.25

Script callbacks and Handle callbacks are all now working as expected. Other proxies which are passed (like ClientCallback?) still need to be checked, lowering priority because everything that MUST function is now functioning.

comment:24 Changed 9 years ago by jmoore

  • Remaining Time changed from 0.25 to 0
  • Resolution set to fixed
  • Status changed from accepted to closed

All the callback that we rely on for client functionality are currently working (chgrp, processor). ClientCallback seems not to be working, at least not for heartbeats, but that may be a server-side issue. But, that is not a required feature.

Therefore, I'm closing this as fixed. The only caveat would be that if during the testing of RC1 we realize there are any problems with the DeleteCallbackI, I would suggest we move it to use CmdCallbackI which I have locally.

comment:25 Changed 9 years ago by Jesse Corrington <jesse.corrington@…>

(In [5e1223fe907595f1bcb8682099ace43bda3d52be/ome.git] on branch develop) Make callbacks work across a firewall. see #2485

The object adapter needed to be created with the router, as it's done in Python

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.70017 sec.)

We're Hiring!