Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11322 (closed)

Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

Bug: Ice3.5 server fails to parse script/share data from DB

Reported by: jamoore Owned by: jamoore
Priority: critical Milestone: OMERO-4.4.9
Component: Deployment Version: 4.4.8
Keywords: ice, binary, version Cc: pwalczysko, rleigh, spli, sbesson, jburel, wmoore, mtbcarroll, bpindelski, hflynn
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: Blocker 4.4.9 (1)

Description

See:

Based on the testing in the spreadsheet, Ice 3.5 servers seem to be having issues parsing the Ice params stored in the database. When a server first parses a script, it uses the available Ice jars to serialize the data and store it as a byte array in Postgres. A later server version must be able with *its* Ice version to deserialize that data. Hopefully it will suffice to set some protocol version setting on the Ice.Communicator which is used to perform the deser.

Note: at the same time, it may make sense to configure all Communicators to use the same protocol, or provide fall back semantics.

Change History (22)

comment:1 Changed 8 years ago by rleigh

Why are we storing this state in the database? Can't we discover this stuff dynamically at server start or on first use of the scripts after server (re)start? If we store it in the db, can this state get out of sync if the script is updated?

comment:2 Changed 8 years ago by jamoore

The state can't get out of sync since it's checksum'ed against the file, but assuming all the scripts execute fairly quickly and there aren't too many, then you're right: we could re-calculate on start and cache the results to disk.

comment:3 Changed 8 years ago by rleigh

If we're storing a checksum, could we store "$checksum$icever" instead? This would force automatic recalculation on ice version changes.

comment:4 Changed 8 years ago by jamoore

  • Owner set to jamoore

comment:5 Changed 8 years ago by jamoore

  • Status changed from new to accepted

comment:6 Changed 8 years ago by jamoore

  • Cc hflynn added

I have a smallish fix for one possible corner case, but by and large I haven't been able to reproduce the "No processor" errors that were seen during testing. (Anyone have an idea?) My guess is that Ice does a pretty good job of handling 1.0 version when at 1.1. However, I think there's no way for Ice to handle 1.1 version at 1.0.

In other words, once you've upgraded to Ice 3.5 you're likely to be stuck there for the moment. Any thoughts on that as a general proposal?

A more complete ticket for scripts will be as, Roger proposed, not storing them in the DB at all. For the shares, however, that won't suffice.

comment:8 follow-up: Changed 8 years ago by jamoore

Error seen when starting ice34 server after ice35:

$ dist/bin/omero admin restart
icegridadmin: server is using a different Slice definition of `::IceGrid::Admin'
icegridadmin: server is using a different Slice definition of `::IceGrid::InternalRegistry'
icegridadmin: server is using a different Slice definition of `::IceGrid::Node'
icegridadmin: server is using a different Slice definition of `::IceGrid::ObjectDescriptor'
icegridadmin: server is using a different Slice definition of `::IceGrid::ReplicaGroupDescriptor'
icegridadmin: server is using a different Slice definition of `::IceGrid::ReplicaSession'
icegridadmin: server is using a different Slice definition of `::IceGrid::Server'
Waiting on shutdown. Use CTRL-C to exit

comment:9 in reply to: ↑ 8 Changed 8 years ago by spli

Replying to jamoore:

Are you definitely using the correct version of the ice binaries?

comment:10 Changed 8 years ago by jamoore

Yes, this is caused by 3.5 data being stored under var/ related to the icegridnode deployment. Then on dropping back down to Ice 3.4, those objects are invalid. In other words, "delete /var on downgrading" though I think the rule is quickly becoming, "no downgrading" if not "no mixing versions, period".

comment:11 Changed 8 years ago by jamoore

Or do I restrict all internal data parsing to the 1.0 version for the moment, and we can make this decision later?

comment:12 follow-up: Changed 8 years ago by spli

"No downgrading" sounds reasonable. Does anyone else support downgrading of anything?

comment:13 Changed 8 years ago by jamoore

  • Resolution set to fixed
  • Status changed from accepted to closed

Added #11454 as a follow-up for the scripts. Closing this in favor of discussion on the PR.

comment:14 in reply to: ↑ 12 Changed 8 years ago by jamoore

Replying to spli:

"No downgrading" sounds reasonable. Does anyone else support downgrading of anything?

Sorry, I missed this before closing, Simon. It's more than just "downgrading" in the sense of PostgreSQL. It means that an OMERO instance would never be able to be moved to a server with an Ice below 3.5. (And even with PG, you can dump to plain and re-import elsewhere, which we wouldn't be providing)

comment:15 Changed 8 years ago by jmoore <josh@…>

(In [9f28b4803b3d6a7412886b6f4b8c08803fa7866d/ome.git] on branch develop) Retry ShareData? parsing with 1.0 encoding (See #11322)

If the first attempt to parse ShareData? from the database
fails (i.e. produces null), then a second attempt is made
with an Ice.Communicator instance initialized with 1.0 as
the Ice.Default.EncodingVersion? value.

Note: Ice.EncodingVersion is not being used because it was
first introduced in 3.5 meaning that this code would not be
compilable with the other zeroc jars.

comment:16 Changed 8 years ago by jmoore <josh@…>

  • Remaining Time set to 0

(In [649d5c2eeda67c6bb5846cf3719057a0388ae5d1/ome.git] on branch develop) Move to strict version 1.0 encoding (Fix #11322)

comment:17 Changed 8 years ago by jmoore <josh@…>

(In [8d0c5b2089d9609e8e6d9a9478041ade711d7536/ome.git] on branch develop) Refactor stream creation into safe helper (Fix #11322)

comment:18 Changed 8 years ago by Josh Moore <josh@…>

(In [db83d8f310c34d4f6edf9359f382b7f3afe770d1/ome.git] on branch develop) Merge pull request #1556 from joshmoore/rebased/develop/11322-ice35-bytes

Enforce 1.0 encoding for scripts/shares (See #11322) (rebased onto develop)

comment:19 Changed 8 years ago by jmoore <josh@…>

(In [501826c49348fb55b646db085a1c918a4dc256ae/ome.git]on branches master, dev_4_4) Retry ShareData? parsing with 1.0 encoding (See #11322)

If the first attempt to parse ShareData? from the database
fails (i.e. produces null), then a second attempt is made
with an Ice.Communicator instance initialized with 1.0 as
the Ice.Default.EncodingVersion? value.

Note: Ice.EncodingVersion is not being used because it was
first introduced in 3.5 meaning that this code would not be
compilable with the other zeroc jars.

comment:20 Changed 8 years ago by jmoore <josh@…>

(In [bea0e0da9b6ec7c2fd6f7f1ae057a37bd9845004/ome.git]on branches master, dev_4_4) Move to strict version 1.0 encoding (Fix #11322)

comment:21 Changed 8 years ago by jmoore <josh@…>

(In [1764294743f871b8ec3c5df2e2e08e1278061f25/ome.git]on branches master, dev_4_4) Refactor stream creation into safe helper (Fix #11322)

comment:22 Changed 8 years ago by Josh Moore <josh@…>

(In [3b2c67610ff72b18d1d83706ada09d3c3440eb23/ome.git]on branches master, dev_4_4) Merge pull request #1479 from joshmoore/11322-ice35-bytes

Enforce 1.0 encoding for scripts/shares (See #11322)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.82247 sec.)

We're Hiring!