Task #11322 (closed)
Bug: Ice3.5 server fails to parse script/share data from DB
Reported by: | jamoore | Owned by: | jamoore |
---|---|---|---|
Priority: | critical | Milestone: | OMERO-4.4.9 |
Component: | Deployment | Version: | 4.4.8 |
Keywords: | ice, binary, version | Cc: | pwalczysko, rleigh, spli, sbesson, jburel, wmoore, mtbcarroll, bpindelski, hflynn |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | 0.0d |
Sprint: | Blocker 4.4.9 (1) |
Description
See:
- https://github.com/openmicroscopy/openmicroscopy/pull/991
- https://docs.google.com/a/glencoesoftware.com/spreadsheet/ccc?key=0AoKiTAl8UOxndG9kcUliT29aY0oyM3huMUxOR19PM3c#gid=2
Based on the testing in the spreadsheet, Ice 3.5 servers seem to be having issues parsing the Ice params stored in the database. When a server first parses a script, it uses the available Ice jars to serialize the data and store it as a byte array in Postgres. A later server version must be able with *its* Ice version to deserialize that data. Hopefully it will suffice to set some protocol version setting on the Ice.Communicator which is used to perform the deser.
Note: at the same time, it may make sense to configure all Communicators to use the same protocol, or provide fall back semantics.
Change History (22)
comment:1 Changed 11 years ago by rleigh
comment:2 Changed 11 years ago by jamoore
The state can't get out of sync since it's checksum'ed against the file, but assuming all the scripts execute fairly quickly and there aren't too many, then you're right: we could re-calculate on start and cache the results to disk.
comment:3 Changed 11 years ago by rleigh
If we're storing a checksum, could we store "$checksum$icever" instead? This would force automatic recalculation on ice version changes.
comment:4 Changed 11 years ago by jamoore
- Owner set to jamoore
comment:5 Changed 11 years ago by jamoore
- Status changed from new to accepted
comment:6 Changed 11 years ago by jamoore
- Cc hflynn added
I have a smallish fix for one possible corner case, but by and large I haven't been able to reproduce the "No processor" errors that were seen during testing. (Anyone have an idea?) My guess is that Ice does a pretty good job of handling 1.0 version when at 1.1. However, I think there's no way for Ice to handle 1.1 version at 1.0.
In other words, once you've upgraded to Ice 3.5 you're likely to be stuck there for the moment. Any thoughts on that as a general proposal?
A more complete ticket for scripts will be as, Roger proposed, not storing them in the DB at all. For the shares, however, that won't suffice.
comment:7 Changed 11 years ago by jamoore
See https://github.com/openmicroscopy/openmicroscopy/pull/1479 for sharedata parsing.
comment:8 follow-up: ↓ 9 Changed 11 years ago by jamoore
Error seen when starting ice34 server after ice35:
$ dist/bin/omero admin restart icegridadmin: server is using a different Slice definition of `::IceGrid::Admin' icegridadmin: server is using a different Slice definition of `::IceGrid::InternalRegistry' icegridadmin: server is using a different Slice definition of `::IceGrid::Node' icegridadmin: server is using a different Slice definition of `::IceGrid::ObjectDescriptor' icegridadmin: server is using a different Slice definition of `::IceGrid::ReplicaGroupDescriptor' icegridadmin: server is using a different Slice definition of `::IceGrid::ReplicaSession' icegridadmin: server is using a different Slice definition of `::IceGrid::Server' Waiting on shutdown. Use CTRL-C to exit
comment:9 in reply to: ↑ 8 Changed 11 years ago by spli
Replying to jamoore:
Are you definitely using the correct version of the ice binaries?
comment:10 Changed 11 years ago by jamoore
Yes, this is caused by 3.5 data being stored under var/ related to the icegridnode deployment. Then on dropping back down to Ice 3.4, those objects are invalid. In other words, "delete /var on downgrading" though I think the rule is quickly becoming, "no downgrading" if not "no mixing versions, period".
comment:11 Changed 11 years ago by jamoore
Or do I restrict all internal data parsing to the 1.0 version for the moment, and we can make this decision later?
comment:12 follow-up: ↓ 14 Changed 11 years ago by spli
"No downgrading" sounds reasonable. Does anyone else support downgrading of anything?
comment:13 Changed 11 years ago by jamoore
- Resolution set to fixed
- Status changed from accepted to closed
Added #11454 as a follow-up for the scripts. Closing this in favor of discussion on the PR.
comment:14 in reply to: ↑ 12 Changed 11 years ago by jamoore
Replying to spli:
"No downgrading" sounds reasonable. Does anyone else support downgrading of anything?
Sorry, I missed this before closing, Simon. It's more than just "downgrading" in the sense of PostgreSQL. It means that an OMERO instance would never be able to be moved to a server with an Ice below 3.5. (And even with PG, you can dump to plain and re-import elsewhere, which we wouldn't be providing)
comment:15 Changed 10 years ago by jmoore <josh@…>
(In [9f28b4803b3d6a7412886b6f4b8c08803fa7866d/ome.git] on branch develop) Retry ShareData? parsing with 1.0 encoding (See #11322)
If the first attempt to parse ShareData? from the database
fails (i.e. produces null), then a second attempt is made
with an Ice.Communicator instance initialized with 1.0 as
the Ice.Default.EncodingVersion? value.
Note: Ice.EncodingVersion is not being used because it was
first introduced in 3.5 meaning that this code would not be
compilable with the other zeroc jars.
comment:16 Changed 10 years ago by jmoore <josh@…>
- Remaining Time set to 0
(In [649d5c2eeda67c6bb5846cf3719057a0388ae5d1/ome.git] on branch develop) Move to strict version 1.0 encoding (Fix #11322)
comment:17 Changed 10 years ago by jmoore <josh@…>
(In [8d0c5b2089d9609e8e6d9a9478041ade711d7536/ome.git] on branch develop) Refactor stream creation into safe helper (Fix #11322)
comment:18 Changed 10 years ago by Josh Moore <josh@…>
(In [db83d8f310c34d4f6edf9359f382b7f3afe770d1/ome.git] on branch develop) Merge pull request #1556 from joshmoore/rebased/develop/11322-ice35-bytes
Enforce 1.0 encoding for scripts/shares (See #11322) (rebased onto develop)
comment:19 Changed 10 years ago by jmoore <josh@…>
(In [501826c49348fb55b646db085a1c918a4dc256ae/ome.git]on branches master, dev_4_4) Retry ShareData? parsing with 1.0 encoding (See #11322)
If the first attempt to parse ShareData? from the database
fails (i.e. produces null), then a second attempt is made
with an Ice.Communicator instance initialized with 1.0 as
the Ice.Default.EncodingVersion? value.
Note: Ice.EncodingVersion is not being used because it was
first introduced in 3.5 meaning that this code would not be
compilable with the other zeroc jars.
comment:20 Changed 10 years ago by jmoore <josh@…>
(In [bea0e0da9b6ec7c2fd6f7f1ae057a37bd9845004/ome.git]on branches master, dev_4_4) Move to strict version 1.0 encoding (Fix #11322)
comment:21 Changed 10 years ago by jmoore <josh@…>
(In [1764294743f871b8ec3c5df2e2e08e1278061f25/ome.git]on branches master, dev_4_4) Refactor stream creation into safe helper (Fix #11322)
comment:22 Changed 10 years ago by Josh Moore <josh@…>
(In [3b2c67610ff72b18d1d83706ada09d3c3440eb23/ome.git]on branches master, dev_4_4) Merge pull request #1479 from joshmoore/11322-ice35-bytes
Enforce 1.0 encoding for scripts/shares (See #11322)
Why are we storing this state in the database? Can't we discover this stuff dynamically at server start or on first use of the scripts after server (re)start? If we store it in the db, can this state get out of sync if the script is updated?