Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #5109 (closed)

Opened 13 years ago

Closed 13 years ago

Last modified 13 years ago

Investigate storing HDF5 values in Lucene

Reported by: jamoore Owned by:
Priority: minor Milestone: OMERO-Beta4.3
Component: General Version: n.a.
Keywords: n.a. Cc: cneves, jburel, cxallan
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: 2011-05-05 (11)

Description (last modified by jmoore)

In order to allow for OmeroSearch access to the OmeroTables values associated with, say, a plate, a FieldBridge? (or other extension to FullTextIndexer) is needed to allow adding a field per column to each image/well/etc.

The major issues will be:

  • efficiently looping through the file v. memory overhead in the indexing process
  • making sure that later modifications on the image/well/etc. still have the data from the plate, though there might not exist an explicit link.

Attachments (2)

attach_table.py​ (1003 bytes) - added by jmoore 13 years ago.
Dirty attachment of a table with ImageColumn? to a dataset
NetCDF.java​ (9.6 KB) - added by jmoore 13 years ago.
Dirty BridgeHelper? extension to parse table

Download all attachments as: .zip

Change History (9)

comment:1 Changed 13 years ago by jmoore

  • Status changed from new to accepted

comment:2 Changed 13 years ago by jmoore

  • Cc cneves-x jburel cxallan added
  • Description modified (diff)

Changed 13 years ago by jmoore

Dirty attachment of a table with ImageColumn? to a dataset

Changed 13 years ago by jmoore

Dirty BridgeHelper? extension to parse table

comment:3 Changed 13 years ago by jmoore

In 4.3, using bin/omero shell --login < attach_table.py after configuring with bin/omero config set omero.search.bridges NetCDF produces:

2011-05-04 10:41:32,121 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.core.Image:Id_355
2011-05-04 10:41:32,142 WARN  [                                  NetCDF] (3-thread-2) Handling annotation ome.model.annotations.FileAnnotation:Id_103
2011-05-04 10:41:32,206 DEBUG [                     ucar.nc2.NetcdfFile] (3-thread-2) Using IOSP ucar.nc2.iosp.hdf5.H5iosp
2011-05-04 10:41:32,255 INFO  [                                  NetCDF] (3-thread-2) Adding foo:0 for row 0
2011-05-04 10:41:32,255 INFO  [                                  NetCDF] (3-thread-2) Adding bar:0 for row 0
2011-05-04 10:41:32,255 INFO  [                                  NetCDF] (3-thread-2) Adding foo:1 for row 1
2011-05-04 10:41:32,255 INFO  [                                  NetCDF] (3-thread-2) Adding bar:1 for row 1
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding foo:2 for row 2
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding bar:2 for row 2
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding foo:3 for row 3
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding bar:3 for row 3
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding foo:4 for row 4
2011-05-04 10:41:32,256 INFO  [                                  NetCDF] (3-thread-2) Adding bar:4 for row 4
2011-05-04 10:41:32,375 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.core.Image:Id_356
2011-05-04 10:41:32,379 WARN  [                                  NetCDF] (3-thread-2) Handling annotation ome.model.annotations.FileAnnotation:Id_103
2011-05-04 10:41:32,379 DEBUG [                     ucar.nc2.NetcdfFile] (3-thread-2) Using IOSP ucar.nc2.iosp.hdf5.H5iosp
2011-05-04 10:41:32,381 INFO  [                                  NetCDF] (3-thread-2) Adding foo:5 for row 5
2011-05-04 10:41:32,381 INFO  [                                  NetCDF] (3-thread-2) Adding bar:5 for row 5
2011-05-04 10:41:32,381 INFO  [                                  NetCDF] (3-thread-2) Adding foo:6 for row 6
2011-05-04 10:41:32,381 INFO  [                                  NetCDF] (3-thread-2) Adding bar:6 for row 6
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding foo:7 for row 7
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding bar:7 for row 7
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding foo:8 for row 8
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding bar:8 for row 8
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding foo:9 for row 9
2011-05-04 10:41:32,382 INFO  [                                  NetCDF] (3-thread-2) Adding bar:9 for row 9
2011-05-04 10:41:32,447 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.containers.Dataset:Id_53
2011-05-04 10:41:32,506 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.core.OriginalFile:Id_253
2011-05-04 10:41:32,554 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.core.OriginalFile:Id_253
2011-05-04 10:41:32,617 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.annotations.FileAnnotation:Id_103
2011-05-04 10:41:32,618 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.annotations.FileAnnotation:Id_103
2011-05-04 10:41:32,618 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.annotations.FileAnnotation:Id_103
2011-05-04 10:41:32,677 INFO  [     ome.services.eventlogs.EventBacklog] (3-thread-2) Added to backlog:ome.model.containers.Dataset:Id_53
2011-05-04 10:41:32,678 WARN  [                                  NetCDF] (3-thread-2) Handling object ome.model.annotations.DatasetAnnotationLink:Id_53
2011-05-04 10:41:32,736 INFO  [   ome.services.fulltext.FullTextIndexer] (3-thread-2) INDEXED 7 objects in 1 batch(es) [722 ms.]
2011-05-04 10:41:36,035 WARN  [                                  NetCDF] (3-thread-3) Handling object ome.model.containers.Dataset:Id_53
2011-05-04 10:41:36,068 INFO  [   ome.services.fulltext.FullTextIndexer] (3-thread-3) INDEXED 1 objects in 1 batch(es) [55 ms.]

comment:4 Changed 13 years ago by jmoore

  • Owner jmoore deleted
  • Status changed from accepted to new

comment:5 Changed 13 years ago by jmoore <josh@…>

  • Remaining Time changed from 0.25 to 0
  • Resolution set to fixed
  • Status changed from new to closed

(In [37487703f645e12563fca92ca8ce09b532f54ea2/ome.git] on branch dev_4_1_custom) Initial TablesBridge implementation (Fix #5109)

comment:6 Changed 13 years ago by jmoore <josh@…>

(In [81b2e8cf1294fa57659082e1aef164ed97063f19/ome.git] on branch dev_4_1_custom) 4.1-specific changes for TablesBridge (See #5109)

comment:7 Changed 13 years ago by jmoore <josh@…>

(In [70fc1b480058d698ebea1ed845711542742e224c/ome.git] on branch develop) Initial TablesBridge implementation (Fix #5109)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.66022 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!