Task #7711 (new)
Opened 12 years ago
Last modified 8 years ago
Bug: slow loading on large number of table columns
Reported by: | jamoore | Owned by: | jamoore |
---|---|---|---|
Priority: | major | Milestone: | Unscheduled |
Component: | Performance | Version: | 4.4.9 |
Keywords: | TAB | Cc: | bhcho@…, spli, ux@… |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | 1.5d |
Sprint: | n.a. |
Description
When reading (table.read(allColumns, 0, numOfRows)) a table of 169 columns and 4000 rows, reading all at the same time can lead to ~6 second delays:
Columns ROWS 1 16 84 152 169 1 0.01 0.01 0.02 0.02 0.02 400 0.02 0.07 0.34 0.62 0.70 2000 0.04 0.31 1.66 3.04 3.38 3600 0.08 0.54 3.02 5.43 6.15 4000 0.08 0.61 3.62 5.93 6.67
Times logged to var/log/Tables-0.log look much better:
2012-01-02 15:22:54,141 INFO [ omero.remote] (Dummy-5 ) Meth: TableI.read 2012-01-02 15:22:54,141 INFO [ omero.tables.TableI] (Dummy-5 ) Table-175FA4C8-5459-4756-92D2-B671BD67C938.read([0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L , 8L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L, 41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 51L, 52L, 53L, 54L, 55L, 56L, 57L, 58L, 59L, 60L, 61L, 62L, 63L, 64L, 65L, 66L, 67L, 68L, 69L, 70L, 71L, 72L, 73L, 74L, 75L, 76L, 77L, 78L, 79L, 80L, 81L, 82L, 83L, 84L, 85L, 86L, 87L, 88L, 89L, 90L, 91L, 92L, 93L, 94L, 95L, 96L, 97L, 98L, 99L, 100L, 101L, 102L, 103L, 104L, 105L, 106L, 107L, 108L, 109L, 110L, 111L, 112L, 113L, 114L, 115L, 116L, 117L, 118L, 119L, 120L, 121L, 122L, 123L, 124L, 125L, 126L, 127L, 128L, 129L, 130L, 131L, 132L, 13 3L, 134L, 135L, 136L, 137L, 138L, 139L, 140L, 141L, 142L, 143L, 144L, 145L, 146L, 147L, 148L, 149L, 150L, 151L, 152L, 153L, 154L, 155L, 156L, 157L, 158L, 159L, 160L, 161L, 162L, 163L, 164L, 165L, 166L, 167L, 168L], 0, 4000) 2012-01-02 15:22:54,158 DEBUG [ omero.perf] (Dummy-5 ) start[1325514174147] time[11] tag[omero.tables.HdfStorage._getrows] 2012-01-02 15:22:54,185 DEBUG [ omero.perf] (Dummy-5 ) start[1325514174158] time[26] tag[omero.tables.HdfStorage._rowstocols] 2012-01-02 15:22:54,185 DEBUG [ omero.perf] (Dummy-5 ) start[1325514174185] time[0] tag[omero.tables.HdfStorage._as_data] 2012-01-02 15:22:54,187 DEBUG [ omero.perf] (Dummy-5 ) start[1325514174141] time[45] tag[omero.tables.TableI.read]
At the moment, I'm assuming this is a serialization issue.
See: http://www.openmicroscopy.org/community/viewtopic.php?f=6&t=990
Attachments (1)
Change History (16)
comment:1 Changed 12 years ago by jmoore
comment:2 Changed 12 years ago by jmoore <josh@…>
(In [eeeab8af075480131dd653333ba9d53e9d6e83cb/ome.git]) Minor refactoring for speed (See #7711)
comment:3 Changed 12 years ago by jmoore <josh@…>
(In [0ad8841f92d44c1dfc4b604c4e1e5bc1dc44cf1d/ome.git]) Fix 'lock' referenced before assignment (See #7711)
comment:4 Changed 12 years ago by jmoore
- Remaining Time set to 1.5
comment:5 Changed 12 years ago by jburel
- Sprint changed from 2012-01-17 (6) to 2012-01-31 (7)
Moved from sprint 2012-01-17 (6)
comment:6 Changed 12 years ago by jmoore
- Sprint changed from 2012-01-31 (7) to 2012-02-14 (8)
Moved from sprint 2012-01-31 (7)
comment:7 Changed 12 years ago by jburel
- Sprint changed from 2012-02-14 (8) to 2012-02-28 (9)
Moved from sprint 2012-02-14 (8)
comment:8 Changed 12 years ago by jburel
- Sprint changed from 2012-02-28 (9) to 2012-03-13 (10)
Moved from sprint 2012-02-28 (9)
comment:9 Changed 12 years ago by jburel
- Sprint changed from 2012-03-13 (10) to 2012-03-27 (11)
Moved from sprint 2012-03-13 (10)
comment:10 Changed 12 years ago by jburel
- Sprint changed from 2012-03-27 (11) to 2012-04-10 (12)
Moved from sprint 2012-03-27 (11)
comment:12 Changed 12 years ago by jmoore
- Milestone changed from OMERO-Beta4.4 to OMERO-Beta4.4.x
Moving to 4.4.x for further scheduling.
comment:13 Changed 11 years ago by jmoore
- Cc spli added; szwells removed
comment:14 Changed 10 years ago by jamoore
- Cc ux@… added
- Keywords TAB added
- Version set to 4.4.9
comment:15 Changed 8 years ago by jamoore
- Milestone changed from 5.x to Unscheduled
Changing the test script (to-be-attached) to load all of the data via batching shows that intermediate values (~100 columns, ~2000 rows) seems to be the best way to load. In two separate runs: