Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #7711 (new)

Opened 12 years ago

Last modified 8 years ago

Bug: slow loading on large number of table columns

Reported by: jamoore Owned by: jamoore
Priority: major Milestone: Unscheduled
Component: Performance Version: 4.4.9
Keywords: TAB Cc: bhcho@…, spli, ux@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 1.5d
Sprint: n.a.

Description

When reading (table.read(allColumns, 0, numOfRows)) a table of 169 columns and 4000 rows, reading all at the same time can lead to ~6 second delays:

        Columns
ROWS	1	16	84	152	169	
1	0.01	0.01	0.02	0.02	0.02	
400	0.02	0.07	0.34	0.62	0.70	
2000	0.04	0.31	1.66	3.04	3.38	
3600	0.08	0.54	3.02	5.43	6.15	
4000	0.08	0.61	3.62	5.93	6.67

Times logged to var/log/Tables-0.log look much better:

2012-01-02 15:22:54,141 INFO  [                            omero.remote] (Dummy-5   )  Meth: TableI.read
2012-01-02 15:22:54,141 INFO  [                     omero.tables.TableI] (Dummy-5   ) Table-175FA4C8-5459-4756-92D2-B671BD67C938.read([0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L
, 8L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L, 
41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 51L, 52L, 53L, 54L, 55L, 56L, 57L, 58L, 59L, 60L, 61L, 62L, 63L, 64L, 65L, 66L, 67L, 68L, 69L, 70L, 71L, 72L, 73L, 
74L, 75L, 76L, 77L, 78L, 79L, 80L, 81L, 82L, 83L, 84L, 85L, 86L, 87L, 88L, 89L, 90L, 91L, 92L, 93L, 94L, 95L, 96L, 97L, 98L, 99L, 100L, 101L, 102L, 103L, 104L, 105L,
 106L, 107L, 108L, 109L, 110L, 111L, 112L, 113L, 114L, 115L, 116L, 117L, 118L, 119L, 120L, 121L, 122L, 123L, 124L, 125L, 126L, 127L, 128L, 129L, 130L, 131L, 132L, 13
3L, 134L, 135L, 136L, 137L, 138L, 139L, 140L, 141L, 142L, 143L, 144L, 145L, 146L, 147L, 148L, 149L, 150L, 151L, 152L, 153L, 154L, 155L, 156L, 157L, 158L, 159L, 160L,
 161L, 162L, 163L, 164L, 165L, 166L, 167L, 168L], 0, 4000)
2012-01-02 15:22:54,158 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174147] time[11] tag[omero.tables.HdfStorage._getrows]
2012-01-02 15:22:54,185 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174158] time[26] tag[omero.tables.HdfStorage._rowstocols]
2012-01-02 15:22:54,185 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174185] time[0] tag[omero.tables.HdfStorage._as_data]
2012-01-02 15:22:54,187 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174141] time[45] tag[omero.tables.TableI.read]

At the moment, I'm assuming this is a serialization issue.

See: http://www.openmicroscopy.org/community/viewtopic.php?f=6&t=990

Attachments (1)

test.py (1.6 KB) - added by jmoore 12 years ago.
File used to test various read times.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 12 years ago by jmoore

Changing the test script (to-be-attached) to load all of the data via batching shows that intermediate values (~100 columns, ~2000 rows) seems to be the best way to load. In two separate runs:

#1 Batch size of:
          Columns
Rows 	  55	 111	 169	
1320	6.63	4.42	7.51	
2640	5.44	3.21	4.45	
4000	6.77	4.46	6.82	

#2 Batch size of:
          Columns
Rows 	  55	 111	 169	
1320	6.69	4.47	6.88	
2640	4.53	3.09	4.92	
4000	7.16	4.48	6.81

Changed 12 years ago by jmoore

File used to test various read times.

comment:2 Changed 12 years ago by jmoore <josh@…>

(In [eeeab8af075480131dd653333ba9d53e9d6e83cb/ome.git]) Minor refactoring for speed (See #7711)

comment:3 Changed 12 years ago by jmoore <josh@…>

(In [0ad8841f92d44c1dfc4b604c4e1e5bc1dc44cf1d/ome.git]) Fix 'lock' referenced before assignment (See #7711)

comment:4 Changed 12 years ago by jmoore

  • Remaining Time set to 1.5

comment:5 Changed 12 years ago by jburel

  • Sprint changed from 2012-01-17 (6) to 2012-01-31 (7)

Moved from sprint 2012-01-17 (6)

comment:6 Changed 12 years ago by jmoore

  • Sprint changed from 2012-01-31 (7) to 2012-02-14 (8)

Moved from sprint 2012-01-31 (7)

comment:7 Changed 12 years ago by jburel

  • Sprint changed from 2012-02-14 (8) to 2012-02-28 (9)

Moved from sprint 2012-02-14 (8)

comment:8 Changed 12 years ago by jburel

  • Sprint changed from 2012-02-28 (9) to 2012-03-13 (10)

Moved from sprint 2012-02-28 (9)

comment:9 Changed 12 years ago by jburel

  • Sprint changed from 2012-03-13 (10) to 2012-03-27 (11)

Moved from sprint 2012-03-13 (10)

comment:10 Changed 12 years ago by jburel

  • Sprint changed from 2012-03-27 (11) to 2012-04-10 (12)

Moved from sprint 2012-03-27 (11)

comment:11 Changed 12 years ago by jmoore

  • Sprint 2012-04-10 (12) deleted

Away for this sprint.

comment:12 Changed 12 years ago by jmoore

  • Milestone changed from OMERO-Beta4.4 to OMERO-Beta4.4.x

Moving to 4.4.x for further scheduling.

comment:13 Changed 11 years ago by jmoore

  • Cc spli added; szwells removed

comment:14 Changed 10 years ago by jamoore

  • Cc ux@… added
  • Keywords TAB added
  • Version set to 4.4.9

comment:15 Changed 8 years ago by jamoore

  • Milestone changed from 5.x to Unscheduled
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.94447 sec.)

We're Hiring!