Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #7711 (new)

Opened 8 years ago

Last modified 3 years ago

Bug: slow loading on large number of table columns

Reported by: jamoore Owned by: jamoore
Priority: major Milestone: Unscheduled
Component: Performance Version: 4.4.9
Keywords: TAB Cc: bhcho@…, spli, ux@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 1.5d
Sprint: n.a.

Description

When reading (table.read(allColumns, 0, numOfRows)) a table of 169 columns and 4000 rows, reading all at the same time can lead to ~6 second delays:

        Columns
ROWS	1	16	84	152	169	
1	0.01	0.01	0.02	0.02	0.02	
400	0.02	0.07	0.34	0.62	0.70	
2000	0.04	0.31	1.66	3.04	3.38	
3600	0.08	0.54	3.02	5.43	6.15	
4000	0.08	0.61	3.62	5.93	6.67

Times logged to var/log/Tables-0.log look much better:

2012-01-02 15:22:54,141 INFO  [                            omero.remote] (Dummy-5   )  Meth: TableI.read
2012-01-02 15:22:54,141 INFO  [                     omero.tables.TableI] (Dummy-5   ) Table-175FA4C8-5459-4756-92D2-B671BD67C938.read([0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L
, 8L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L, 
41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 51L, 52L, 53L, 54L, 55L, 56L, 57L, 58L, 59L, 60L, 61L, 62L, 63L, 64L, 65L, 66L, 67L, 68L, 69L, 70L, 71L, 72L, 73L, 
74L, 75L, 76L, 77L, 78L, 79L, 80L, 81L, 82L, 83L, 84L, 85L, 86L, 87L, 88L, 89L, 90L, 91L, 92L, 93L, 94L, 95L, 96L, 97L, 98L, 99L, 100L, 101L, 102L, 103L, 104L, 105L,
 106L, 107L, 108L, 109L, 110L, 111L, 112L, 113L, 114L, 115L, 116L, 117L, 118L, 119L, 120L, 121L, 122L, 123L, 124L, 125L, 126L, 127L, 128L, 129L, 130L, 131L, 132L, 13
3L, 134L, 135L, 136L, 137L, 138L, 139L, 140L, 141L, 142L, 143L, 144L, 145L, 146L, 147L, 148L, 149L, 150L, 151L, 152L, 153L, 154L, 155L, 156L, 157L, 158L, 159L, 160L,
 161L, 162L, 163L, 164L, 165L, 166L, 167L, 168L], 0, 4000)
2012-01-02 15:22:54,158 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174147] time[11] tag[omero.tables.HdfStorage._getrows]
2012-01-02 15:22:54,185 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174158] time[26] tag[omero.tables.HdfStorage._rowstocols]
2012-01-02 15:22:54,185 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174185] time[0] tag[omero.tables.HdfStorage._as_data]
2012-01-02 15:22:54,187 DEBUG [                              omero.perf] (Dummy-5   ) start[1325514174141] time[45] tag[omero.tables.TableI.read]

At the moment, I'm assuming this is a serialization issue.

See: http://www.openmicroscopy.org/community/viewtopic.php?f=6&t=990

Attachments (1)

test.py (1.6 KB) - added by jmoore 8 years ago.
File used to test various read times.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 8 years ago by jmoore

Changing the test script (to-be-attached) to load all of the data via batching shows that intermediate values (~100 columns, ~2000 rows) seems to be the best way to load. In two separate runs:

#1 Batch size of:
          Columns
Rows 	  55	 111	 169	
1320	6.63	4.42	7.51	
2640	5.44	3.21	4.45	
4000	6.77	4.46	6.82	

#2 Batch size of:
          Columns
Rows 	  55	 111	 169	
1320	6.69	4.47	6.88	
2640	4.53	3.09	4.92	
4000	7.16	4.48	6.81

Changed 8 years ago by jmoore

File used to test various read times.

comment:2 Changed 8 years ago by jmoore <josh@…>

(In [eeeab8af075480131dd653333ba9d53e9d6e83cb/ome.git]) Minor refactoring for speed (See #7711)

comment:3 Changed 8 years ago by jmoore <josh@…>

(In [0ad8841f92d44c1dfc4b604c4e1e5bc1dc44cf1d/ome.git]) Fix 'lock' referenced before assignment (See #7711)

comment:4 Changed 8 years ago by jmoore

  • Remaining Time set to 1.5

comment:5 Changed 8 years ago by jburel

  • Sprint changed from 2012-01-17 (6) to 2012-01-31 (7)

Moved from sprint 2012-01-17 (6)

comment:6 Changed 7 years ago by jmoore

  • Sprint changed from 2012-01-31 (7) to 2012-02-14 (8)

Moved from sprint 2012-01-31 (7)

comment:7 Changed 7 years ago by jburel

  • Sprint changed from 2012-02-14 (8) to 2012-02-28 (9)

Moved from sprint 2012-02-14 (8)

comment:8 Changed 7 years ago by jburel

  • Sprint changed from 2012-02-28 (9) to 2012-03-13 (10)

Moved from sprint 2012-02-28 (9)

comment:9 Changed 7 years ago by jburel

  • Sprint changed from 2012-03-13 (10) to 2012-03-27 (11)

Moved from sprint 2012-03-13 (10)

comment:10 Changed 7 years ago by jburel

  • Sprint changed from 2012-03-27 (11) to 2012-04-10 (12)

Moved from sprint 2012-03-27 (11)

comment:11 Changed 7 years ago by jmoore

  • Sprint 2012-04-10 (12) deleted

Away for this sprint.

comment:12 Changed 7 years ago by jmoore

  • Milestone changed from OMERO-Beta4.4 to OMERO-Beta4.4.x

Moving to 4.4.x for further scheduling.

comment:13 Changed 6 years ago by jmoore

  • Cc spli added; szwells removed

comment:14 Changed 6 years ago by jamoore

  • Cc ux@… added
  • Keywords TAB added
  • Version set to 4.4.9

comment:15 Changed 3 years ago by jamoore

  • Milestone changed from 5.x to Unscheduled
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.102994 sec.)

We're Hiring!