Task #10285 (closed)
Opened 11 years ago
Closed 11 years ago
Bug: Importer performance does not scale to large quantities of images
| Reported by: | omero-qa | Owned by: | jburel |
|---|---|---|---|
| Priority: | major | Milestone: | Unscheduled |
| Component: | Import | Version: | 4.4.9 |
| Keywords: | n.a. | Cc: | jsluka@… |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | n.a. |
| Sprint: | n.a. |
Description
https://www.openmicroscopy.org/qa2/qa2/qa/feedback/4892/
Testing import performance of 30000 ~2.5KiB PNG files.
4.4/howe: ~1s/image; est import time: 8hours
4.5/gretzky: ~1.5s/image; est import time: 12hours
I'll have better timings once the imports complete.
On 4.4, the file handles remain open, so I currently have >3600 open image files for one import. There's already a ticket open for this. Does not appear to affect gretsky/FS.
There is obviously an excessive amount of overhead here on both systems. The data is only ~70MiB total, so the import should really be complete in under a minute. Is the roundtrip for thumbnail generation a factor here? Given that it's not strictly necessary for importing, could this be skipped entirely or at least not be done synchronously?
Change History (3)
comment:1 Changed 11 years ago by rleigh
- Component changed from from QA to Import
- Priority changed from minor to major
- Summary changed from Bug: Importer performance to Bug: Importer performance does not scale to large quantities of images
comment:2 Changed 11 years ago by sylittlewood
- Owner changed from sylittlewood to jburel
comment:3 Changed 11 years ago by jburel
- Resolution set to wontfix
- Status changed from new to closed
- Version set to 4.4.9
No longer required to wait for the import to complete to close client.
As soon as all the files are uploaded the "client import" part is done.
This will not be ported to 4.4, implemented on develop only