Task #13225 (closed)
RFE: paging for cleansing large repositories
Reported by: | mtbcarroll | Owned by: | mtbcarroll |
---|---|---|---|
Priority: | major | Milestone: | Unscheduled |
Component: | Client | Version: | OMERO-5.2.0 |
Keywords: | fs | Cc: | server@…, olivier.miquel@… |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | n.a. |
Sprint: | n.a. |
Description
In [ome-users] excessive time to upgrade database / cleanse it is reported that bin/omero admin cleanse dies in PublicRepositoryI.listFiles (RepositoryDaoImpl.getOriginalFiles) on a directory described at http://lists.openmicroscopy.org.uk/pipermail/ome-users/2016-May/005992.html as,
It's a 25G directory containing 54 436 files. root at kansas:/home/omero/apps/OMERO/OMERO.data/ManagedRepository# du -hs bellisj_454/2015-10/01 25G bellisj_454/2015-10/01 root at kansas:/home/omero/apps/OMERO/OMERO.data/ManagedRepository# find ./bellisj_454/2015-10/01 -type f | wc -l 54436
We need to reproduce this problem in our own testing then introduce paging as we should be able to cope with large repositories.
Change History (6)
comment:1 Changed 8 years ago by mtbcarroll
- Cc olivier.miquel@… added
comment:2 Changed 7 years ago by jamoore
comment:3 Changed 7 years ago by mtbcarroll
- Owner changed from jamoore to mtbcarroll
I can batch that query for 5.4.0.
comment:4 Changed 7 years ago by mtbcarroll
comment:5 Changed 7 years ago by mtbcarroll
- Resolution set to fixed
- Status changed from new to closed
comment:6 Changed 7 years ago by mtbcarroll
Fix should be in OMERO 5.4.0.
Talking to Andrea, assuming is_empty_dir is failing, then likely https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/blitz/src/ome/services/blitz/repo/RepositoryDaoImpl.java#L497 needs to use paging.