Task #10892 (closed)
Opened 6 years ago
Closed 6 years ago
Bug: chgrp/delete performance
| Reported by: | jamoore | Owned by: | jamoore |
|---|---|---|---|
| Priority: | blocker | Milestone: | 5.0.0-beta1 |
| Component: | Services | Version: | n.a. |
| Keywords: | fs | Cc: | mtbcarroll, fs@… |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | n.a. |
| Sprint: | FS demo 4.4 |
Description
There have been numerous reports of slow graph operations. Moving 56 lei for example taking minutes. Minimally some investigation should be done into where this is happening.
A previous report about passing many Delete("/OriginalFile") instances in a single DoAll? seemed to be waiting completely at the JDBC level, i.e. the size of the transaction with the savepoints was simply too large.
Attachments (1)
Change History (9)
comment:1 Changed 6 years ago by jamoore
comment:2 Changed 6 years ago by jburel
- Sprint changed from FS demo 4.2 to FS Demo 4.3
Moved from sprint FS demo 4.2
comment:3 Changed 6 years ago by jamoore
I'm happy to look at this for demo 4.3, but I'm not very confident that a solution can be found in that time scale. The most likely solution that occurs to me is to drop the use of PG savepoints/rollbacks completely and rely on our own pre-/post-check logic. But replacing a DB feature like that requires (like our permission checks) requires extensive testing. Happy to be proved wrong though.
comment:4 Changed 6 years ago by wmoore
I just tried moving 3 images from Leica/joel/three_channel_seqen.lei in web, which always fails (locally) with the web hanging and a server restart needed. The log file from the chgrp submission to "hang" is attached http://trac.openmicroscopy.org.uk/ome/attachment/ticket/10892/chgrp-lei.log.zip
comment:5 Changed 6 years ago by jamoore
Will, by "locally" you mean on your own server? If so, what's the the memory setting? Do you have any OutOfMemory? lines in the log?
comment:6 Changed 6 years ago by jburel
- Sprint changed from FS Demo 4.3 to FS demo 4.4
Moved from sprint FS Demo 4.3
comment:7 Changed 6 years ago by jamoore
- Owner set to jamoore
comment:8 Changed 6 years ago by jamoore
- Resolution set to fixed
- Status changed from new to closed
The BTM fix introduced in https://github.com/openmicroscopy/openmicroscopy/pull/1243 for #11033 looks to have largely solved the performance issues. If problems arise again, please re-open.
As mentioned under PR 1142, chgrping the lei can succeed, but is running into O(n^2) issues since for each /Image, all the /FilesetEntry links are also attempted, which is unnecessary.