Task #7240 (closed)
Opened 12 years ago
Closed 10 years ago
Bug: Drive space reporting.
Reported by: | wmoore | Owned by: | jamoore |
---|---|---|---|
Priority: | major | Milestone: | 5.1.0 |
Component: | Services | Version: | 4.4.8 |
Keywords: | Delete,IO | Cc: | fs@… |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | 0.0d |
Sprint: | n.a. |
Description (last modified by wmoore)
UPDATE: Seems that this issue is really due to webadmin Drive space incorrectly reporting space used by failed Big Image imports. i.e. It doesn't take into account the fact that the import failed and the pyramid never got generated. So, when you delete the image, webadmin tells you that a large amount of data got removed (from pie-chart) but very little free space appears.
Solution: need to have a 'disk-space' column on Pixels table, that is updated on import, pyramid generation etc.
This may need to be a "User Story" to include various bits of this work. NB: disk space query could also include file attachments!
Below is the original reporting of this ticket:
Seems that Scott has deleted a lot of images on gretzky (Big Images)? that have not freed up space on disk.
Free disk space (using $ df) is approximately the same as that reported in webadmin, so this seems to be reporting correctly. Must also be including the deleted imgaes/pixels.
In Scott's 'JRS-read-only' group, under Orphaned images, there are a bunch of images with failing thumbnails. E.g.
id | permissions | methodology | physicalsizex | physicalsizey | physicalsizez | sha1 | sizec | sizet | sizex | sizey | sizez | timeincrement | version | waveincrement | wavestart | creation_id | external_id | group_id | owner_id | update_id | dimensionorder | image | pixelstype | relatedto | image_index | path | name | repo | params -------+-------------+-------------+---------------+---------------+---------------+------+-------+-------+-------+-------+-------+---------------+---------+---------------+-----------+-------------+-------------+----------+----------+-----------+----------------+-------+------------+-----------+-------------+---------------+-------+------+---------------- 28690 | -39 | | | | | Foo | 3 | 1 | 34814 | 13067 | 1 | | | | | 327964 | | 5 | 155 | 327964 | 1 | 28890 | 5 | | 0 | Files/Dir-014 | 14779 | | {{image_no,2}}
This references a file on disk:
omero@gretzky:~$ ls -al /OMERO/Files/Dir-014/14779 -rw-r--r-- 1 omero omero 421455123 2011-10-17 11:58 /OMERO/Files/Dir-014/14779
This file is not removed when the image is deleted, even though the row is removed from pixels table.
Change History (29)
comment:1 Changed 12 years ago by jmoore
- Cc cblackburn added
comment:2 Changed 12 years ago by cblackburn
Is the log available that covers the deletes?
comment:3 Changed 12 years ago by wmoore
The log is on gretzky. I was doing the above delete example just now, while writing the ticket.
You should also be able to reproduce the bug by following the same steps.
comment:4 Changed 12 years ago by cblackburn
There are no logged warnings of failed binary deletes. Is it posible to restart gretzky with debug on and then trying to reproduce?
comment:5 Changed 12 years ago by cblackburn
I've not been able to reproduce this locally with jpg big images. But the params field file above is {{image_no,2}} which should mean it's a multi-image format. Do you know what the original image was so that I can investigate further?
comment:6 Changed 12 years ago by wmoore
These were all svs files. I can't point you to them on squig right now (no VPN) but hopefully you can find some similar ones.
comment:7 Changed 12 years ago by saloynton
The selection of images used would have been under data_repo/test_images_good/svs on squig
comment:8 Changed 12 years ago by cblackburn
- Owner changed from jmoore to cblackburn
- Remaining Time set to 0.5
- Status changed from new to accepted
Thanks. I'll test with SVS files from squig.
comment:9 Changed 12 years ago by cblackburn
- Owner changed from cblackburn to jmoore
- Remaining Time 0.5 deleted
I can't reproduce this locally using SVS files. Not sure how to push this further other than trying to reproduce on gretzky with extra logging on the deletes.
comment:10 Changed 12 years ago by cblackburn
- Owner changed from jmoore to cblackburn
- Remaining Time set to 0.5
Starting a round of investigations using gretzky.
comment:11 Changed 12 years ago by cblackburn
- Cc jmoore added
I've not been able to reproduce this on gretzky using SVS files. The debug log for delete looks okay all files are being identified and then subsequently deleted. Not sure of the next step. It may be that we leave debugging on for delete and monitor the situation?
comment:12 Changed 12 years ago by jmoore
Will, Scott - Any ideas on how we could reproduce this? If not, Colin, I think you're right: let's just leave debugging turned on and wait.
comment:13 Changed 12 years ago by wmoore
Just tried to reproduce on mage with no success. Deletes reported previously didn't take into account that multiple images share a single /File/ so it shouldn't be removed until the last image is deleted. BUT it seems like the issue above may be due to the webadmin pie-chart incorrectly reporting the size of failed Big-Image imports. Currently, the total disk usage reported on gretzky is 212 GB, with 18 GB free although the total disk space is much smaller:
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 129G 104G 19G 86% / none 7.9G 220K 7.9G 1% /dev none 7.9G 0 7.9G 0% /dev/shm none 7.9G 80K 7.9G 1% /var/run none 7.9G 0 7.9G 0% /var/lock none 7.9G 0 7.9G 0% /lib/init/rw
NB: Nice feature for the Pie-Chart in webadmin would be a display of the total disk usage (Sum of all users).
comment:14 Changed 12 years ago by wmoore
- Cc atarkowska added; jmoore removed
- Description modified (diff)
- Owner changed from cblackburn to jmoore
- Summary changed from Bug: deleted images not freeing space to Bug: Drive space reporting.
comment:15 Changed 12 years ago by jmoore
- Priority changed from major to critical
- Type changed from Task to User Story
Turning into a story, then. We need to decide sooner rather than later if we want DB changes to do this, or if we can manage it with FS work.
comment:16 Changed 12 years ago by jmoore
- Owner changed from jmoore to cxallan
Passing off to Chris for appropriate scheduling.
comment:17 Changed 12 years ago by saloynton
I am taking this story out of the pfelts requirement so that user story can be closed.
comment:18 Changed 12 years ago by cxallan
- Cc atarkowska removed
- Milestone changed from OMERO-Beta4.4 to OMERO-Beta4.5
- Owner cxallan deleted
Pushing this to milestone:OMERO-Beta4.5 to be with a lot of other OMERO.fs work.
comment:19 Changed 12 years ago by jmoore
- Milestone changed from OMERO-4.5 to OMERO-4.4.x
comment:20 Changed 12 years ago by jmoore
- Type changed from User Story to Task
Turning back into a ticket to put under #909.
comment:21 Changed 11 years ago by atarkowska
- Keywords IO added
- Milestone changed from OMERO-4.4.9 to OMERO-4.4.x
- Owner set to jamoore
- Priority changed from critical to major
- Version set to 4.4.8
comment:22 Changed 10 years ago by jamoore
- Cc fs@… added; cxallan saloynton cblackburn removed
- Milestone changed from 5.x to 5.1.0
Assigning to Mark per https://github.com/openmicroscopy/openmicroscopy/pull/2270#issuecomment-40693489 This will require a re-thinking of the IO level such that "hidden" files like those for pyramids are somehow represented.
comment:23 Changed 10 years ago by jamoore
- Owner changed from jamoore to mtbcarroll
comment:24 Changed 10 years ago by mtbcarroll
I have no idea what is going on here: I'll need much more information if I'm to do anything on this ticket. Do we want this extra database column? (I am guessing that if I try to track down where pyramid generation happens I will find that pixels objects are created and persisted?) Is there still a problem with files not being deleted? Etc., etc. Basically: at this point, what is wanted, and what needs to happen to deliver it?
comment:25 Changed 10 years ago by jamoore
Probably easiest to just chat. This goes back to abstracting away more of the IO level, and possibly includes the subclasses of OriginalFile that "due the right thing". For example, nowhere in the DB is it recorded whether there is a pyramid file nor how large it is. That would be a good starting point.
comment:26 Changed 10 years ago by mtbcarroll
So, a submittable cmd object should be able to check through original files, pixels objects, pyramids, etc. server-side. ome.io.nio.PixelsService._getPixelBuffer has some guidance here.
comment:27 Changed 10 years ago by mtbcarroll
https://github.com/openmicroscopy/openmicroscopy/pull/2919 addresses the action item from the chat. Whether there is still other work to be done on this ticket by others, I do not know.
comment:28 Changed 10 years ago by mtbcarroll
- Owner changed from mtbcarroll to jamoore
Reassiging for further analysis / action items.
comment:29 Changed 10 years ago by Josh Moore <josh@…>
- Remaining Time set to 0
- Resolution set to fixed
- Status changed from accepted to closed
(In [3667342755fc8a42585ee6004ee55706772b5ea1/ome.git] on branch develop) Merge pull request #2919 from mtbc/trac-7240-disk-usage-count
fix #7240: provide disk usage count
Colin, thoughts?