Bug: Drive space reporting.

[wmoore]

UPDATE: Seems that this issue is really due to webadmin Drive space incorrectly reporting space used by failed Big Image imports. i.e. It doesn't take into account the fact that the import failed and the pyramid never got generated. So, when you delete the image, webadmin tells you that a large amount of data got removed (from pie-chart) but very little free space appears.

Solution: need to have a 'disk-space' column on Pixels table, that is updated on import, pyramid generation etc.
This may need to be a "User Story" to include various bits of this work. NB: disk space query could also include file attachments!

Below is the original reporting of this ticket:

Seems that Scott has deleted a lot of images on gretzky (Big Images)? that have not freed up space on disk.

Free disk space (using $ df) is approximately the same as that reported in webadmin, so this seems to be reporting correctly. Must also be including the deleted imgaes/pixels.

In Scott's 'JRS-read-only' group, under Orphaned images, there are a bunch of images with failing thumbnails. E.g.

  id   | permissions | methodology | physicalsizex | physicalsizey | physicalsizez | sha1 | sizec | sizet | sizex | sizey | sizez | timeincrement | version | waveincrement | wavestart | creation_id | external_id | group_id | owner_id | update_id | dimensionorder | image | pixelstype | relatedto | image_index |     path      | name  | repo |     params     
-------+-------------+-------------+---------------+---------------+---------------+------+-------+-------+-------+-------+-------+---------------+---------+---------------+-----------+-------------+-------------+----------+----------+-----------+----------------+-------+------------+-----------+-------------+---------------+-------+------+----------------
 28690 |         -39 |             |               |               |               | Foo  |     3 |     1 | 34814 | 13067 |     1 |               |         |               |           |      327964 |             |        5 |      155 |    327964 |              1 | 28890 |          5 |           |           0 | Files/Dir-014 | 14779 |      | {{image_no,2}}

This references a file on disk:

omero@gretzky:~$ ls -al /OMERO/Files/Dir-014/14779
-rw-r--r-- 1 omero omero 421455123 2011-10-17 11:58 /OMERO/Files/Dir-014/14779

This file is not removed when the image is deleted, even though the row is removed from pixels table.

[jmoore]

Colin, thoughts?

[cblackburn]

Is the log available that covers the deletes?

[wmoore]

The log is on gretzky. I was doing the above delete example just now, while writing the ticket.
You should also be able to reproduce the bug by following the same steps.

[cblackburn]

There are no logged warnings of failed binary deletes. Is it posible to restart gretzky with debug on and then trying to reproduce?

[cblackburn]

I've not been able to reproduce this locally with jpg big images. But the params field file above is {{image_no,2}} which should mean it's a multi-image format. Do you know what the original image was so that I can investigate further?

[wmoore]

These were all svs files. I can't point you to them on squig right now (no VPN) but hopefully you can find some similar ones.

[saloynton]

The selection of images used would have been under data_repo/test_images_good/svs on squig

[cblackburn]

Thanks. I'll test with SVS files from squig.

[cblackburn]

I can't reproduce this locally using SVS files. Not sure how to push this further other than trying to reproduce on gretzky with extra logging on the deletes.

[cblackburn]

Starting a round of investigations using gretzky.

[cblackburn]

I've not been able to reproduce this on gretzky using SVS files. The debug log for delete looks okay all files are being identified and then subsequently deleted. Not sure of the next step. It may be that we leave debugging on for delete and monitor the situation?

[jmoore]

Will, Scott - Any ideas on how we could reproduce this? If not, Colin, I think you're right: let's just leave debugging turned on and wait.

[wmoore]

Just tried to reproduce on mage with no success. Deletes reported previously didn't take into account that multiple images share a single /File/ so it shouldn't be removed until the last image is deleted. BUT it seems like the issue above may be due to the webadmin pie-chart incorrectly reporting the size of failed Big-Image imports. Currently, the total disk usage reported on gretzky is 212 GB, with 18 GB free although the total disk space is much smaller:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             129G  104G   19G  86% /
none                  7.9G  220K  7.9G   1% /dev
none                  7.9G     0  7.9G   0% /dev/shm
none                  7.9G   80K  7.9G   1% /var/run
none                  7.9G     0  7.9G   0% /var/lock
none                  7.9G     0  7.9G   0% /lib/init/rw

NB: Nice feature for the Pie-Chart in webadmin would be a display of the total disk usage (Sum of all users) See #7289.

[jmoore]

Turning into a story, then. We need to decide sooner rather than later if we want DB changes to do this, or if we can manage it with FS work.

[jmoore]

Passing off to Chris for appropriate scheduling.

[saloynton]

I am taking this story out of the pfelts requirement so that user story can be closed.

[cxallan]

Pushing this to milestone:OMERO-Beta4.5 to be with a lot of other OMERO.fs work.

[jmoore]

Turning back into a ticket to put under #909.

[jamoore]

Assigning to Mark per ​https://github.com/openmicroscopy/openmicroscopy/pull/2270#issuecomment-40693489 This will require a re-thinking of the IO level such that "hidden" files like those for pyramids are somehow represented.

[mtbcarroll]

I have no idea what is going on here: I'll need much more information if I'm to do anything on this ticket. Do we want this extra database column? (I am guessing that if I try to track down where pyramid generation happens I will find that pixels objects are created and persisted?) Is there still a problem with files not being deleted? Etc., etc. Basically: at this point, what is wanted, and what needs to happen to deliver it?

[jamoore]

Probably easiest to just chat. This goes back to abstracting away more of the IO level, and possibly includes the subclasses of OriginalFile that "due the right thing". For example, nowhere in the DB is it recorded whether there is a pyramid file nor how large it is. That would be a good starting point.

[mtbcarroll]

So, a submittable cmd object should be able to check through original files, pixels objects, pyramids, etc. server-side. ome.io.nio.PixelsService._getPixelBuffer has some guidance here.

[mtbcarroll]

​https://github.com/openmicroscopy/openmicroscopy/pull/2919 addresses the action item from the chat. Whether there is still other work to be done on this ticket by others, I do not know.

[mtbcarroll]

Reassiging for further analysis / action items.

[Josh Moore <josh@…>]

(In [3667342755fc8a42585ee6004ee55706772b5ea1/ome.git] on branch develop) Merge pull request #2919 from mtbc/trac-7240-disk-usage-count

fix #7240: provide disk usage count