Bug: Dropbox / CLI import on Windows

[sylittlewood]

While testing Dropbox on Windows 7 I encountered problems importing some image types.

Generally they were large images (pyramid)

Some image types failed to produce thumbnails and some failed to import to allow viewing the image and for some the images were corrupt (i think) it's hard to tell as test_images_good does not have anything to compare against.

The logs are attached in zip files.

[sylittlewood]

log file from windows box

[jmoore]

Primary issue looks to be:

2012-09-21 09:46:57,415 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-23 ) at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:574)
2012-09-21 09:46:57,415 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-23 ) ... 3 more
2012-09-21 09:46:57,415 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-23 ) 
2012-09-21 09:46:57,415 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-23 ) java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
2012-09-21 09:46:57,415 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-23 ) at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:752)

But there's also an issue with the err files:

2012-09-21 09:47:45,292 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-21 ) Import of c:\OMERO\DropBox\dave\jpeg2000\052_L050071_56_x2hv.jp2 completed (session=bd29e7d5-bb51-4d45-bac8-2bafe9b60ae9)
2012-09-21 09:47:45,344 INFO  [                    omero.util.Resources] (Thread-38 ) Halted
2012-09-21 09:47:45,512 WARNI [                                  stderr] (Thread-21 ) Exception in thread Thread-21:
Traceback (most recent call last):
  File "C:\Python25\lib\threading.py", line 488, in __bootstrap_inner
    self.run()
  File "C:\omero_dist\lib\python\omero\util\concurrency.py", line 98, in run
    self.function(*self.args, **self.kwargs)
  File "c:\omero_dist\lib\python\fsDropBoxMonitorClient.py", line 492, in importFileWrapper
    self.importFile(fileId, exName)
  File "C:\omero_dist\lib\python\omero\util\decorators.py", line 28, in handler
    return func(*args, **kwargs)
  File "c:\omero_dist\lib\python\fsDropBoxMonitorClient.py", line 645, in importFile
    remove_path(t)
  File "C:\omero_dist\lib\python\omero\util\temp_files.py", line 340, in remove_path
    return manager.remove_path(file)
  File "C:\omero_dist\lib\python\omero\util\temp_files.py", line 283, in remove_path
    p.remove()
  File "C:\omero_dist\lib\python\path.py", line 939, in remove
    os.remove(self)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: u'C:\\Windows\\system32\\config\\systemprofile\\AppData\\Roaming\\omero\\tmp\\omero_SCOTT-WINDOWS-7$\\3892\\dropboxq_71wmerr'

I'll keep looking, but if there are no other issues, then this is likely not 4.4.4 related.

[jmoore]

Another issue is:

2012-09-19 16:51:59,203 WARNI [                                  stderr] (Thread-17 ) Joined session 39b1f483-84b0-401f-9ea3-e7ebc8e488f2 (dave@localhost:4064). Idle
 timeout: 10.0 min. Current group: system2012-09-19 16:52:02,970 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) Import of c:\OMERO\DropBox\dave\tiff-big-images\2kx2k.tif failed=2 (session=39b
1f483-84b0-401f-9ea3-e7ebc8e488f2)
2012-09-19 16:52:02,970 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) ***** start of output from importer-cli to stderr *****
2012-09-19 16:52:02,970 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) 2012-09-19 16:52:00,079 0          [      main] INFO          ome.formats.impor
ter.ImportConfig  - OMERO Version: 4.4.3-ice33-b5
...
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) 2012-09-19 16:52:02,861 2782       [      main] ERROR     ome.formats.importer.cli.ErrorHandler  - INTERNAL_EXCEPTION: c:\OMERO\DropBox\dave\tiff-big-images\2kx2k.tif
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) omero.ApiUsageException
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) serverStackTrace = "ome.conditions.ApiUsageException: Write-method not permitted.
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) at ome.io.nio.RomioPixelBuffer.throwIfReadOnly(RomioPixelBuffer.java:109)
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) at ome.io.nio.RomioPixelBuffer.setRegion(RomioPixelBuffer.java:623)
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) at ome.io.nio.RomioPixelBuffer.setTile(RomioPixelBuffer.java:901)
2012-09-19 16:52:02,986 ERROR [         fsclient.fsDropBoxMonitorClient] (Thread-17 ) at ome.services.RawPixelsBean.setTile(RawPixelsBean.java:791)

This could be caused by dropping your database but not dropping your /OMERO/Files directory, Elwood. Otherwise, this might point to an actual bug.

[cblackburn]

The err files issue is an age-old problem with Windows file locking -should we catch that temporary file delete just to keep the logs tidy?

It certainly looks like it is worth retesting with heap space increased and /OMERO completely cleaned.

[cblackburn]

There's a small fix in PR810, see ​https://github.com/openmicroscopy/openmicroscopy/pull/810#issuecomment-14561695

When that is tested the other issues above need to be confirmed as still relevant or not.

[Colin Blackburn <colin@…>]

(In [3751a38f422b55649e539f148d551fe54750eb25/ome.git] on branch develop) Log failed delete of tmp files, see #9648

[Colin Blackburn <colin@…>]

(In [52c9f665e4f49f4f3f6f2185998ce3bddcae69a6/ome.git] on branch develop) Revert "Log failed delete of tmp files, see #9648"

This reverts commit 0120ad8fcc9c0676bda24dc7d10a1526f3cac208.

[Colin Blackburn <colin@…>]

(In [8449e9e82963fc2ca39ffe33a94b925ae3b5d43d/ome.git] on branch develop) Move logging of failed delete to util class, see #9648

[jburel]

The ticket should have been closed.