Bug: Insight AWT thread hangs when large directories are selected for import

[mtbcarroll]

During the hang jstack reveals the culprit,

"AWT-EventQueue-0" prio=6 tid=1170d2000 nid=0x116793000 runnable [116790000]
   java.lang.Thread.State: RUNNABLE
	at java.io.UnixFileSystem.list(Native Method)
	at java.io.File.list(File.java:973)
	at java.io.File.listFiles(File.java:1051)
	at sun.awt.shell.ShellFolder.listFiles(ShellFolder.java:102)
	at sun.awt.shell.ShellFolder.listFiles(ShellFolder.java:98)
	at sun.awt.shell.DefaultShellFolder.listFiles(DefaultShellFolder.java:46)
	at org.apache.commons.io.FileUtils.sizeOfDirectory(FileUtils.java:1514)
	at org.apache.commons.io.FileUtils.sizeOfDirectory(FileUtils.java:1522)
	at org.apache.commons.io.FileUtils.sizeOfDirectory(FileUtils.java:1522)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.FileElement.getFileLength(FileElement.java:140)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.FileElement.getFileLengthAsString(FileElement.java:151)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.FileSelectionTable.addFiles(FileSelectionTable.java:751)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.ImportDialog.addFiles(ImportDialog.java:377)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.ImportDialog.showLocationDialog(ImportDialog.java:385)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.ImportDialog.propertyChange(ImportDialog.java:1465)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:339)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:276)
	at java.awt.Component.firePropertyChange(Component.java:8255)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.FileSelectionTable.actionPerformed(FileSelectionTable.java:842)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2028)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2351)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
	at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:272)
	at java.awt.Component.processMouseEvent(Component.java:6382)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3275)
	at java.awt.Component.processEvent(Component.java:6147)
	at java.awt.Container.processEvent(Container.java:2083)
	at java.awt.Component.dispatchEventImpl(Component.java:4744)
	at java.awt.Container.dispatchEventImpl(Container.java:2141)
	at java.awt.Component.dispatchEvent(Component.java:4572)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4619)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4280)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4210)
	at java.awt.Container.dispatchEventImpl(Container.java:2127)
	at java.awt.Window.dispatchEventImpl(Window.java:2489)
	at java.awt.Component.dispatchEvent(Component.java:4572)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:710)
	at java.awt.EventQueue.access$400(EventQueue.java:82)
	at java.awt.EventQueue$2.run(EventQueue.java:669)
	at java.awt.EventQueue$2.run(EventQueue.java:667)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
	at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
	at java.awt.EventQueue$3.run(EventQueue.java:683)
	at java.awt.EventQueue$3.run(EventQueue.java:681)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:680)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

[jburel]

General Java problem when dealing with large directory.
The file chooser will not be able to handle. I guess B-F scanning will run forever (not tested)

[jamoore]

FileUtils.sizeOfDirectory is very straight-forward, essentially:

        long size = 0;

        File[] files = directory.listFiles();
        for (int i = 0; i < files.length; i++) {
            File file = files[i];

            if (file.isDirectory()) {
                size += sizeOfDirectory(file);
            } else {
                size += file.length();
            }
        }

I would suggest we make the same loop (possibly in Bio-Formats) with cancellation in place (as with ImportCandidates? recently). Something like:

public BigInteger sizeOfDirectory(File directory) {
   return sizeOfDirectory(File f, long count) {
}

public BigInteger sizeOfDirectory(File directory, long count) {
    ...
    File [] files = directory.listFiles();
    for (int i = 0; i < files.length; i++) {
        count++;
        if (count % 100) {
            callback(files[i], count);
        }
    }
} 

public abstract void callback(File f, long count);

Note: there may be something we can do more intelligently with Guava functors and similar.

Once that's in place, the operation should likely take place in a separate thread with cancellation similar to import, etc.