Investigate GUI automated testing

[mtbcarroll]

It would be good to promptly detect when a PR breaks something. Petr's test scenarios are very helpful but they also consume much of his valuable time in execution; it's especially difficult to notice when we broke something we didn't think of, because applications like Insight are complex and featureful so it would take a large team to frequently test everything.

Perhaps a review of some of the offerings on pages like ​http://java-source.net/open-source/testing-tools and ​https://github.com/demetriusnunes/swinger for Insight and similar stuff for web would be well worth it if we can find a tenable way to automate the testing of Petr's scenarios.

[jburel]

I have tested various tools e.g. Maveryx Abbot Java GUI , marathon

[jburel]

​https://github.com/jalian-systems/Marathon for marathon on github

[mtbcarroll]

I wonder if this ticket should be split up into Java/Swing and Web/Django branches.

My immediate plan for Insight is to take a look at Jacareto, Marathon, Maveryx, Robot Framework (Swing Library), and to work with Petr in getting some scenario running with the least appalling of them.

[jamoore]

mtbc: Almost certainly. Will and Carlos will likely need/want to setup a parallel story.

[jburel]

Turning it into a story. see #4452

[jburel]

One good example (we had issue with that):

• Select an item in the data manager.
• Check that the options of the right click menu are enabled or not depending on the selected item(s).

[agilo]

Updated status, related task in progress

[jburel]

We have decided to move forward with the Robot Framework on the Java front.
I had a closer look at the selenium library for Robot Framework (pointed by Mark). This seems promising, easy for example:

• to run the selenium tests using the text file used by the framework
• to configure browser to test against (tested with firefox and safari for example)

The advantage there will be that we could use, when applicable, the same files to test insight or web. This could also reduce the time spent writing the tests and also help validating the UI unification, only one thing to learn etc.

Next step is to investigate in the context of the web client and see how we can exclude some sections of a test when not applicable to a client
I am also hopeful that we could re-use some of the files for testing the CLI client too.

[jburel]

To test the CLI client, we can use the OperatingSystem Library for Robot Framework. Several Run like methods can be used.
Follow to example test cases

*** Test Cases ***


Valid Login
    Login
    Logout

Import
    Login
    Run  ${LOCATION}/${COMMAND} import a.jpg 2>test.txt
    Logout

where Login and Logout are 2 created keyword using the run method.
In the example the output or the command is written in a local file text.txt, could be any file e.g. a test.log we wish to check stored.
The output of the Import test case is for example.

Using session 473dd378-0756-4f2f-91c9-179c6dea30f5 (root@localhost:4064). Idle timeout: 10.0 min. Current group: system
2013-08-12 17:06:21,781 0          [      main] INFO          ome.formats.importer.ImportConfig  - OMERO Version: 4.4.8-DEV-ice34
2013-08-12 17:06:21,785 4          [      main] INFO          ome.formats.importer.ImportConfig  - Bioformats version: 4.4.9-DEV revision: bcb65a7 date: 12 August 2013
2013-08-12 17:06:21,799 18         [      main] INFO   formats.importer.cli.CommandLineImporter  - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2013-08-12 17:06:22,326 545        [      main] INFO      ome.formats.importer.ImportCandidates  - Depth: 4 Metadata Level: MINIMUM
2013-08-12 17:06:22,835 1054       [      main] INFO      ome.formats.importer.ImportCandidates  - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 506ms. (509ms total) [0 unknowns]
2013-08-12 17:06:22,891 1110       [      main] ERROR                   ome.system.UpgradeCheck  - Error reading from url: http://upgrade.openmicroscopy.org.uk?version=4.4.8-DEV-ice34;os.name=Mac+OS+X;os.arch=x86_64;os.version=10.8.4;java.runtime.version=1.6.0_51-b11-457-11M4509;java.vm.vendor=Apple+Inc. "Server returned HTTP response code: 400 for URL: http://upgrade.openmicroscopy.org.uk?version=4.4.8-DEV-ice34;os.name=Mac+OS+X;os.arch=x86_64;os.version=10.8.4;java.runtime.version=1.6.0_51-b11-457-11M4509;java.vm.vendor=Apple+Inc."
2013-08-12 17:06:23,118 1337       [      main] INFO       ome.formats.OMEROMetadataStoreClient  - Attempting initial SSL connection to localhost:4064
2013-08-12 17:06:23,951 2170       [      main] INFO       ome.formats.OMEROMetadataStoreClient  - Insecure connection requested, falling back
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - Server: 4.4.8
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - Client: 4.4.8-DEV-ice34
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - Java Version: 1.6.0_51
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - OS Name: Mac OS X
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - OS Arch: x86_64
2013-08-12 17:06:34,936 13155      [      main] INFO       ome.formats.OMEROMetadataStoreClient  - OS Version: 10.8.4
2013-08-12 17:06:36,172 14391      [      main] INFO         ome.formats.importer.ImportLibrary  - File format: JPEG
2013-08-12 17:06:36,173 14392      [      main] INFO         ome.formats.importer.ImportLibrary  - Base reader: loci.formats.in.JPEGReader
2013-08-12 17:06:36,173 14392      [      main] INFO         ome.formats.importer.ImportLibrary  - Metadata only import? false
2013-08-12 17:06:36,173 14392      [      main] INFO         ome.formats.importer.ImportLibrary  - Archiving enabled? false
2013-08-12 17:06:36,173 14392      [      main] INFO         ome.formats.importer.ImportLibrary  - Container metadata only import? false
2013-08-12 17:06:36,173 14392      [      main] INFO         ome.formats.importer.ImportLibrary  - Reader is not of HCS domain, use metafile: true
2013-08-12 17:06:38,902 17121      [      main] INFO   ormats.importer.cli.LoggingImportMonitor  - IMPORT_DONE
Imported pixels:
2013-08-12 17:06:38,902 17121      [      main] INFO      ome.formats.importer.cli.ErrorHandler  - Number of errors: 0

[jburel]

Conclusion: we should be able to use the RobotFramework to test insight, web and CLI.
It seems also possible to store the result of the run into MySQL. I will investigate that further

[mtbcarroll]

The work on #4452 and #11210 is being done so as to try to write similar Robot Framework scripts for both Web and Insight. As well as easing the script maintenance burden, this should usefully highlight remaining differences between the UIs, which should then be noted in the UX/RFE documentation.