Automated GUI testing of Insight

[mtbcarroll]

Investigate tools like

• FEST
• Jacareto
• Marathon
• Maveryx
• Robot Framework (Swing Library)
• WindowTester

and work with Petr to get workflow / test scenarios running in such software for automated testing of Insight.

[mtbcarroll]

My current plan is to give each of the above four a few hours' reading and figuring out and playing to see if it seems viable, and, of those that do, a few more hours to set up something simple and share it with Petr, and encourage him to be thinking in terms of the automatable test categories provided. Then, we can decide, with which of these packages, if any, to plunge in further with our first real test scenarios.

[mtbcarroll]

keep Jason in the loop regarding plans, progress

[mtbcarroll]

After today, until the 22nd instant I have only three working days, so initial progress will be slow. Nevertheless, this month we should be able to select a framework and get simple examples working in it. Then, more can be added next month, and we can write developer documentation about how to do that.

[mtbcarroll]

Much as I hate to add another to the list, bpindelski points out WindowTester and that does also look good. I guess it's worth making this initial framework choice well, but I'll try not to go far down many rabbit holes.

[mtbcarroll]

Jacareto looks less actively maintained than I'd first thought, I'll ignore it for now unless others are found wanting.

[mtbcarroll]

Hmmm, Marathon have a separate commercial version if one wants things like object map maintenance.

[mtbcarroll]

Maveryx is on sourceforge. It's frustratingly difficult to get to free, decent technical documentation for, though, and the user manual's from 2011.

[mtbcarroll]

Interestingly, Robot Framework at ​https://code.google.com/p/robotframework/ offers both Selenium2 scripting at ​https://github.com/rtomac/robotframework-selenium2library and Swing scripting at ​https://github.com/robotframework/SwingLibrary

As Insight and Web continue to converge, I don't know if that affords any opportunity to bring them closer in automated testing too. I'll ask Will.

[mtbcarroll]

WindowTester looks not to have had a release in over a year and the FAQ is somewhat out of date.

[mtbcarroll]

The Robot Framework SwingLibrary doesn't come with a nice GUI recorder of app interactions, but that could be tolerable if some other tool makes it easy to find what is needed to specify the GUI interactions in the test scripts. A quick glance at a couple of such tools doesn't fill me with joy, but I've not yet looked hard; hopefully there's an Eclipse plugin that makes Swing component properties easily browsed. A separate but related issue is that it may behoove us to pervade Insight with more Component.setName().

(I'm not assuming that an "easy" recorder tool would insulate us from the Swing component details anyway if the hope is to write scripts that don't break every time the UI changes a little.)

[mtbcarroll]

I'll persist with Robot Framework for now, falling back to Marathon if need be.

If I install Robot Framework with jython setup.py install then I seem to be able to execute scripts from components/insight/OUT/app/ if I use jybot with swinglibrary-1.6.0.jar and compiled and libs/* in my Java classpath.

I can also get Swing Explorer working okay as an Eclipse plug-in, to explore the component hierarchy.

[mtbcarroll]

SwingLibrary is looking very workable, though there is some initial investment in adding Component.setName() calls around the code; this probably makes it easier to write robust test scripts for any automated UI testing tool. I have run into one issue, probably with Jemmy that it uses under the hood; it is noted at ​https://github.com/robotframework/SwingLibrary/issues/43

[mtbcarroll]

Progress in adding Component.setName() is at ​https://github.com/mtbc/openmicroscopy/compare/openmicroscopy:dev_4_4...Component.setName

Chapter 2 of ​http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.8.1 and the SwingLibrary keywords at ​http://robotframework.org/SwingLibrary/SwingLibrary-1.6.0.html cover how I have been writing test scripts. For instance,

*** Settings ***
Library  SwingLibrary

*** Test Cases ***
Log In
    Launch Application  org.openmicroscopy.shoola.Main
    Select Window  login window
    List Components In Context  formatted
    Button Should Be Enabled  config server button
#    Click On Component  config server button
#    Select Dialog  server dialog
#    List Components In Context  formatted
#    Button Should Be Disabled  apply button
#    ${localhost}=  Find Table Row  server table  localhost  host
#    Click On Table Cell  server table  ${localhost}  0
#    Button Should Be Enabled  apply button
#    Click On Component  apply button
#    Select Window  login window
    Clear Text Field  user
    Type Into Text Field  user  root
    Type Into Text Field  pass  omero
    List Components In Context  formatted
    Click On Component  login button
    Select Window  top window
    List Components In Context  formatted

to try to log in and see the names of the main window Swing components. (The commenting-out in the script is due to the Jemmy issue referenced above.)

Component names are also revealed through ​http://www.swingexplorer.com/ which works okay as a plug-in for my Eclipse Indigo.

In contrast to the above, note that test scripts may be written in reStructuredText format which may be a great convenience if we wish to include them in our Sphinx documentation.

So, with regard to Robot Framework, pwalczysko, I'd suggest you

• assume I can give names to whichever parts of the Insight UI we wish to test or prod, even the little yellow annotation marking in thumbnail title bars, but preferably not a hundred different ones at first
• review the list of keywords offered by SwingLibrary (plenty of "checking" ones as well as "doing" ones)
• think about what simple test scripts we could start out with (note #11214).

Hopefully the above Jemmy problem with Click On Component will be addressed in the meantime or otherwise revealed to be easily solved.

[mtbcarroll]

Marathon has various uglinesses, including doing cut-and-paste in text fields with control instead of command, that make me guess that it might be nicer on Windows. The testing projects appear to use files that are variously in CSV, XML, YAML. The manual's somewhat Ruby- and not-Mac-centric, but I think it's still workable enough for Python test scripts on a Mac.

[mtbcarroll]

With SwingLibrary I appear able to do things like,

    ${localhost}=  Find Table Row  server table  localhost  host
    Click On Table Cell  server table  ${localhost}  0

to choose a server from the list at login. From the Marathon documentation it is extremely unclear how to emulate this. It covers the simple stuff somewhat well, but, unless I'm missing something, the technical detail stops a step short of what we would actually need. For instance, select can "set the state corresponding to the given text" but nowhere in the manual have I yet found specification of if I should expect to be able to do things like,

select('server table', 'rows:[localhost],columns:[host]')

[mtbcarroll]

With regard to Marathon, pwalczysko, I'd suggest that you briefly peruse the manual from the bottom of ​http://marathontesting.com/downloads/ to get an idea of what it is. Also, should you feel moved to, don't feel inhibited from taking a look as best you can at other Java UI testing frameworks, especially ones that seem to be actively maintained and with a growing user base. (For instance, how old's the manual? How busy are their mailing lists?)

To help you to compare with my above script quotes for SwingLibrary, "Script Methods" at the beginning of Appendix A in that Marathon manual is largely equivalent to the list of keywords for SwingLibrary, but, unfortunately, not in such detail.

Marathon does provide a "recorder" so that you can do things in Insight and then it writes you a script, but it doesn't buy us a whole lot: we would still need to work on that script manually in non-trivial ways because it doesn't know what we were really aiming for when we did things -- i.e. how what we did was affected by what we saw. For instance, in the server list, was I aiming for "localhost", or for the seventh row down, or what? Did I open that server dialog only because "localhost" wasn't already selected? That kind of test fixture thing ends up needing to be hand-coded.

[mtbcarroll]

Marathon's IDE seems a bit strange and clunky too. So far I seem to have caused it to generate scripts that it can't then execute even with its own bundled Jython. (Naturally, project settings are on the unhelpfully named "Marathon" menu.)

Barring reasons to change course, I think that my initial plan upon returning from leave would be to plug a debugger into the latest Jemmy 2.x to see if I can easily find a fix for the current issue with getting Robot Framework to run my SwingLibrary script, then to try building a conditional branching into it.

[pwalczysko]

@mtbcarrol:

• Read the document as indicated above
• cloned the git repository and managed to start marathon from cmd line (does not work from the downloaded app - will open and immediately close)
• created a project and tried to aim it at my local Insight, namely at omero.insight.jar in

  openmicroscopy/target/OMERO.insight-4.4.8-DEV-ice33-mac/OMERO.insight.app/Contents/Resources/Java/
  

  • clicked the "Record" button in Marathon
  • it ends up with an error

    java.io.IOException: Cannot run program "/Users/petr/Work/openmicroscopy/target/OMERO.insight-4.4.8-DEV-ice33-mac/OMERO.insight.app/Contents/Resources/Java/omero.insight.jar" (in directory "/Users/petr/Work/openmicroscopy/target/OMERO.insight-4.4.8-DEV-ice33-mac/OMERO.insight.app/Contents/Resources/Java" error=13, Permission denied
       at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
       at net.sourceforge.marathon.runtime.JavaRuntimeFactory.launchVM(JavaRuntimeFactory.java:94)
       at net.sourceforge.marathon.runtime.JavaRuntimeFactory.createRuntime(JavaRuntimeFactory.java:64)
       at net.sourceforge.marathon.display.Display.createRuntime(Display.java:364)
       at net.sourceforge.marathon.display.Display.record(Display.java:224)
       at net.sourceforge.marathon.display.DisplayWindow.onRecord(DisplayWindow.java:2936)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:601)
       at net.sourceforge.marathon.display.ActionInjector$1.actionPerformed(ActionInjector.java:92)
       at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
       at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
       at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
       at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
       at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
       at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:290)
       at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
       at java.awt.Component.processMouseEvent(Component.java:6505)
       at javax.swing.JComponent.processMouseEvent(JComponent.java:3321)
       at java.awt.Component.processEvent(Component.java:6270)
       at java.awt.Container.processEvent(Container.java:2229)
       at java.awt.Component.dispatchEventImpl(Component.java:4861)
       at java.awt.Container.dispatchEventImpl(Container.java:2287)
       at java.awt.Component.dispatchEvent(Component.java:4687)
       at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
       at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
       at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
       at java.awt.Container.dispatchEventImpl(Container.java:2273)
       at java.awt.Window.dispatchEventImpl(Window.java:2719)
       at java.awt.Component.dispatchEvent(Component.java:4687)
       at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:723)
       at java.awt.EventQueue.access$200(EventQueue.java:103)
       at java.awt.EventQueue$3.run(EventQueue.java:682)
       at java.awt.EventQueue$3.run(EventQueue.java:680)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
       at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
       at java.awt.EventQueue$4.run(EventQueue.java:696)
       at java.awt.EventQueue$4.run(EventQueue.java:694)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
       at java.awt.EventQueue.dispatchEvent(EventQueue.java:693)
       at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
       at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
       at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
       at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
       at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
       at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
    Caused by: java.io.IOException: error=13, Permission denied
       at java.lang.UNIXProcess.forkAndExec(Native Method)
       at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
       at java.lang.ProcessImpl.start(ProcessImpl.java:130)
       at java.lang.ProcessBuilder.start(ProcessBuilder.java:1021)
       ... 48 more
    

Did you sort out how to circumvent this please ? The Insight in the /target/ folder works fine when double clicked.

[mtbcarroll]

We can compare Marathon project setups today. It works for me now but it took a couple of tries and I may have found myself a nasty solution instead of a good one.

[mtbcarroll]

here are some of my current settings for the project

[mtbcarroll]

Okay, I am having some success with SwingLibrary. While not being sure how I overcame previous problems, I now have a script that, when it starts Insight, checks to see if we are logging into localhost, and if not it opens the server dialog and chooses it, then in either case it logs in okay. For instance,

    ${server}=  Get Text Field Value  server host name
    Run Keyword If  '${server}' != 'localhost'  Choose Localhost Server

I wonder how to write that in Marathon.

With SwingLibrary I figure that the next steps are,

1. try rewriting my script to have a basic logged-into-Insight test, such as an initial fragment of #11214 that assumes some previous imports, occurring with the existing login script as a pre-test fixture parameterized by OMERO user name
2. try rewriting the script in reStructuredText format so that it can then be easily integrated into internal Sphinx documentation
3. explain it all to pwalczysko this week.

Another, hopefully unnecessary, course of action would be,

4. try to port the script to Marathon.

I could certainly have a go at that though, replacing gaps in documentation with some intelligent guesses.

[mtbcarroll]

With SwingLibrary I've made some progress in separating logging into OMERO through Insight out into parameterized test setup, and I have made a start with the suggested tests of #11214, but I have run into an odd issue with testing which popup menu items are enabled that I will try to figure out a fix for tomorrow. I have also run into an issue where test suite teardown won't like actually quitting out of Insight (the System.exit seems to be bringing some test stuff down too), about which I enquire in ​https://groups.google.com/forum/#!topic/robotframework-users/cRwRPE5IeFo

[mtbcarroll]

The popup menu item issue is proving tricky. I think that Insight is doing the right thing: even under the control of Jemmy the popup menu doesn't get shown until the action underlying the "Orphaned Images -> Import..." option is disabled, but TreeNodePopupKeywords.treeNodePopupMenuItemShouldBeDisabled throws java.lang.AssertionError: Menu item 'Import...' was enabled.

Before I dig into SwingLibrary's code to see if there is a problem apparent there, first I shall try out ​https://github.com/Rethought/swinglibrary_package because I think I am seeing a related issue at the same time in the test case script.

[mtbcarroll]

Okay, the Rethought script does seem to correct that old-Abbot issue, even for SwingLibrary 1.6.0, but doesn't fix the popup menu item issue that is the real problem, so tomorrow I'll have to dig into SwingLibrary some.

Still, for anybody else trying to use the Rethought script on a Mac: it expects wget and GNU sed, both of which are available through homebrew.

[mtbcarroll]

This popup menu item issue is such a small thing given how otherwise adequate SwingLibrary is seeming, and we have all the source so it should be quite solvable. Gah. When I find the solution I expect that I will have learned something new.

With my abbreviated test script, in PopupMenu.createMenuItems during the run of the test script Insight is constructing only one Import... JMenuItem. I attach a property change listener to it that does seem to see changes to its action's enabled property. I even create this instance as a special subclass that overrides isEnabled and setEnabled to keep them false and I adjust ImportAction similarly.

Then, SwingLibrary does manage to find a JMenuItem instance with Import... text and its isEnabled() returns true! So, I wonder if this instance isn't actually the one originally created by Insight; if so, then from where does it hail?

[mtbcarroll]

Another framework to investigate is ​http://fest.easytesting.org/ which does seem to have seen some activity this year despite the age of the releases in ​http://code.google.com/p/fest/downloads/list