Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11210 (closed)

Opened 8 years ago

Closed 8 years ago

Automated GUI testing of Insight

Reported by: mtbcarroll Owned by:
Priority: major Milestone: Testing and Docs
Component: Insight Version: 4.4.8
Keywords: n.a. Cc: jrswedlow, ux@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: Testing and Docs (2)

Description (last modified by mtbcarroll)

Investigate tools like

  • FEST
  • Jacareto
  • Marathon
  • Maveryx
  • Robot Framework (Swing Library)
  • WindowTester

and work with Petr to get workflow / test scenarios running in such software for automated testing of Insight.

Attachments (8)

Marathon - Config App - Main.png (196.6 KB) - added by mtbcarroll 8 years ago.
here are some of my current settings for the project
Marathon - Config App - Class Path.png (55.9 KB) - added by mtbcarroll 8 years ago.
fixture.rest (7.3 KB) - added by mtbcarroll 8 years ago.
test fixture
test-11214.rest (4.7 KB) - added by mtbcarroll 8 years ago.
simple test case
report.html (192.8 KB) - added by mtbcarroll 8 years ago.
report of running tests
log.html (193.7 KB) - added by mtbcarroll 8 years ago.
execution log for single test
swinglibrary-1.6.0a.jar (3.6 MB) - added by mtbcarroll 8 years ago.
replacement Swing library JAR, corrects issue with old Abbot on Java 7
green.png (103 bytes) - added by mtbcarroll 8 years ago.
solid green image for testing image blankness

Change History (62)

comment:1 Changed 8 years ago by mtbcarroll

My current plan is to give each of the above four a few hours' reading and figuring out and playing to see if it seems viable, and, of those that do, a few more hours to set up something simple and share it with Petr, and encourage him to be thinking in terms of the automatable test categories provided. Then, we can decide, with which of these packages, if any, to plunge in further with our first real test scenarios.

comment:2 Changed 8 years ago by mtbcarroll

  • Cc jrswedlow added
  • Status changed from new to accepted

keep Jason in the loop regarding plans, progress

comment:3 Changed 8 years ago by mtbcarroll

After today, until the 22nd instant I have only three working days, so initial progress will be slow. Nevertheless, this month we should be able to select a framework and get simple examples working in it. Then, more can be added next month, and we can write developer documentation about how to do that.

comment:4 Changed 8 years ago by jburel

  • Sprint set to Testing and Docs (1)

comment:5 Changed 8 years ago by mtbcarroll

  • Description modified (diff)

Much as I hate to add another to the list, bpindelski points out WindowTester and that does also look good. I guess it's worth making this initial framework choice well, but I'll try not to go far down many rabbit holes.

comment:6 Changed 8 years ago by mtbcarroll

Jacareto looks less actively maintained than I'd first thought, I'll ignore it for now unless others are found wanting.

comment:7 Changed 8 years ago by mtbcarroll

Hmmm, Marathon have a separate commercial version if one wants things like object map maintenance.

comment:8 Changed 8 years ago by mtbcarroll

Maveryx is on sourceforge. It's frustratingly difficult to get to free, decent technical documentation for, though, and the user manual's from 2011.

comment:9 Changed 8 years ago by mtbcarroll

Interestingly, Robot Framework at https://code.google.com/p/robotframework/ offers both Selenium2 scripting at https://github.com/rtomac/robotframework-selenium2library and Swing scripting at https://github.com/robotframework/SwingLibrary

As Insight and Web continue to converge, I don't know if that affords any opportunity to bring them closer in automated testing too. I'll ask Will.

comment:10 Changed 8 years ago by mtbcarroll

WindowTester looks not to have had a release in over a year and the FAQ is somewhat out of date.

comment:11 Changed 8 years ago by mtbcarroll

The Robot Framework SwingLibrary doesn't come with a nice GUI recorder of app interactions, but that could be tolerable if some other tool makes it easy to find what is needed to specify the GUI interactions in the test scripts. A quick glance at a couple of such tools doesn't fill me with joy, but I've not yet looked hard; hopefully there's an Eclipse plugin that makes Swing component properties easily browsed. A separate but related issue is that it may behoove us to pervade Insight with more Component.setName().

(I'm not assuming that an "easy" recorder tool would insulate us from the Swing component details anyway if the hope is to write scripts that don't break every time the UI changes a little.)

comment:12 Changed 8 years ago by mtbcarroll

I'll persist with Robot Framework for now, falling back to Marathon if need be.

If I install Robot Framework with jython setup.py install then I seem to be able to execute scripts from components/insight/OUT/app/ if I use jybot with swinglibrary-1.6.0.jar and compiled and libs/* in my Java classpath.

I can also get Swing Explorer working okay as an Eclipse plug-in, to explore the component hierarchy.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:13 Changed 8 years ago by mtbcarroll

SwingLibrary is looking very workable, though there is some initial investment in adding Component.setName() calls around the code; this probably makes it easier to write robust test scripts for any automated UI testing tool. I have run into one issue, probably with Jemmy that it uses under the hood; it is noted at https://github.com/robotframework/SwingLibrary/issues/43

comment:14 Changed 8 years ago by mtbcarroll

Progress in adding Component.setName() is at https://github.com/mtbc/openmicroscopy/compare/openmicroscopy:dev_4_4...Component.setName

Chapter 2 of http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.8.1 and the SwingLibrary keywords at http://robotframework.org/SwingLibrary/SwingLibrary-1.6.0.html cover how I have been writing test scripts. For instance,

*** Settings ***
Library  SwingLibrary

*** Test Cases ***
Log In
    Launch Application  org.openmicroscopy.shoola.Main
    Select Window  login window
    List Components In Context  formatted
    Button Should Be Enabled  config server button
#    Click On Component  config server button
#    Select Dialog  server dialog
#    List Components In Context  formatted
#    Button Should Be Disabled  apply button
#    ${localhost}=  Find Table Row  server table  localhost  host
#    Click On Table Cell  server table  ${localhost}  0
#    Button Should Be Enabled  apply button
#    Click On Component  apply button
#    Select Window  login window
    Clear Text Field  user
    Type Into Text Field  user  root
    Type Into Text Field  pass  omero
    List Components In Context  formatted
    Click On Component  login button
    Select Window  top window
    List Components In Context  formatted

to try to log in and see the names of the main window Swing components. (The commenting-out in the script is due to the Jemmy issue referenced above.)

Component names are also revealed through http://www.swingexplorer.com/ which works okay as a plug-in for my Eclipse Indigo.

In contrast to the above, note that test scripts may be written in reStructuredText format which may be a great convenience if we wish to include them in our Sphinx documentation.

So, with regard to Robot Framework, pwalczysko, I'd suggest you

  • assume I can give names to whichever parts of the Insight UI we wish to test or prod, even the little yellow annotation marking in thumbnail title bars, but preferably not a hundred different ones at first
  • review the list of keywords offered by SwingLibrary (plenty of "checking" ones as well as "doing" ones)
  • think about what simple test scripts we could start out with (note #11214).

Hopefully the above Jemmy problem with Click On Component will be addressed in the meantime or otherwise revealed to be easily solved.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:15 Changed 8 years ago by mtbcarroll

Marathon has various uglinesses, including doing cut-and-paste in text fields with control instead of command, that make me guess that it might be nicer on Windows. The testing projects appear to use files that are variously in CSV, XML, YAML. The manual's somewhat Ruby- and not-Mac-centric, but I think it's still workable enough for Python test scripts on a Mac.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:16 Changed 8 years ago by mtbcarroll

With SwingLibrary I appear able to do things like,

    ${localhost}=  Find Table Row  server table  localhost  host
    Click On Table Cell  server table  ${localhost}  0

to choose a server from the list at login. From the Marathon documentation it is extremely unclear how to emulate this. It covers the simple stuff somewhat well, but, unless I'm missing something, the technical detail stops a step short of what we would actually need. For instance, select can "set the state corresponding to the given text" but nowhere in the manual have I yet found specification of if I should expect to be able to do things like,

select('server table', 'rows:[localhost],columns:[host]')
Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:17 Changed 8 years ago by mtbcarroll

With regard to Marathon, pwalczysko, I'd suggest that you briefly peruse the manual from the bottom of http://marathontesting.com/downloads/ to get an idea of what it is. Also, should you feel moved to, don't feel inhibited from taking a look as best you can at other Java UI testing frameworks, especially ones that seem to be actively maintained and with a growing user base. (For instance, how old's the manual? How busy are their mailing lists?)

To help you to compare with my above script quotes for SwingLibrary, "Script Methods" at the beginning of Appendix A in that Marathon manual is largely equivalent to the list of keywords for SwingLibrary, but, unfortunately, not in such detail.

Marathon does provide a "recorder" so that you can do things in Insight and then it writes you a script, but it doesn't buy us a whole lot: we would still need to work on that script manually in non-trivial ways because it doesn't know what we were really aiming for when we did things -- i.e. how what we did was affected by what we saw. For instance, in the server list, was I aiming for "localhost", or for the seventh row down, or what? Did I open that server dialog only because "localhost" wasn't already selected? That kind of test fixture thing ends up needing to be hand-coded.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:18 Changed 8 years ago by mtbcarroll

Marathon's IDE seems a bit strange and clunky too. So far I seem to have caused it to generate scripts that it can't then execute even with its own bundled Jython. (Naturally, project settings are on the unhelpfully named "Marathon" menu.)

Barring reasons to change course, I think that my initial plan upon returning from leave would be to plug a debugger into the latest Jemmy 2.x to see if I can easily find a fix for the current issue with getting Robot Framework to run my SwingLibrary script, then to try building a conditional branching into it.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:19 Changed 8 years ago by pwalczysko


  • Read the document as indicated above
  • cloned the git repository and managed to start marathon from cmd line (does not work from the downloaded app - will open and immediately close)
  • created a project and tried to aim it at my local Insight, namely at omero.insight.jar in
    • clicked the "Record" button in Marathon
    • it ends up with an error
      java.io.IOException: Cannot run program "/Users/petr/Work/openmicroscopy/target/OMERO.insight-4.4.8-DEV-ice33-mac/OMERO.insight.app/Contents/Resources/Java/omero.insight.jar" (in directory "/Users/petr/Work/openmicroscopy/target/OMERO.insight-4.4.8-DEV-ice33-mac/OMERO.insight.app/Contents/Resources/Java" error=13, Permission denied
         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
         at net.sourceforge.marathon.runtime.JavaRuntimeFactory.launchVM(JavaRuntimeFactory.java:94)
         at net.sourceforge.marathon.runtime.JavaRuntimeFactory.createRuntime(JavaRuntimeFactory.java:64)
         at net.sourceforge.marathon.display.Display.createRuntime(Display.java:364)
         at net.sourceforge.marathon.display.Display.record(Display.java:224)
         at net.sourceforge.marathon.display.DisplayWindow.onRecord(DisplayWindow.java:2936)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:601)
         at net.sourceforge.marathon.display.ActionInjector$1.actionPerformed(ActionInjector.java:92)
         at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
         at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
         at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
         at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
         at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
         at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:290)
         at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
         at java.awt.Component.processMouseEvent(Component.java:6505)
         at javax.swing.JComponent.processMouseEvent(JComponent.java:3321)
         at java.awt.Component.processEvent(Component.java:6270)
         at java.awt.Container.processEvent(Container.java:2229)
         at java.awt.Component.dispatchEventImpl(Component.java:4861)
         at java.awt.Container.dispatchEventImpl(Container.java:2287)
         at java.awt.Component.dispatchEvent(Component.java:4687)
         at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
         at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
         at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
         at java.awt.Container.dispatchEventImpl(Container.java:2273)
         at java.awt.Window.dispatchEventImpl(Window.java:2719)
         at java.awt.Component.dispatchEvent(Component.java:4687)
         at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:723)
         at java.awt.EventQueue.access$200(EventQueue.java:103)
         at java.awt.EventQueue$3.run(EventQueue.java:682)
         at java.awt.EventQueue$3.run(EventQueue.java:680)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
         at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
         at java.awt.EventQueue$4.run(EventQueue.java:696)
         at java.awt.EventQueue$4.run(EventQueue.java:694)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
         at java.awt.EventQueue.dispatchEvent(EventQueue.java:693)
         at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
         at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
         at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
         at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
         at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
         at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
      Caused by: java.io.IOException: error=13, Permission denied
         at java.lang.UNIXProcess.forkAndExec(Native Method)
         at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
         at java.lang.ProcessImpl.start(ProcessImpl.java:130)
         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1021)
         ... 48 more

Did you sort out how to circumvent this please ? The Insight in the /target/ folder works fine when double clicked.

Last edited 8 years ago by pwalczysko (previous) (diff)

comment:20 Changed 8 years ago by mtbcarroll

We can compare Marathon project setups today. It works for me now but it took a couple of tries and I may have found myself a nasty solution instead of a good one.

Last edited 8 years ago by mtbcarroll (previous) (diff)

Changed 8 years ago by mtbcarroll

here are some of my current settings for the project

Changed 8 years ago by mtbcarroll

comment:21 Changed 8 years ago by mtbcarroll

Okay, I am having some success with SwingLibrary. While not being sure how I overcame previous problems, I now have a script that, when it starts Insight, checks to see if we are logging into localhost, and if not it opens the server dialog and chooses it, then in either case it logs in okay. For instance,

    ${server}=  Get Text Field Value  server host name
    Run Keyword If  '${server}' != 'localhost'  Choose Localhost Server

I wonder how to write that in Marathon.

With SwingLibrary I figure that the next steps are,

  1. try rewriting my script to have a basic logged-into-Insight test, such as an initial fragment of #11214 that assumes some previous imports, occurring with the existing login script as a pre-test fixture parameterized by OMERO user name
  2. try rewriting the script in reStructuredText format so that it can then be easily integrated into internal Sphinx documentation
  3. explain it all to pwalczysko this week.

Another, hopefully unnecessary, course of action would be,

  1. try to port the script to Marathon.

I could certainly have a go at that though, replacing gaps in documentation with some intelligent guesses.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:22 Changed 8 years ago by mtbcarroll

With SwingLibrary I've made some progress in separating logging into OMERO through Insight out into parameterized test setup, and I have made a start with the suggested tests of #11214, but I have run into an odd issue with testing which popup menu items are enabled that I will try to figure out a fix for tomorrow. I have also run into an issue where test suite teardown won't like actually quitting out of Insight (the System.exit seems to be bringing some test stuff down too), about which I enquire in https://groups.google.com/forum/#!topic/robotframework-users/cRwRPE5IeFo

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:23 Changed 8 years ago by mtbcarroll

The popup menu item issue is proving tricky. I think that Insight is doing the right thing: even under the control of Jemmy the popup menu doesn't get shown until the action underlying the "Orphaned Images -> Import..." option is disabled, but TreeNodePopupKeywords.treeNodePopupMenuItemShouldBeDisabled throws java.lang.AssertionError: Menu item 'Import...' was enabled.

Before I dig into SwingLibrary's code to see if there is a problem apparent there, first I shall try out https://github.com/Rethought/swinglibrary_package because I think I am seeing a related issue at the same time in the test case script.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:24 Changed 8 years ago by mtbcarroll

Okay, the Rethought script does seem to correct that old-Abbot issue, even for SwingLibrary 1.6.0, but doesn't fix the popup menu item issue that is the real problem, so tomorrow I'll have to dig into SwingLibrary some.

Still, for anybody else trying to use the Rethought script on a Mac: it expects wget and GNU sed, both of which are available through homebrew.

comment:25 Changed 8 years ago by mtbcarroll

This popup menu item issue is such a small thing given how otherwise adequate SwingLibrary is seeming, and we have all the source so it should be quite solvable. Gah. When I find the solution I expect that I will have learned something new.

With my abbreviated test script, in PopupMenu.createMenuItems during the run of the test script Insight is constructing only one Import... JMenuItem. I attach a property change listener to it that does seem to see changes to its action's enabled property. I even create this instance as a special subclass that overrides isEnabled and setEnabled to keep them false and I adjust ImportAction similarly.

Then, SwingLibrary does manage to find a JMenuItem instance with Import... text and its isEnabled() returns true! So, I wonder if this instance isn't actually the one originally created by Insight; if so, then from where does it hail?

comment:26 Changed 8 years ago by mtbcarroll

  • Description modified (diff)

Another framework to investigate is http://fest.easytesting.org/ which does seem to have seen some activity this year despite the age of the releases in http://code.google.com/p/fest/downloads/list

comment:27 Changed 8 years ago by mtbcarroll

Well, the JMenuItem instance with Import... text found by the testing framework certainly isn't the same instance originally created by Insight.

comment:28 Changed 8 years ago by jburel

github account for FEST https://github.com/alexruiz

comment:29 Changed 8 years ago by jburel

Mark: https://guts.kenai.com/guts-base/guts-gui/apidocs/net/guts/gui/naming/package-summary.html that could be useful if we decide to use RobotFramework? to name all the components

comment:30 Changed 8 years ago by mtbcarroll

For net.guts.gui.naming we'll have to investigate the depth of the Guice dependency.

For using fields in Insight like my local branch's,

public class ImportAction
    public static final String NAME = "Import...";

it turns out that in Robot Framework I can add a user keyword thus,

public class JavaSupportLibrary
    public static final String ROBOT_LIBRARY_SCOPE = "GLOBAL";   

    public static String staticValue(String className, String fieldName)
    throws ClassNotFoundException, IllegalAccessException, NoSuchFieldException {
	return (String) Class.forName(className).getField(fieldName).get(null);

and then use it as,

    ${import}=  Static Value  org.openmicroscopy.shoola.agents.treeviewer.actions.ImportAction  NAME
    Tree Node Popup Menu Item Should Be Disabled  tree  root root|Orphaned Images  ${import}

It would be easy to offer an Insight-specific keyword that prefixes the class name argument from the script with org.openmicroscopy.shoola.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:31 Changed 8 years ago by mtbcarroll

For investigating this popup menu issue, I still run jybot largely as described in comment 12, but I add some System.err.println (yes, really!) to the source with the help of two things:

I have a checkout of git://github.com/robotframework/SwingLibrary and, so that I can easily investigate the target menu item, I have replaced the use of createPopupMenuItem in TreeNodePopupKeywords.menuItemIsEnabled more in the vein of,

    final Component tree = treeOperator(identifier).getSource();
    final JTreeLocation treeLocation = new JTreeLocationFactory().parseArgument(nodeIdentifier);
    new JComponentTester().actionShowPopupMenu(tree, treeLocation);
    final JMenuItem menuItem = (JMenuItem) new BasicFinder().find(new JMenuItemMatcher(menuPath));
    final boolean isEnabled = menuItem.isEnabled();
    return isEnabled;

This I build with mvn clean package -DskipTests.

Then, I have a checkout of git://github.com/Rethought/swinglibrary_package and I have hacked build.sh to copy a local Abbot (instead of checking it out each time) and to use SwingLibrary 1.6.0, also copied locally. Note that the script expects GNU sed (brew install gnu-sed) rather than a BSD sed as comes with one's Mac. Using the hacked SwingLibrary this script creates removes an annoying delay and warning caused by using an old Abbot with Java 7. However, it would be okay to start out by skipping this step.

So, in checking things out, I can do things like, in view.PopupMenu,

    ((JMenuItem) importElement).addPropertyChangeListener(new PropertyChangeListener() {
        public void propertyChange(PropertyChangeEvent event) {
            if (event.getPropertyName().equals("enabled")) {
                System.err.println("import element: " + event.getPropertyName() + "=" + event.getNewValue());

comment:32 Changed 8 years ago by mtbcarroll

I think that my provisional plan for tomorrow is to ignore this popup menu puzzlement and to package the little I have nicely for others to look at. Given that the test framework source code is all easily available and quite readable, perhaps somebody else can figure out why that test script is failing when it really looks as if it ought not.

comment:33 Changed 8 years ago by jburel

After having a closer look at FEST and following a discussion with Petr, we have decided not to consider the tool, mainly b/c we have to write the tests using Java syntax.

We might reconsider it if we cannot handle the cases we wish to cover using the Robot framework

comment:34 Changed 8 years ago by jburel

I came across Window Tester Pro https://developers.google.com/java-dev-tools/wintester/html/?csw=1
I will play a bit with it

comment:35 Changed 8 years ago by mtbcarroll

Hmm, yes, if you look at the beta version and the subversion repository WindowTester is not as unmaintained as first impressions suggest.

comment:36 Changed 8 years ago by mtbcarroll

It seems a good time to share a simple case for others to start playing with. I shifted format to reStructuredText markup which makes it easy to include the tables in Sphinx using ..include:: and to make this work for jybot I had to install docutils for Jython. There are probably plenty of tools that make editing simple ASCII tables easy; I used emacs' table stuff.

This simple example isn't initially ideal because it tries to log in to Insight before each test, which obviously won't work the second time around, but it does allow some initial playing.

Changed 8 years ago by mtbcarroll

test fixture

Changed 8 years ago by mtbcarroll

simple test case

Changed 8 years ago by mtbcarroll

report of running tests

Changed 8 years ago by mtbcarroll

execution log for single test

Changed 8 years ago by mtbcarroll

replacement Swing library JAR, corrects issue with old Abbot on Java 7

comment:37 Changed 8 years ago by mtbcarroll

Split out popup menu issue into ticket #11326.

comment:38 Changed 8 years ago by mtbcarroll

One interesting question is if there is a better way to get Insight's AWT components named for reference by test scripts than by manually adding setName invocations in the code.

comment:39 Changed 8 years ago by mtbcarroll

Another interesting question is test fixture setup stuff. Insight can retain state across "switch user". Perhaps there should be some outside script that runs a suite with multiple jybot invocations that each start up Insight again and log in to the server (only takes a few seconds) and that then do many script steps while they have Insight open. Once they are all run, their reports can be somehow collated.

And, with the database, it would be nice to purge it in between each jybot invocation. Possibly Colin's scripting work can help with easy setup of users and data after each purge. Though, one important question is that of populating the database with data rapidly. For instance, can the recent work on fake screens provide test data for test scripts that don't require waiting for real screens to be imported? What about, say, ROIs? Can we get the data for test scripts into the server quickly enough if it is wiped each time?

Will has thoughts about all this, and of course is facing the same issue in #11314, though partly through the use of helpers like #11317. (His tests each start in a new "browser".)

comment:40 Changed 8 years ago by mtbcarroll

My next move is to try developing a different test script that explores questions other than which menu items are enabled in popup menus. We'll then see if I run into a different surprising bug / behavior.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:41 Changed 8 years ago by mtbcarroll

I have now managed to write a keyword that allows checking that icons are correct. It may work only for certain cases, and uses Abbot under the hood, but it looks rather likely that covering other cases won't be any more difficult. One can now write script steps like,

${icon_name}=  Get Image Icon Name  administration browser
Should Be Equal  ${icon_name}  nuvola_personal16.png

I wonder how hard it is to write a keyword that, for instance, checks that an image (thumbnail, whatever) isn't monochromatic.

So, I am accumulating these *.rest files, and also now some *.java source files; the Get Image Icon Name above likes to be compiled against both the SwingLibrary JAR and the swingx JAR that Insight uses, though it doesn't need Insight itself. To where should I be saving and pushing these files to share with others? I can use squig for now. I would venture that the test cases and the keyword implementations should be in the same repository, probably the main openmicroscopy one, but its build system intimidates me and I don't want to spread the SwingLibrary dependency further than necessary.

Changed 8 years ago by mtbcarroll

solid green image for testing image blankness

comment:42 Changed 8 years ago by mtbcarroll

Recent work is being pushed to the more generally named https://github.com/mtbc/openmicroscopy/commits/Insight-UI-testing-11210

This includes new user keywords like Thumbnail Should Not Be Blank and Thumbnails Should Be Identical.

comment:43 Changed 8 years ago by mtbcarroll

  • Sprint changed from Testing and Docs (1) to Testing and Docs (2)

comment:44 Changed 8 years ago by mtbcarroll

Pushed code to above branch that checks synchronization between tree viewer node selection and thumbnail selection.

comment:45 Changed 8 years ago by mtbcarroll

A note to those implementing new Robot Framework keywords in Java: within the SwingLibrary JAR the Abbot and Jemmy classes, as used by keywords provided by SwingLibrary, are available underneath the org.robotframework package. (They would be elsewhere in vanilla Abbot and Jemmy JARs.)

comment:46 Changed 8 years ago by mtbcarroll

The above branch now includes fixes for #11326 and #11359. Next week I will try to get Insight test cases running that fit the web scripts in http://github.com/will-moore/openmicroscopy/tree/Insight-UI-testing-11210

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:47 Changed 8 years ago by mtbcarroll

Jean-Marie points out the possibility of using Robot Framework to also test Insight launched through Web Start.

comment:49 Changed 8 years ago by mtbcarroll

I had some trouble getting that working with SwingLibrary but I may well have not tried hard enough. https://github.com/ombre42/jrobotremoteserver/wiki/User-Guide may also offer something useful in all that, I don't know.

comment:50 Changed 8 years ago by mtbcarroll

We have decided to avoid the restructured text format tables and use a more compact format for writing test cases. If Sphinx documentation requires tables someday, they can easily be generated from the *.txt file format. Note that Sublime and Emacs (http://github.com/sakari/robot-mode) have modes for editing Robot Framework text files.

I get a good start on converting the *.rest files by using,

for i in `find . -type f -name '*.rest'` ; do <$i grep -v -- --- | grep -v === | sed -e 's/\ \.\.\ /    /' -e 's/\ \|\ /  /g' -e 's/^\| //' -e 's/[[:space:]]*\|$//' >`echo $i | sed -e 's/\.rest/.txt/'` ; done

comment:51 Changed 8 years ago by mtbcarroll

A start on the Component.setName(String) work is now in https://github.com/openmicroscopy/openmicroscopy/pull/1410

On my separate branch I now additionally have a simple test case for creating and deleting projects. The test cases are all passing.

Jean-Marie and Kenny have put much good work into creating the ant build scripts around the UI testing and setting up the continuous integration node so that Jenkins may run the tests automatically.

My focus next week will be on adjusting the Insight test scripts -- what they do and how they are written -- to match the Robot Framework web tests as much as they conveniently can.

Another outstanding issue is the setting of the config.txt file from OMERO.setup.

Also, Petr, the principal end user of all this, is back next week.

comment:52 Changed 8 years ago by mtbcarroll

Now opened http://github.com/openmicroscopy/openmicroscopy/pull/1424 which captures much of the above work, implementing ant targets that run UI tests.

Insight keywords, thus test cases, remain to be harmonized with web testing. Integration with OMERO.setup via existing config.txt not yet in place. Jenkins node for UI testing nearly ready to go. Some of these pending issues may warrant their own specific ticket, allowing this one to be closed.

Last edited 8 years ago by mtbcarroll (previous) (diff)

comment:53 Changed 8 years ago by mtbcarroll

  • Owner mtbcarroll deleted

I am moving on to bugfixing for the 4.4.9 release, anyone can pick up the remaining work when time allows, but at least I do now have the PR open that has build targets that run automated UI tests.

comment:54 Changed 8 years ago by jburel

  • Resolution set to fixed
  • Status changed from accepted to closed

ticket created for comment 52 see #11406

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.82451 sec.)

We're Hiring!