Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11271 (closed)

Opened 8 years ago

Closed 6 years ago

Evaluate integration/unit testing frameworks

Reported by: bpindelski Owned by:
Priority: minor Milestone: Testing2
Component: General Version: 4.4.8
Keywords: n.a. Cc: java@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.


Currently integration and unit tests use two frameworks: jUnit and TestNG.

  • Compare the which framework is used more often and see if the codebase could be move to just using one,
  • Evaluate if Cucumber, FitNesse? or CodePro? could in any way provide a valuable alternative to vanilla jUnit/TestNG tests,
  • If staying with jUnit or TestNG, update the JAR (and fix any tests that could have broken),
  • Check if moving to JDK6 or JDK7 would have any impact on structuring tests.

Change History (10)

comment:1 Changed 8 years ago by bpindelski

  • Status changed from new to accepted

comment:2 Changed 8 years ago by bpindelski

An initial evaluation of Fitnesse has produced the following results:

  • as a tool for business analysts, it is very useful and provides a very quick and easy way to write acceptance tests (only method and class names are in use - the underlying code is the responsibility of the module developer),
  • the fact that it can be run as a stand-alone server makes it very portable,
  • the visual indication of failing tests in the manner of tables is an efficient way to provide feedback.

On the other hand, there are several major drawbacks:

  • it is a wiki system, so that implies learning new wiki syntax (it's not markdown) and writing each test case as a wiki page (quite time consuming, as it uses table forms - see below),
  • almost everything (besides some descriptive text) is a table (as fitnesse is based on the Fit testing framework) - that doesn't help the look and feel; more time is spend on creating tables with the "|" symbol than putting in logic,
  • it would be very useful in a greenfield project; migrating existing integration tests will be an enormous amount of work,
  • as it is also a webserver, the JAR file has to be executed in a separate JVM first to be able to run tests and present results,
  • there is only a JUnit (no TestNG) plugin class that allows Fitnesse to consume Junit assertion results.

In general, Fitnesse is an interesting tool with a goal of enabling and increasing communication between business teams and developers. The way a test fixture is written and presented is helpful for non-developers. pwalczysko might be interested in trying to move one scenario and substituting each "Check" sentence with a fitnesse assertion, but that only applies to simple checks. From a developer's perspective, this tool wouldn't bring any value unless we start writing integration tests from scratch and using the fitnesse taxonomy for class and method names. Fitnesse has plug-ins for Python and Spring, but the 2009 date on them isn't encouraging.

Last edited 8 years ago by bpindelski (previous) (diff)

comment:3 Changed 8 years ago by bpindelski

In the next step, Cucumber together with the Gherkin DSL was considered. Cucumber is a Ruby rewrite of jBehave (might be also worthwhile to look at that). The project provides two JAR files that are relevant to the Java tests - cucumber-core and cucumber-java. Additionally, a runner is needed if the tests are meant to be executed inside a testing framework (e.g. JUnit). There is a project on github that is meant to add TestNG support to cucumber-jvm (https://github.com/cucumber/cucumber-jvm/pull/526). cucumber-jvm.jar is not available on maven central though. A quick evaluation will consist of trying to write a simple Gherkin file and run it against a Junit test class.

comment:4 Changed 8 years ago by bpindelski

Cucumber has been considered a slight disappointment. There is value in the Gherkin syntax for writing human-readable test cases for scenarios. Deployment together with OmeroJava was very unpleasant (https://github.com/bpindelski/openmicroscopy/tree/cucumber). The build.xml file would have to be substantially modified. Getting Cucumber to start up was possible, but on .feature file execution time ClassNotFound? errors were thrown for some of the cucumber-java.jar dependencies. It is a tool worth keeping an eye at, but probably easier to use in a Ruby environment. The next step will be trying to integrate jBehave.

comment:5 Changed 8 years ago by bpindelski

A general consensus was that introducing BDD tools will increase the maintenance burden of keeping the tests up-to-date. Although it would be nice to describe certain functions of the API in natural language (e.g. interactions with the Admin service wrt. permissions), it is not currently the priority. The goal should be major refactoring with more modern testing/mocking frameworks (e.g. JUnit, TestNG, jMock, Mockito) with the possibility of introducing BDD concepts in the future. The effort will now focus on opening a branch based on dev_4_4 with OmeroJava restructured/updated.

comment:6 Changed 8 years ago by bpindelski

  • Sprint changed from Testing and Docs (1) to Testing and Docs (2)

Referencing ticket #11258 has changed sprint.

comment:7 Changed 8 years ago by bpindelski

During the OmeroJava refactoring effort in-memory DBs have been investigated: H2 and HSQLDB. Both would provide a portable way of executing integration tests (no need for a running PostgreSQL instance and ICE_CONFIG). Additionally, the schema could be created for each suite of tests (e.g. chgrp, permissions etc.) thus guaranteeing the same conditions in which is suite is run. Finally, it would be a work-around for the lack of user delete functionality. Sadly, the SQL schema that is generated by bin/omero db script makes use of PostgreSQL native data types. Even setting the compatibility modes in both DBMS resulted in errors during the parsing of the schema. If we ever end up with a schema less locked to Postgres, then H2 would be the strongest contender to use as an in-memory DB.

comment:8 Changed 8 years ago by jamoore

  • Cc java@… added
  • Milestone changed from Testing and Docs to Testing2
  • Sprint Testing and Docs (2) deleted

As I understood it, we've largely decided to put this on the backburner for the moment while we refactor our tests to see what's necessary in the way of a framework. Moving to milestone:Testing2. If that's incorrect, Blazej, please feel free to move it back as you see fit.

comment:9 Changed 8 years ago by bpindelski

In general, I have a feeling that it will be safe to start moving towards TestNG throughout the openmicroscopy codebase (excluding BioFormats). There is an easy way of automatic conversion from JUnit to TestNG in Eclipse. It also will allow to make the style look more consistent and remove the discrepancy between Java imports (annotations from TestNG and assertions from JUnit). This is one of the maintenance tasks after OmeroJava gets moved to a new component.

comment:10 Changed 6 years ago by sbesson

  • Resolution set to fixed
  • Status changed from accepted to closed

From https://www.openmicroscopy.org/site/support/omero5.2/developers/testing.html, TestNG is definitely marked as the default framework for our Java tests. We'll keep reviewing existing tests as we review our code.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.67465 sec.)

We're Hiring!