Task #11271 (closed)
Opened 11 years ago
Closed 8 years ago
Evaluate integration/unit testing frameworks
Reported by: | bpindelski | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | Testing2 |
Component: | General | Version: | 4.4.8 |
Keywords: | n.a. | Cc: | java@… |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | n.a. |
Sprint: | n.a. |
Description
Currently integration and unit tests use two frameworks: jUnit and TestNG.
- Compare the which framework is used more often and see if the codebase could be move to just using one,
- Evaluate if Cucumber, FitNesse? or CodePro? could in any way provide a valuable alternative to vanilla jUnit/TestNG tests,
- If staying with jUnit or TestNG, update the JAR (and fix any tests that could have broken),
- Check if moving to JDK6 or JDK7 would have any impact on structuring tests.
Change History (10)
comment:1 Changed 11 years ago by bpindelski
- Status changed from new to accepted
comment:2 Changed 11 years ago by bpindelski
comment:3 Changed 11 years ago by bpindelski
In the next step, Cucumber together with the Gherkin DSL was considered. Cucumber is a Ruby rewrite of jBehave (might be also worthwhile to look at that). The project provides two JAR files that are relevant to the Java tests - cucumber-core and cucumber-java. Additionally, a runner is needed if the tests are meant to be executed inside a testing framework (e.g. JUnit). There is a project on github that is meant to add TestNG support to cucumber-jvm (https://github.com/cucumber/cucumber-jvm/pull/526). cucumber-jvm.jar is not available on maven central though. A quick evaluation will consist of trying to write a simple Gherkin file and run it against a Junit test class.
comment:4 Changed 11 years ago by bpindelski
Cucumber has been considered a slight disappointment. There is value in the Gherkin syntax for writing human-readable test cases for scenarios. Deployment together with OmeroJava was very unpleasant (https://github.com/bpindelski/openmicroscopy/tree/cucumber). The build.xml file would have to be substantially modified. Getting Cucumber to start up was possible, but on .feature file execution time ClassNotFound? errors were thrown for some of the cucumber-java.jar dependencies. It is a tool worth keeping an eye at, but probably easier to use in a Ruby environment. The next step will be trying to integrate jBehave.
comment:5 Changed 11 years ago by bpindelski
A general consensus was that introducing BDD tools will increase the maintenance burden of keeping the tests up-to-date. Although it would be nice to describe certain functions of the API in natural language (e.g. interactions with the Admin service wrt. permissions), it is not currently the priority. The goal should be major refactoring with more modern testing/mocking frameworks (e.g. JUnit, TestNG, jMock, Mockito) with the possibility of introducing BDD concepts in the future. The effort will now focus on opening a branch based on dev_4_4 with OmeroJava restructured/updated.
comment:6 Changed 11 years ago by bpindelski
- Sprint changed from Testing and Docs (1) to Testing and Docs (2)
Referencing ticket #11258 has changed sprint.
comment:7 Changed 11 years ago by bpindelski
During the OmeroJava refactoring effort in-memory DBs have been investigated: H2 and HSQLDB. Both would provide a portable way of executing integration tests (no need for a running PostgreSQL instance and ICE_CONFIG). Additionally, the schema could be created for each suite of tests (e.g. chgrp, permissions etc.) thus guaranteeing the same conditions in which is suite is run. Finally, it would be a work-around for the lack of user delete functionality. Sadly, the SQL schema that is generated by bin/omero db script makes use of PostgreSQL native data types. Even setting the compatibility modes in both DBMS resulted in errors during the parsing of the schema. If we ever end up with a schema less locked to Postgres, then H2 would be the strongest contender to use as an in-memory DB.
comment:8 Changed 11 years ago by jamoore
- Cc java@… added
- Milestone changed from Testing and Docs to Testing2
- Sprint Testing and Docs (2) deleted
As I understood it, we've largely decided to put this on the backburner for the moment while we refactor our tests to see what's necessary in the way of a framework. Moving to milestone:Testing2. If that's incorrect, Blazej, please feel free to move it back as you see fit.
comment:9 Changed 11 years ago by bpindelski
In general, I have a feeling that it will be safe to start moving towards TestNG throughout the openmicroscopy codebase (excluding BioFormats). There is an easy way of automatic conversion from JUnit to TestNG in Eclipse. It also will allow to make the style look more consistent and remove the discrepancy between Java imports (annotations from TestNG and assertions from JUnit). This is one of the maintenance tasks after OmeroJava gets moved to a new component.
comment:10 Changed 8 years ago by sbesson
- Resolution set to fixed
- Status changed from accepted to closed
From https://www.openmicroscopy.org/site/support/omero5.2/developers/testing.html, TestNG is definitely marked as the default framework for our Java tests. We'll keep reviewing existing tests as we review our code.
An initial evaluation of Fitnesse has produced the following results:
On the other hand, there are several major drawbacks:
In general, Fitnesse is an interesting tool with a goal of enabling and increasing communication between business teams and developers. The way a test fixture is written and presented is helpful for non-developers. pwalczysko might be interested in trying to move one scenario and substituting each "Check" sentence with a fitnesse assertion, but that only applies to simple checks. From a developer's perspective, this tool wouldn't bring any value unless we start writing integration tests from scratch and using the fitnesse taxonomy for class and method names. Fitnesse has plug-ins for Python and Spring, but the 2009 date on them isn't encouraging.