Task #11269 (closed)
Opened 11 years ago
Closed 11 years ago
Deploy and evaluate MozTrap
Reported by: | bpindelski | Owned by: | pwalczysko |
---|---|---|---|
Priority: | minor | Milestone: | Testing and Docs |
Component: | QA | Version: | n.a. |
Keywords: | n.a. | Cc: | ux@…, cxallan, mtbcarrol |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | n.a. |
Sprint: | Testing and Docs (2) |
Description
To allow for a streamlined experience during scenario testing, MozTrap? has been considered a valuable addition to the set of tools used by the team.
It has been deployed (internal access only) at https://moztrap.openmicroscopy.org. Setup documentation left in OME google docs. Petr has been assigned the role of Test Manager, so the next steps will be to convert a scenario and have a test run of the system with the team involved.
Attachments (12)
Change History (23)
comment:1 Changed 11 years ago by bpindelski
comment:2 Changed 11 years ago by bpindelski
- Owner changed from bpindelski to pwalczysko
comment:3 Changed 11 years ago by pwalczysko
@bpindelski: On Friday at 5 pm tried to login with my LDAP credentials. This went fine. Was able to create test suites, test cases, test runs.
But for the Environments, I have massive problems. Don't seem to be able to create a new Environment profile and add Environments to it.
More exactly, can create an Environment Profile (such as OS or Browser) and edit it. But cannot add a new Environment (such as Linux or Firefox).
When I edit the Environment Profile and go "Add Environment", I am allowed to add for example Firefox and anything I like into the line.
Then I click "Done Editing". Then I edit the Environment Profile again, but no Firefox is shown - looks like my previous editiing was ignored.
Is it a permissions issue ?
I cannot run tests now, because I cannot add the Environment to the Product Version. When I try to add these, it goes as above - apparently allowing me this, then not performing the action.
I cannot run the test without an environment.
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
comment:4 Changed 11 years ago by bpindelski
All the limitations have been discovered and worked around during sessions spent with Petr. A test of moztrap among the team is planned for 31/07/2013. Feedback will be collected and reacted upon.
comment:5 Changed 11 years ago by bpindelski
Smoke test results and feedback have been collected in a g.doc (MozTrap? testing). For reference, the development tracking page for moztrap is https://www.pivotaltracker.com/s/projects/280483.
comment:6 Changed 11 years ago by bpindelski
The discussion around the testing results has been captured in a g.doc (MozTrap? feedback). General next action points are:
- Take sphinx docs from moztrap and start modifying it for ourselves
- Work on test assignments to users
- Add a g.doc spreadsheet-style view of results
- Try out API calls with a "automated-test" user and moztrap-connect (https://github.com/camd/moztrap-connect/)
comment:7 Changed 11 years ago by bpindelski
- Sprint changed from Testing and Docs (1) to Testing and Docs (2)
Referencing ticket #11258 has changed sprint.
comment:8 Changed 11 years ago by pwalczysko
@jburel @jamoore @bpindelski @mtbcarrol
After testing of MozTrap? we are having a look at Tarantula.
Considering the points raised during the test of MozTrap? and asking : Are these points solved by Tarantula ? The answer is yes. See also screenshots.
Went to http://www.testiatarantula.com/features/demo/ - username user, pwd user. (Also this link is helpful to get started http://www.testiatarantula.com/features/getting-started-with-tarantula/)
Went through the workflow of creating an Annotate scenario (Test Case) for Insight and Web.
Points we were looking for (sorted according to importance):
- It is not possible to do comments on every step in MozTrap? as you are testing - this is sorted out by Tarantula - you can do comment at every step, also you can see where you are in the test, what was done, what not. Also, in Tarantula you can fail a step and still be allowed to test the scenario to the end (not possible in MozTrap?.
- Overview of the results is poor in MozTrap?, need a convenience of the Googledoc - we have this solved in Tarantula using "Case execution list" under "Report" tab - see the results as per Test Case or as per "Execution" (=Test Run in MozTrap?). Tarantula gives us a table with obvious, immediate info about who TESTED it, lists ALL the comments on the first page (no need to click through the links to get to see the comments). If you need to see single steps in the Test Case (which steps were were failed, which were passed exactly, you can see it under Test tab - when you open the correct execution, you will see a history of the test at a glance as well. Executions tab shows additionally the progress of the test cases, and again, who was assigned, who executed at first glance, and whether the test was run or not yet. Also you can see how long did it take for the tester to run the test.
- Assign tests to people - sorted by Tarantula, in every Execution every Test Case is compulsorily assigned to a tester - does not matter if the tester does not have the time to execute it and asks a colleague to help him out and execute the test for him - Tarantula will remember both persons, and hightlight the more important one (= who actually tested) in the results / overview tables
Furthermore, the pluses for Tarantula are also the ease of writing scenarios, I have unlimited possibility to copy steps from any scenario into any other scenario (no cloning like in MozTrap?, no restrictions). I like this.
Drawbacks of Tarantula:
- Screenshots - not really obvious at this time how this should work, markup impossible according to Blazej. Possibly we could go away without screenshots ? Up to the testers, I don't need them.
- (A Minor one) - not seeing now how to link a ticket from Trac to the test comment like it was possible with MozTrap? - on the other hand Tarantula directly supports connection to Bugzilla and jira - Jmarie is looking into the possibility to widen this to Trac -> this would give Tarantula immediate plus against MozTrap? as well.
- LDAP not possible - absolutely no problem for me - as I understand it, the fear here is that we will have to create all the users manually -> more than happy to do this myself - creation of even 30 users (pretty unrealistic number, actually we talk about ~10) is just nothing in comparison to creating and maintaining ~100 scenarios. I would probably need to be an admin to do this, but why not ?
In general, pretty happy with Tarantula. Seems that there are just pluses, except the Screenshots problem. Certainly worth a serious try.
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
comment:9 Changed 11 years ago by pwalczysko
- Cc ux@… cxallan mtbcarrol added
comment:10 Changed 11 years ago by jburel
Petr: you can add the screenshots to the case itself using the "add attachments" option.
comment:11 Changed 11 years ago by jamoore
- Resolution set to fixed
- Status changed from new to closed
Closing since we are likely to evaluate tarantula after Petr's return. The investigation here is all still quite value and if this needs to be re-opened, please do so.
Assigning to pwalczysko for general demo preparation - i.e. moving of a selected scenario into moztrap. Next step will be a demo session with the team.