CSE 403, Spring 2009
Assignment #6a: Evaluation of Beta Release

Due: Tuesday, May 18, by 5:00pm
Turnin:

A beta release is an opportunity to learn about the strengths and weaknesses of your system.

Recall that each team is acting as the customer for another team. (And you should be learning from that team's successes and failures!) As the customer, you have been asked to provide the vendor with some user feedback on an early release of the product. You will provide this feedback by running tests.

The testing serves three purposes:

1. It gives you the experience of being a tester of code that you do not fully understand and cannot change. This can be both challenging and rewarding.
2. It gives you the experience of reporting bugs in enough detail that a developer can track down the problem.
3. It gives concrete feedback to your vendor, at a time when it can still have a positive effect on the final version of their project.

Please perform the following testing for that team's software.

1. Follow the user instructions for installing the software.
2. Follow the user instructions for using the software via its UI. (The functionality might be limited so far; that is OK so long as it is documented.) Testing need only be conducted manually (the vendor is responsible for automated testing and turning your bug reports into regression tests).
3. Follow the developer instructions for downloading the software, building it, and making a release.
4. Evaluate the design documentation. For example, does it enable you to understand the architecture? Do you have confidence that you have enough information to get started fixing a bug or to take over development of the project?

(If the software is well-packaged, then #1 and #3 should take very little time. If you have problems with #1 that prevent you from proceeding to #2 (or problems in #3), then contact the vendor to fix the problem. In other words, it is important to start early on this assignment.)

Report any problems that you encounter, following the project's bug-reporting instructions. Be sure to provide enough information to enable the developers to reproduce the problem. You do not need to speculate as to the root cause. An incomplete list of reportable bugs includes the following:

• Incorrect or missing functionality
• Incomplete, incorrect, or confusing documentation
• Poor user experience (UI problems)
• Performance problem
• Feature request

As with any testing, you should focus on the most important issues. (You may report small problems, too, but not at the expense of the big ones.)

Do not test parts of the system that are documented as not yet working. (However, by the release of a beta, significant functionality should be present, and it should be possible to use the software in a useful, even if limited, way. If that isn't the case, then the beta release was a failure and you should so indicate in your assessment.) Do not report duplicates of ones that already appear in the bug tracking system.

There is really no end to how much time you can spend on testing. Please allow yourselves 6-8 person hours collectively to conduct your testing and report your findings. You must spread the testing across at least three members of your team in an organized fashion.

Finally, after you have finished, send a brief email with subject line “Beta evaluation for project“ (with project properly filled in, duh) that includes:

• A two-paragraph assessment, indicating both strengths (what worked) and also the most important things that did not work or need to be improved.
• A list of Bug IDs for all the bugs you reported. The staff will use this information during grading.

You will be graded on the quality and thoroughness of the feedback you provide. As with all CSE403 assignments, this is a team assignment, with all members responsible for work. No matter how many people work on it, all team members will be graded on the result.

A requirement for all teams: When your team resolves a bug, you must notify the person who initially reported it. (This happens automatically with most bug tracking systems.)