CSE 403, Spring 2011
Assignment #6a: Evaluation of Beta Release

Due: Tuesday, May 17, by 11:59pm
Turnin:

A beta release is an opportunity to learn about the strengths and weaknesses of your system.

Recall that each team is acting as the customer for another team. (You should be learning from that team's successes and failures!) As the customer, you have been asked to provide the vendor with feedback on an early release of the product, by testing it. The deliverable is an email message, described below.

The testing serves three purposes:

1. It gives you the experience of being a tester of code that you do not fully understand and cannot change. This can be both challenging and rewarding.
2. It gives you the experience of reporting bugs in enough detail that a developer can track down the problem.
3. It gives concrete feedback to your vendor, at a time when it can still have a positive effect on the final version of their project.

You will do testing from both the user's point of view, and the developer's point of view. We strongly suggest that you assign some members of your group to evaluate the user experience, and different members of your group to evaluate the developer experience.

User testing

Please do the following user testing:

1. Follow the user instructions for installing the software.
2. Follow the user instructions for using the software via its UI. The functionality might be limited so far; that is OK so long as it is documented. Perform manual testing; the vendor is responsible for automated testing and for turning your bug reports into regression tests. Think like a good tester, and try to find situations where the software performs incorrectly.

If the software is well-packaged, then #1 should take very little time. If you have problems with #1 that prevent you from proceeding to #2 (or problems in #3), then contact the vendor to fix the problem. In other words, it is important to start early on this assignment.

Developer testing

Please do the following developer testing:

1. Follow the developer instructions for downloading the software, building it, and making a release. Verify that you can install your newly-made release and execute at least one use case.
2. Evaluate the design documentation. For example, does it enable you to understand the architecture? Do you have confidence that you have enough information to get started fixing a bug or to take over development of the project?

As above, if the software is well-packaged, then #1 should take very little time.

Reporting problems

Report any problems that you encounter, following the project's bug-reporting instructions. Be sure to provide enough information to enable the developers to reproduce the problem. You do not need to speculate as to the root cause. An incomplete list of reportable bugs includes the following:

• Incorrect or missing functionality
• Incomplete, incorrect, or confusing documentation
• Poor user experience (UI problems)
• Performance problem
• Feature request

Focusing your work

As with any testing, you should focus on the most important issues. (You may report small problems, too, but not at the expense of the big ones.)

Do not test parts of the system that are documented as not yet working. (However, by the beta release, significant functionality should be present, and it should be possible to use the software in a useful, even if limited, way. If that isn't the case, then the beta release was a failure and you should so indicate in your assessment.) Do not report duplicates of bugs that already appear in the bug tracking system.

There is really no end to how much time you can spend on testing. Please allow yourselves 6-8 person hours collectively to conduct your testing and report your findings. You must spread the testing across at least three members of your team in an organized fashion.

Deliverable

After you have finished, send a brief email (copied to your vendor, the CSE 403 staff, and to your group) with subject line “Beta evaluation for project“ (with project properly filled in, duh) that includes:

• A two-paragraph assessment, indicating both strengths (what worked) and also the most important things that did not work or need to be improved.
• A list of Bug IDs and/or URLs for all the bugs you reported. The staff will use this information during grading.

You will be graded on the quality and thoroughness of the feedback you provide. As with all CSE403 assignments, this is a team assignment, with all members responsible for work. No matter how many people work on it, all team members will be graded on the result.

A requirement for all teams: When your team resolves a bug, you must notify the person who initially reported it. (This happens automatically with most bug tracking systems.)