Copyright (c) 1997 by the University of Washington. All rights reserved. This is a proprietary and confidential document. Under no circumstances may this document be used, modified, copied, distributed, or sold without the express written permission of the copyright holder.

Etch Automated Testing Procedures

This document presents a description the simple automated testing facility that we use to verify the operation of new versions of Etch during the development process.

The testing facility is driven by a script, the goal of which is (1) to log the behavior of the current version of Etch, when applied to a standard set of applications, and (2) to detect behavioral differences between the current version and the previous versions of Etch. We run the test script every night in order to verify the changes checked in during the day. The automated testing instruments a suite of application programs, runs the instrumented programs, archives the results, and reports the differences in outputs with those from the previous Etch versions. The following sections describe each of these steps in more detail. The automated test script runs on a PC with Windows NT 4.0.

Running scripts

The test scripts are written in Perl. We use wrapper .bat scripts to invoke the equivalent Perl scripts. For example, auto_test.bat calls "perl auto_test.pl". In this document, we will refer to the script as auto_test. Some per application driving test script use MSTest (e.g. winword). For these tests, MS Test 4.0 needs to be installed and mt.exe needs to be added to the path environment.

Also, MS Test's include directory needs to be appended to your include environment variable. The main Etch test script is called auto_test. The next section will describe how to run the automated testing scripts. Following that section is a more detailed description of the implementation of the scripts.

Example of running the automated test scripts

This section illustrates how to run the automated testing scripts. It is assumed that the individual per application driving scripts already exist. We provide some example driving scripts under etch/etchtest/{forloop,ie,mpegplay,netscape,...}.

Setup the environment

First set the environment to run Etch (via etch/bin/run-setup.bat). Next set the test environment by running test-setup.bat in the test etch/etchtest/bin directory. To use the driving test scripts (e.g. etch/etchtest/winword/test.pl) manually from within Visual Etch, you still need to run test-setup.bat. For example,

Describe the test suite

Copy and edit examples/example.test.vetch. This file describes how the test will be run. Lines beginning with # are ignored. Each line in this file describes an individual test. The main parameters per test are

The input script my.test can live anywhere and is used as input to auto_test.

Initialize the vetch project with dynamic dlls

To initialize the vetch project with the dynamic dlls, auto_test will run the test applications with the dllwatch tool. In this initialize mode, auto_test ignores the specified tool. This initialization need only be done once.

Run the tests

Results

The main results are reported in two files: etch.logs and mail.logs with the current date appended. Mail.logs is a filtered version of etch.logs. Etch.logs contains the full details of instrumentation.

Script Implementations

The main script, auto_test, calls on another script, called vetch_programs, to manage the actual instrumentation and testing of applications. Following testing, auto_test gathers and reports the results of the test run.

vetch_programs

The script vetch_programs is a Perl script that runs Etch on a suite of programs. The input to vetch_programs is a test description file containing a list of tests to perform. Each line in the test file specifies: (1) a target directory, (2) a Visual Etch project, (3) an Etch tool, (4) an optional runtime script, and (5) an application to test. vetch_programs reads the test script, performs the specified experiments for each application in the list, and writes the results to the specified target directory. vetch_programs produces two result files: vetch.logs and vetch.summary.logs. Each log file contains the aggregate of the logs for all the applications tested. The file vetch.logs is a detailed log containing the full output of Etch instrumentation messages; vetch.summary.logs contains instrumentation failures and warnings messages (if any). These two files are processed by auto_test.

vetch_program

For each application to be tested, vetch_programs calls on a helper script, vetch_program, to perform the actual experiment. The script vetch_program instruments the application with the specified tool and optionally tests the instrumented application. To do this, it issues commands to the Visual Etch command line interface. Each run of vetch_program produces at least three logs: vetch_program.log, vetch_program.error.log, and vetch_program.warning.log. If a test script is specified, the test script is responsible for producing the fourth log file, vetch.run.log, that contains the result of running the test on the instrumented application.

The first log file, vetch.log, is a detailed log of instrumentation messages and contains the Visual Etch log etch.log. The other two log files, vetch.error.log and vetch.warning.log, contain instrumentation error and warning messages, respectively.

Testing the instrumented application

If a driving test script is specified to vetch_program, that script will be run to test the instrumented application. The script performs the test (possibly supplying data to the application) and writes its results to a file called vetch.run.log.

For some programs, we test the instrumented program to see if the behavior and output is what we expect. The script that tests each application is application specific, but in all cases, this application test script is a Perl script invoked by vetch_program. The test script may invoke other scripts to accomplish its testing (e.g. a Visual Test script). The test script is responsible for starting the instrumented application, checking it's results, and reporting whether the test succeeded or failed. For batch programs, the test script can simply run the instrumented application and check the output. For interactive applications, the test script invokes a Microsoft Visual Test script to start and drive the program.

Gathering and archiving the results

After instrumenting and testing each application, auto_test processes the two output logs, vetch.logs and vetch.summary.logs, from vetch_programs. The script auto_test renames both logs and appends the current date to the name. To detect changes in Etch, auto_test compares the current detailed log (vetch.logs) against the previous day's version. To do this, auto_test runs diff on both logs and puts the results in a difference log. In our environment, auto_test emails the summary log and difference log to the development group when it finishes the tests, so that they can examine the results.

Example of testing an application

This section describes how we automated the testing of Etch on Microsoft Word with the ophisto (opcode histogram) tool using the above scripts. The input to auto_test (which is also passed to vetch_programs) is a test description file. For this example, the file contains one test entry:

 c:\tests test.vpr ophisto c:\tests\winword\test1.pl c:\msoffice\winword\winword.exe

For this test entry, auto_test will (1) place the results in c:\tests, (2) use test.vpr as the Visual Etch project, (3) instrument with the ophisto tool, (4) use c:\tests\winword\test1.pl to drive the instrumented Word, and (5) instrument c:\msoffice\winword\winword.exe as our target application.

To test the instrumented Microsoft Word, the Perl script test1.pl does two levels of checks. The first is to see that Word behaved as expected. This is accomplished with a Microsoft Visual Test script. The second is to check that the output matches that of an uninstrumented version of Word. The test1.pl script will invoke a Visual Test script to check behavior and then will compare the resulting file.

The Visual Test script is responsible for detecting whether Word behaved as expected (e.g. the open file dialog window appeared after invoking the open file menu). The Visual Test script will start and drive the instrumented Word to open a file, make some edits, and save the document as a text file. After the Visual Test script runs, it reports the results of its run to test1.pl.

If no failures are detected after running the Visual Test script, test1.pl compares the output generated by the instrumented Word with that of an uninstrumented Word. The script test1.pl then writes the results of the test run to vetch.run.log. The file vetch.run.log is the file that vetch_program looks for the results of running the application. After completing testing, test1.pl cleans up state for the later runs (e.g. removing the generated text file).

Automatically updating Etch and archiving results

This section describes how we automatically update the testing version of Etch and archive test results. The details of carrying out these tasks are specific to our evironment and are presented as examples.

Our version of the testing script auto_test, called auto_NT, is augmented two ways. First it automatically updates and builds Etch. When executed, the script first obtains a copy of the latest version of Etch and builds the latest version of Etch. In our environment, we archive the Etch distribution on UNIX machines using the CVS code maintenance facility, while Etch development and testing is performed on PCs running Windows NT 4.0. The auto_NT script obtains the latest copy of Etch from the CVS repository by updating its UNIX code tree. To do this, the script executes, via NT's rsh command, a "cvs update" on the UNIX tree. After the UNIX tree is updated, auto_NT copies the entire Etch tree to the test PC (via NT's rcp command). The script now has a copy of the latest Etch on the test PC. The script then executes the "genall.bat" script to rebuild the entire Etch system, at which point it is ready to test Etch on a suite of applications.

After auto_NT completes, the mail log is mailed to the Etch group and the logs are archived on an UNIX machine. To mail the mail log, auto_NT copies the log file to UNIX (via NT's rcp command) to a UNIX machine and then mails the file from the UNIX machine (by rsh'ing a mail command on the UNIX machine).

To archive the logs with a web page, we use a separate script to copy the logs (also from other possible test machines) to a UNIX machine and run a script to generate a web page indexing all the logs.