First, the user invokes vetch, bringing up the Visual Etch graphical user interface. Before instrumenting the program, the user uses DLL Discovery (the DLLwatch tool) to identify the full set of DLLs imported by the application.
After using Visual Etch to select the application, tool, and any special parameter settings, the user then clicks "Execute Action". This passes the user-selected settings to the Module Management routines, which perform three major tasks: instrumenting the binary and each of the selected DLLs, creating the environment in which to run the etched program, and presenting the results.
Each module (the binary and each DLL) goes through several phases as it is rewritten. All of the phases rely on the Executable File Interfaces to provide an abstract interface to the Windows32 executable file format, and on the Disassembler to provide an abstract interface to the x86 instruction set. Code Discovery analyzes the control flow of the program and separates code from data. Then the main rewriting loop of the Etch rewriting engine iterates through the source program, making appropriate callbacks to the tool instrumentation code. Finally, after all the modules are rewritten, the import tables are patched to reflect the names of the etched modules by the Import Table Patching (dllwalk) tool.
As the etched program runs, Etch Runtime code is invoked as necessary to emulate the environment of the original program and to support to the tool runtime code. The tool runtime code may also make use of the interface to the Hardware Performance Counter Interface.
Visual Etch is the GUI front-end to Etch. It allows the user to select a program to transform, select the transformation ("tool") to apply, and customize set of DLLs to transform. It invokes Etch via the Module Management component.
Visual Etch provides several major services:
DLLwatch reports the set of DLLs loaded by a running application. It simply runs the target application as a debuggee process, and is notified by the operating system for each DLL loaded by the application. It logs the resulting DLL names to a file, which is post-processed by the Perl script dllwatch.pl.
This component manages the instrumentation of an application and its DLLS given a tool name, an application, a set of modules to be transformed, and any special options. It is invoked by Visual Etch, and invokes etch.exe (multiple times, if necessary).
Visual Etch calls coordinator.pl, which parses the project files that Visual Etch provides and calls tool specific scripts that are mentioned in the file list. The tool scripts in turn can invoke Perl library routines provided by this module to:
At the end of the run, coordinator.pl calls routines in log.pl to create a log file.
The user specifies options through Visual Etch. Visual Etch puts these options into text files which coordinator.pl parses into perl arrays. The routines in this module use these perl arrays to determine the appropriate flags to Etch.
These classes provide an interface to the Windows 32-bit executable file header. They specify both the position of sections in the executable file itself, as well as the position of the sections as mapped into the address space of Etch during instrumention. (As a result, all translations between file offsets and virtual addresses in the input executable file are mediated by the ExeFile class.) They allow manipulation of the file header, including methods to add sections; change the entry point; change the location of the relocation, import, and export tables; and write the header to a file.
These classes represent Windows 32-bit executable import tables. In addition to allowing the import tables to be read, these classes support "wrapping" (replacing selected functions with calls to similarly named functions in a wrapper DLL, a la etchwrap.dll); and adding the export table for a specified DLL to the import table (for example, adding the routines exported by a tool runtime DLL to the etched program's import table).
These classes represent a Windows 32-bit executable's export table, allowing the export table to be scanned for exported functions and functions "forwarded" to other DLLs.
These classes represent the relocation records in the executable file as a simple array of relative virtual addresses. Etch uses relocation records in order to identify pointers in the original program. For example, this is used in code discovery to identify possible procedure pointers. In addition, Etch must update the relocation records correctly and augment the relocation records to reflect any new pointers added in order for the new program to run correctly if it is relocated at load time.
This code provides an interface to the NB09 debugging information in a binary, if present. Used to associate names with procedures and as a source of additional information for code discovery.
Standalone x86 decoding library. The instruction information is used by code discovery, the main rewriting loop, and tool writer instrumentation code. The decoding of x86 instructions is driven by tables. The instruction opcodes index into these tables. The main parsing tables are defined in maps.c. An instruction eventually maps to an entry that gives the mnemonic, number of operands, and operand encodings. For example, an instruction that starts with 0x01 maps to the entry {ADD,2,"Ev","Gv",0}. The operand encodings have the same meanings as the addressing and operand type encodings of the opcode maps in the Intel manuals.
To determine the operand read/write type, dcinfo_map (maps.c) has an entry for each instruction mnemonic (DCOpcode_t in instruction.h) indicating whether the operand is read, written, or both.
We provide a partial disassembler, disasm.exe, to help test the decoder. Its input is the raw text section of a program. It will print to stdout how we decoded the instructions. The output format is very similar to "dumpbin /disasm".
What is missing to make this program truly standalone and disassemble executables/dlls is to provide a simple wrapper inside of disasm.exe that reads in an executable and points the print routine to the start and end of various text sections.
To add instructions to the decoder, three modifications have to be made. The first is to add the appropriate entries in the opcode maps in maps.c. If the mnemonic does not already exist for it in the DCOpcode_t enumeration (instruction.h), add it. If a mnemonic was added, an entry must be added (in the same order as DCOpcode) for the instruction in dcinfo_map (maps.c). The function check_dcinfo() checks to see that dcinfo_map is in the same order as DCOpcode_t.
The primary purpose of code discovery is to distinguish between code and data in the text sections of a binary. Code discovery is coordinated by the routine discover.cxx:DiscoverCode. Functionally, DiscoverCode initializes code discovery using traverse.cxx:TraverseInit, and then invokes code discovery on various entry points in the binary. These entry points include the entry point of the binary (TraverseAll), export entries (TraverseExportEntries), relocations in the text and data sections (TraverseCodePointersInText and TraverseCodePointersInData), and procedure entries from debugging info (TraverseDebugInfo). Once all entry points have been processed, the main Etch engine finalizes code discovery using TraverseCollapse, and the results of code discovery can now be used by the rest of the Etch engine.
If code discovery is set to its most aggressive mode (which it is by default), then, just before TraverseCollapse is invoked, DiscoverCode also invokes TraverseUnknownByteSequences. TraverseUnknownByteSequences examines the remaining unknown byte sequences in the binary's text section, and heuristically tries to determine whether those unknown byte sequences contain code or data.
Internally, the code discovery engine maintains a data structure that keeps track of the state of every byte in all text sections of the binary. The state of a byte can be kCode, kData, or kUnknown, corresponding to code discovery's estimate of whether the byte is code, data, or status still unknown. This data structure is an array of packed longs stored in gDataBitArray, with two bits used to represent a byte in a text section. Accessing and setting the state of a byte are handled by two procedures, ExtractBits and InstallBits. A set of macros, AddrIsCode, AddrIsData, AddrIsUnknown, AddrIsKnown, and MarkAddrIsCode, MarkAddrIsData, MarkAddrIsUnknown, is used as a set of convenience functions for manipulating the entries of the array.
Another purpose of the code discovery engine is to determine the boundaries of all basic blocks and procedures in a binary's text section, in particular when no debugging information is available. Code discovery accomplishes this in two steps. First, as the code discovery engine discovers instructions, it records information from particular instructions that helps it reconstruct basic block and procedures. For basic blocks, code discovery records the targets of all control flow instructions in an array named gControlTargets. For procedures, code discovery records the targets of all call instructions in an array named gCallTargets.
Once code discovery is finished, it then uses the recorded information to reconstruct basic block and procedure boundaries. The boundaries are reconstructed using the two procedures DetermineBasicBlockBoundaries and DetermineProcedureBoundaries, both of which are invoked from TraverseCollapse.
The main engine for Etch takes as input a binary program, tool-specific callbacks (as described in the Etch tool-writers API). It produces as output a transformed binary. The engine is divided into the following subcomponents:
After argument parsing and initial setup, the rewriting process is directed by the driver routine program.cxx:PatchProgram. PatchProgram creates InProgram and OutProgram objects, which contain all the components of the input and output binaries (file headers, relocations, imports, exports, and so on). PatchProgram also invokes code discovery, if necessary.
Finally, PatchProgram calls OutProgram::WriteFile, which writes the file header directly to the output stream, and then writes each section ("EtchSection" or a subclass) of the output file. When the EtchedTextSection is written out, it calls TransformAllInstructions to invoke the text section rewriting subcomponent. OutProgram::WriteFile then rewrites the file header to account the actual section sizes, and finishes by calling into the label management subcomponent to resolve any unresolved labels.
(The top-level also calls program.cxx:PatchRuntime to create a custom version of etchrt.dll that includes calls to code inserted by the tool's InstrumentProgram callbacks; and if necessary, calls program.cxx:CreateHusk if the -husk option was specified on the command line.)
All output data is written through a FileStream object. (There will be one FileStream per output file section.) Stream classes provide methods for writing data, as well as special methods for writing pointers, branch offsets, and various special cases for pointers (according to whether they should be updated reflect the movement of code in the etched binary, whether to generate a relocation record, and so on.) Pointers and branch offsets whose new value cannot be determined use the label management subcomponent to leave a placeholder behind to be fixed on the second pass.
The label management routines provide placeholders for a variety of values that cannot always be determined on a single pass: forward branch offset, forward pointers, and so on. As the output file is generated a set of unresolved branches grows, and then are resolved on a second pass.
The Buffer class simply mediates between RVAs and file offsets, and consists of an array of (file position, RVA base address) pairs, one per output section.
To eliminate label management overhead for "nearby" forward references, the BufferedStream object buffers the output stream in memory briefly before writing to the "real" Stream.
The PCMap is a collection of mappings from original address to etched address, one mapping per section. It is updated to reflect the new position of instructions in the output binary, and the new location of the import table in the output binary. It is consulted as needed to determine the correct value of pointers and branch targets.
The second purpose is to update the import table of a binary so that it imports etched or patched DLLs rather than the originals. The updating is done according to entries in a map file, which is typically generated by Visual Etch according to the configuration of the Modules dialog. Dllwalk updates a binary's import table by mapping the binary and iteratively checking the original imports against the entries in the map file. For those imports that match entries in the map file, the imports are changed to the new values specified by the map file entries. Since the new import names are typically larger than the original names, dllwalk relies upon Etch to provide sufficient storage in the import table for the new names when Etch transforms the binary.
The Etch Runtime provides two primary services. First, etchwrap.dll mimics the Windows execution environment of the original program (primarily, the directory and file names of the original executable and DLLs).
Second, etchrt.dll provides support to tool run-time code: in particular, support for code inserted during the ModuleBefore, ModuleAfter, ProgramBefore, and ProgramAfter callbacks, and support for the ArgBranchTarget argument type.
etchwrap.dll mimics the Windows execution environment by intercepting application calls to the Windows32 API calls LoadLibrary, GetModuleHandle, and GetModuleFileName, and modifying the arguments to refer to etched and patched DLLs rather than the original DLLs.
etchwrap.dll is linked into the etched module as part of the rewriting (etching or patching process), as specified by the '-wrap' arguments on the Etch command line. All statically identified calls to the relevant calls to LoadLibrary, etc. in kernel32.dll are replaced with calls to EtchLoadLibrary, etch. in etchwrap.dll. The code for replacing the calls lives in outimport.cxx:OutImportTable::ReplaceImports.
At runtime, etchwrap.dll reads a configuration file (etch.config) generated by the Module Management component that provides the directory of the original executable and the names of any DLLs that were neither etched or patched (and hence should not be replaced with etched or patched DLLs.) For each attempt to refer to an unmodified DLL X.dll, etchwrap.dll searches for X-patch.dll and X-etch.dll. If neither is found it alerts the user that a problem exists (using the utility program simpledialog.exe), and asks the user whether to proceed with the experiment. As a side-effect, it generates a file, etch-orphans.log, that lists all DLLs for which no etched or patched version was found. This logfile can then be incorporated into the Visual Etch modules window for the next experiment.
etchrt.dll provides runtime support to Etch tools. This includes runtime code inserted during the ModuleBefore, ModuleAfter, ProgramBefore, and ProgramAfter instrumention-time callbacks (etchrt*), and to support the ArgBranchTarget argument type for indirect calls (backmap*).
At instrumentation time, any code inserting during the ModuleBefore and ModuleAfter callbacks is inserted into the etched binary. For DLLs, the entry point of the DLL is replaced with a call to etchrt.c:EtchHandleDllEntry. (For executables, the entry point is simply replaced with code that calls the ModuleBefore code, if present, and then transfers control to the etched version of the original entry point.) In addition, etchrt.dll is rewritten to include stubs that call any code inserted during the ProgramBefore and ProgramAfter callbacks. This code is implemented in instrument.c:{InstallModuleCallbacks,InstallProgramCallbacks}.
At runtime, etchrt.c:EtchHandleDllEntry makes the appropriate calls to the ProgramBefore/ProgramAfter/ModuleBefore/ModuleAfter routines, and then transfers controls to the etched version of the original entry point.
Instructions whose branch target is determined dynamically (indirect branches and returns) need special support when tools use the ArgBranchTarget argument type to tool runtime code. At runtime these instructions will generate the branch target in the context of the etched program, while typically the toolwriter will be interested in the branch target in the context of the original program. To support this functionality, a map from etched pc to original pc is inserted into the etched module, if needed. At instrumentation time, calldll.cxx:PushBranchTarget inserts code into the etched binary to consult this table and pass it along to the tool runtime code.
Device drivers and user-level routines to control the hardware performance counters. These routines are provided to assist creators of etch tools.
Both the NT and the Win95 drivers use the same ioctl based interface, so the user-level code that interacts with the drivers is almost identical. This code is in the hwcounter dll, and the source is in hwcounter.c. On NT, the drivers are installed as an NT service, and we use a command line program to perform the installation. The source code for this lives in driverinst.c. The perfselect utility in this directory is a window based interface used by Visual Etch to allow the user to select which hardware counters should be active for measuring a given application. Finally, the perfutil program is a command line interface used to activate counters and measure programs using the counters. Perfutil is a client of the hwcounter dll.