CSE 599E1: User studies in software engineering

Autumn 2016

Instructor: Michael Ernst
Class meetings: MW 9:30-10:50 in room 203; project presentations during final exam slot
Prerequisites: Graduate status (including CSEM) or instructor's permission
Office hours: by appointment, anytime I am available
Webpage
Calendar
CollectIt Dropbox for submitting assignments

This course covers research methods for understanding software development. Specific types of methods covered include controlled experiments, case studies, and surveys.

The course projects and examples are chosen from the domain of software engineering, for concreteness and to keep the class focused. However, the principles are more general and what you learn will be applicable to user studies in other domains. Furthermore, even if you already have some familiarity with user studies, you will learn from seeing them applied in this domain. Therefore, people with previous experience are encouraged to register.

Course objectives

This course will teach you research methods that enable you to decide what to measure and how to measure it.

By the end of this course, you will be able to:

• Identify research questions about software engineering tools and practices
• Choose an appropriate research method to evaluate a research question
• Design a user study that focuses on the relevant information
• Conduct the study ethically
• Analyze the data, qualitatively or quantitatively as appropriate
• Report your results

In every grad class I have taught, a few groups have followed up on their work and had it published at a research conference. That is not a requirement, but your work should be at the same level of rigor as for any scientific research project.

Topics

Topics include the following:

• Controlled experiments
• Experimental design (repeated measures, independent samples, matched pairs, factorial design)
• Case studies
• Observational studies
• Statistics
• Surveys/Interviews
• Grounded theory and hypothesis formulation
• Ethnographic studies
• Replication
• Ethics

The specific week-by-week schedule is forthcoming.

Structure of the class

Activities in this class include:

• learning about research methods, from textbooks and other sources
• reading and critiquing user studies performed by others, from research papers
• performing your own user study, in a group project

Class sessions will be discussion-based rather than lecture-based. You are expected to do the readings and participate in class. Passive listening is not an effective way to learn. (You would be right to ask for evidence of this claim. That evidence exists in the form of both case studies and experiments!) I don't mind if you have your computer open during class, but use it for notetaking or other class activities; plase don't attempt to multitask or do anything that would distract you from the class discussion.

You may be assigned to summarize some of the readings. Each student will lead at least one class session. The instructor is available to help you prepare.

You will choose the topic of your user study, with the instructor's approval. Suggestions will be provided. The topic is permitted to be related to your research. A replication of a published study is permitted.

You will work in a small group of usually two students. This is the major part of the course, since the best way to master a topic is by doing it (possibly making mistakes along the way and thoughtfully learning from them).

In order to let you focus on the project, there will be few or no weekly set assignments, though there will be milestones due regularly for your project.

Grading

Grades will be determined approximately as follows:
Class participation: 30%
In-class presentation: 20%
Project: 50%

I expect everyone to work hard and learn from the class, and I expect everyone who does so to receive a high grade.