Three approaches to case study methods in education: Yin, Merriam, and Stake by Bedrettin Yazan TQR, The Qualitative Report Vol 20 #2 (2015) Three well-known books on case studies were written by Yin, Merriam, and Stake. These books take different perspectives on the purpose of a case study and how to perform it. This paper compares and contrasts the approaches. It does not take a stand regarding which is best nor even how to evaluate the different approaches. It describes differences in the books' guidance, but it doesn't give concrete examples (in the context of a specific case study) of actions that a researcher would take differently if accepting one type of guidance over the other. The book assumes some familiarity with case studies, so it may not be the best introduction to what a case study is. However, it does give a relatively concise overview of different perspectives on case study research. The English is a bit stilted, but always comprehensible. There is a lot of discussion of the epistemological position of the authors. * Yin is rationalist: All knowledge can be deduced from observations of the world, without the need for introspection. * Stake is constructivist and existentialist (non-determinist). * Merriam is constructivist: All knowledge is inherently subjective. There is no one truth; each person has a different truth. Merriam says, "reality is constructed by individuals interacting with their social worlds". She adds, "One of the assumptions underlying qualitative research is that reality is holistic, multidimensional, and ever-changing; it is not a single, fixed, objective phenomenon waiting to be discovered, observed, and measured as in quantitative research." * The author is careful to state his own epistemological perspective (constructivist). Is this extraneous or important? Should everyone do so in their research papers? Yin: * You should plan the case study ahead of time. If you don't know exactly what effects you are trying to observe, and if you don't have a plan to observe them, then you are less likely to obtain a rigorous and meaningful result. The researcher should decide propositions, and have a detailed study design, before starting the study. * Design quality is important. Four conditions are related to design quality: * construct validity (relevant measurements) * internal validity (alternative explanations) * external validity (generalize beyond subjects) * reliability (reproduce) "According to Yin, Case study researchers need to guarantee construct validity (through the triangulation of multiple sources of evidence, chains of evidence, and member checking), internal validity (through the use of established analytic techniques such as pattern matching), external validity (through analytic generalization), and reliability (through case study protocols and databases)." * collect both qualitative and quantitative data * data validation is key, and Yin comes back to it repeatedly * need a chain of evidence/reasoning from data to conclusions Stake: * the researcher is permitted to make major changes even after proceeding from design to research * do data collection and analysis simultaneously * does not require as much design preparation * problem: without a roadmap, the researcher may get "lost or stuck" * should collect only qualitative data, not quantitative data * I disagree: whereas quantitative data should not be statistically analyzed (because you didn't have a concrete plan for collecting it and because you probably don't have enough of it for statistical analysis anyway), quantitative data nonetheless is valuable to augment qualitative data * Stake says, "A considerable proportion of all data is impressionistic, picked up informally as the researcher first becomes acquainted with the case" * author of the paper says Stake "gives precedence to intuition and impression rather than guidance of the protocol" * triangulation: Four strategies for triangulation: data source triangulation, investigator triangulation, theory triangulation, and methodological triangulation. * The researcher's goal is persuasion, to convince the reader that the conclusion "makes sense". Merriam: similar to Stake but not quite as extreme * "The qualitative study provides the reader with a depiction in enough detail to show that the author's conclusion 'makes sense'". What is a case study? * Yin: "boundaries between a phenomenon and context are not clear and the researcher has little control over the phenomenon and context." When would you do a case study? How should you do a case study? * I would argue that early, flexible studies such as those championed by Stake should be done for the benefit of the researchers but not reported as a research result. Concrete examples: can we give some? What is "evidence-based software engineering"? The term is by analogy with "evidence-based medicine", where it means using data from experiments and observations to make treatment decisions, rather than using intuition and seat-of-the-pants guesses. In medicine, this was a step forward. However, all science is by definition evidence-based, so talking about "evidence-based X" where X is a branch of science is not meaningful. =========================================================================== Believing is Seeing: Confirmation Bias Studies in Software Engineering http://ieeexplore.ieee.org/document/7302437/ There is an 8-page submitted version and a shorter 4-page accepted version. This is a well-written and readable paper. The related work section surveys 12 papers that discuss confirmation bias in software engineering, mostly showing that it exists. They did two studies to confirm that confirmation bias exists in software engineering contexts. First study: The authors asked 26 IT professionals whether they preferred fixed-price or hourly contracts, in terms of whether the project will meet its goals (benefit to client, in a cost-efficient way). [Clients generally preferred fixed-price and providers generally preferred hourly.] Then, they showed the professionals neutral, randomly created, project data about contract type, client benefit, and cost-efficiency. They asked the professionals "Does the project data, in your opinion, give more support for fixed price or for hourly paid contracts?" There was a statistically significant bias: everyone who preferred fixed-price contracts found the data to support fixed price or neither, but among those who preferred hourly contracts, the plurality found the data to support hourly contracts. Overall, half of the respondents (a plurality) found the data to support neither; these individuals perhaps weren't biased in their analysis of the data. Second study: A meta-analysis of 35 published, peer-reviewed papers on cost estimation models. Two general cost-estimation methodologies are analogy-based (a project will probably cost as much as an analogous project did) and regression-based (a project will probably cost as much as a similarly-sized project). A previous systematic literature review of these 35 papers reported "that analogy-based methods had better mean estimation accuracy than regression-based methods in 66% of the published comparisons (23 out of 35) and that analogy- based models, in general, seem to outperform regression-based models." However, the authors of this paper note "In as many as 20 of the 35 studies the researchers included their own analogy-based estimation model in the accuracy comparison with other models. The remaining 15 comparisons did not include any self-developed estimation model. A tendency to find evidence confirming the presumably desired outcome that one's own model performs better than the rest may consequently lead to a bias towards better outcome for the analogy-based models." The author's analysis of the 35 studies shows that analogy-based methods are about 15% more effective. However, when they remove the 20 studies that were analyzing their authors' own analogy-based method, the remaining 15 studies show that regression-based models are 12% more effective! The authors give some reasons that confirmation bias may occur, not all of which are due to malice on the part of the experimenters: * Biased selection of project datasets. * Biased factor selection. * Biased removal of outliers. * Biased effort on model tuning. * Biased data processing and analysis. * Publication bias. Discussion: * What are other sources of bias besides those listed by the author? * How can one limit biases?