This case study draws on my quantitative study of college entrance exam preparation (also known as “test prep”) to illustrate how a researcher goes about setting up a quantitative project using existing datasets. I describe how I decided between two different datasets, a decision which required attention to how the research question was asked. I also discuss how I decided on an analytic strategy, and how to interpret results from logistic regression analysis. I offer a detailed description of the ways that expert background knowledge shape research questions and decisions about the research from the very beginning. In particular, I discuss how the lack of expert knowledge might hinder efforts at understanding the statistical phenomenon at hand or even result in misunderstanding it altogether. I use the case study to illustrate a more general problem (“what does no mean?”) in the interpretation of quantitative survey questions which embed time-sensitive processes. Finally, I recommend that researchers draw on subject-matter expertise (their own or others’) in designing and interpreting their studies.
The Need for Topical Expertise in Quantitative Methods: The Example of College Test Preparation