Replication

Replication is reanalysis of a study, building on a new data set that was constructed and statistically analyzed in the same way as the original work. Repeating the statistical analysis on the original data set is known as verification (or replication of the statistical analysis). Replicability should be maximized in both quantitative and qualitative works, as replication studies and verification of existing data sets may be extremely useful in evaluating the robustness of the original findings and in revealing new and interesting results.

Even if a given work will never actually be replicated, it still needs to be replicable, or else there is no possibility of refuting its findings; that is, it fails to hold the falsifiability criterion for scientific work. Because replicability is an underlying principle in science, most disciplines in social sciences hold some replication standards for publications, determining [Page 720]what information needs to be disclosed such that researchers may replicate the study without further guidance from the authors. In the case of survey research, the design and analysis of surveys call for many distinct decisions regarding the sampling, measurement, and methods used. As such, replication and verification of a survey is oftentimes extremely difficult to conduct, even by the original researchers. Thus, researchers should be sure to closely document their work process when gathering and analyzing their data.

To be able to replicate a given survey, researchers need to hold exact information on the sampling and the instruments used. First, decisions regarding the sample design and management need to be recorded, such as what list was used to sample from and how the sampling was conducted; who the interviewers (if any) were and how they were instructed and trained; what strata, quotas, or weights were used, if any; how many times people were recontacted; and how missing data was dealt with. Second, the exact instruments used should be recorded, including the question wording and order, split-half experimentation and any other randomizations (e.g. question order), counterbalancing or changes between questions, and whether the respondents were interviewed in their first language and what translations of the questionnaires were used, and so forth.

Another component of the work that needs to be rep-licable is the statistical analysis of the survey. In that vein, the researcher should document the construction of the variables, such as the coding of any open-ended questions (e.g. the list of categories, how many coders were employed, how they were instructed and trained, and what the intercoder reliability was), the construction of all variables used (e.g. the exact scales, techniques to deal with missing data, any rounding, mathematical transformations), as well as the exact software (statistical package and version) and statistical methods.

Even if replicable, studies in the social sciences usually cannot be entirely replicated when the measured phenomenon has changed between the original study and its replication attempt. This means that even if a researcher is able to retrieve the information from the real world and process and analyze it in the exact same way as did the original study, the results still may be different because the population under study had changed. Nevertheless, researchers should aim to maximize the replicability of their survey and analysis, and try to make certain that a full verification will be possible.

...