Paradata

Collecting Paradata

Since paradata are defined simply as data describing the primary data collection process, paradata can be collected in every survey mode. However, the amount, type, and level of detail of the captured paradata will vary depending on whether the data have to be manually recorded or whether they are automatically logged by computer software. A crude distinction can also be made between paradata describing the data collection process as a whole (calls, follow-up procedures, etc.) and more specifie paradata referring to how a survey questionnaire was filled in.

Case management software (such as that used in centralized computer-assisted telephone interviewing [CATI] facilities) can record a wide array of paradata about the survey process as a whole. The software is capable of logging the time, duration, and outcome of each call to a sample unit, although some of this information may need to be supplied by the interviewer (e.g. call outcome). In noncomputerized settings, paper call sheets filled in by the interviewers can serve to collect paradata.

The most efficient way of collecting paradata on how a survey questionnaire is filled in is to use a computerized survey questionnaire with software that logs meaningful actions such as ticking response options, navigating through the questionnaire, and so on. This could be viewed as an automatic behavior-coding system.

Whether and which paradata can be collected depends on the software used to create the survey questionnaire. Many CAI software packages allow the recording of paradata. For Web surveys, JavaScript code has been developed to collect detailed paradata similar to keystroke data generated by CAI software. (Apart from these data, Web server logs can also be used to collect less-detailed paradata.)

Uses Of Paradata

Paradata can assist survey questionnaire pretests. For instance, data on how long it took to answer survey questions could be of importance in this phase. Long response latencies could indicate problems with particular questions. Paradata from keystroke files can reveal where errors were made, which may indicate poor interface design.

Paradata can also be collected during the actual field work. Recently, researchers have used paradata to adapt the survey design while the field work is still ongoing in order to improve survey cost efficiency and to achieve more precise, less biased estimates (these are so-called responsive design surveys).

In interviewer-administered surveys, paradata can be used to evaluate interviewer behavior. Time data can help identify interviewers who administered all or parts of the questionnaire too quickly. As in pretests, keystroke data can reveal where errors are being made. If these analyses are conducted during the field work, corrective measures can still be implemented in this phase.

When conducting experiments in a survey (e.g. within the questionnaire), researchers can use paradata as an additional source of information about the effects of the experimental treatment. Apart from a test of the effect of the treatment (e.g. survey instructions) on the dependent variable (e.g. omission errors in a self-administered questionnaire), paradata allow the researcher to see the effects of the treatment on the response behavior itself. This may provide additional insight into the reason why a specifie treatment is effective or not.

...