Using Paradata for Instrument Evaluation and Refinement (Presentation)

Paradata are a potentially useful tool for improving questionnaire design by helping understand the questions that are more or less effective, the significance of question placement, and the length of time spent on questions by respondents. For a national study of the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), we used paradata to evaluate instrument burden and the order of topic presentation, as well as identify potentially problematic questions. Our innovative use of paradata informed our recommendations for refining the instrument for future administration.       

We administered a two-part web survey to 90 state-level and approximately 1,800 local WIC agencies (the response rate was 91% to both parts of the survey). In this paper, we will describe how we used paradata to analyze respondent burden, break-offs, and the order in which respondents completed survey modules. Specifically, we will discuss: 1) using question- and module-level burden estimates to identify survey topics to consider revising in future rounds of administration; 2) considering the frequency of break-offs as a marker of problematic questions; 3) examining the number of login attempts as an indicator of how agency respondents shared responsibility for participation; and 4) analyzing the order of module completion to inform their order of presentation to respondents. Using paradata for questionnaire evaluation is a promising approach with the potential to reduce burden and cost and increase data quality in future administrations.