How do Modes of Data Collection Affect the Measurement of Demographic Key Indicators? Findings from a Three-Country Experiment Using the Generations and Gender Survey

Abstract

Rising survey costs, increasing internet usage, and restrictions during the pandemic have accelerated a shift from face-to-face (Computer Assisted Personal Interviews, or CAPI) to web-based interviews (Computer Assisted Web Interviews, or CAWI). This paper analyses the consequences of this shift for the comparability of survey-based measurements of demographic key indicators. We use data from a unique experiment on the modes of data collection such as CAPI or CAWI (in the following: “modes”), conducted in the Generations and Gender Survey in Germany, Croatia, and Portugal. We calculate mode-specific means and ratios of selected indicators and perform regression analyses to estimate mode effects. In all countries, we find a few distinct mode-selection effects: Less educated people are more underrepresented in CAWI, and urban regions are underrepresented in CAPI mode. Mode-measurement effects vary widely across indicators and country contexts. For sensitive questions, there are less social desirability biases in CAWI. For questions with a high respondent burden, there are less survey satisficing biases in CAPI. Loop questions with a high respondent burden are of particular concern. The mode shift challenges comparability between data collections, in longitudinal analyses and in cross-country comparisons. Data users should use weights for descriptive analyses, control for the mode where possible, and consider mode differences when interpreting results. The unique data source allows insights into mode effects among demographic indicators that were not possible before.