Assessing Mode Effects in SC Approaches: Comparing F2F and Push-to-Web

• In Event: Methodological Innovations in the Study of Public Opinion

Abstract

The golden standard of survey research from a methodological point of view has been until recently/today computer-assisted in-person interviews. This data collection mode has become ever more costly and response rates are continuously decreasing. There is always some dispute about the appropriate mode and the pros and cons of choosing a specific mode and the possible consequences in terms of, e.g., representativeness, non-response bias, response rates, unit- & item-nonresponse, etc. -- the so-called mode-effects. The chosen survey mode can affect responses in multiple ways and in different directions. In an interviewer-driven setting not only can survey responses be influenced by the mere presence of an interviewer but also by characteristics of the specific interviewer because of dynamic interactions between respondent and interviewer. F2f interviews may lead to less item non-response because of the presence of an interviewer and the fact that interviewers may probe “don’t know”-responses. An interviewer can also motivate a respondent to increase their mental effort in answering survey questions. In a self-completion environment respondents may answer questions using less energy and/or effort (also called `satisficing’) because they are not only concentrating on the survey but also doing other things and an encouraging interviewer is missing. Self-completion surveys may also lead to more nondifferentiation, i.e., respondents don’t use the range of options offered but stick to specific options, e.g., the middle one. Social interactions also entail a phenomenon called `social desirability’ which potentially biases survey responses. The respondent does not answer `honestly’ but anticipates what the interviewer may expect and what is socially desirable, thus the answer is biased. The larger the social distance is between two or more individuals the less the consequences of `social desirability’ may be, thus the social distance in a self-completion mode is larger compared to a f2f-situation, empirical testing of this `candour-hypothesis’ yields mixed results. Generally, a bias stemming from social desirability induced behaviour only occurs during questions touching sensitive issues, e.g. attitudes towards racism or sexual behaviour.
While continuously decreasing response rates and steadily increasing costs in face-to-face (f2f)-survey research may lead to a mode change in the medium term, the COVID19-pandemic forced some survey endeavors to change their mode immediately because of COVID-19 restrictions and social distancing rules. The COVID-19 pandemic made problems connected with f2f-interviewing not only more visible and more severe but also motivated some research projects to experiment with different modes and generally with strategies of conducting surveys that avoid f2f interviews.
I will present Austrian P2W data from an experiment conducted in 2021, compare it to external data sources and analyse possible mode effects using ESS round 9 data also from Austria. I follow different strategies to assess possible mode effects, comparison to official statistics, comparisons of means and standard deviations between the ptw and ESS9 data, comparing empirical cumulative distributions between those two datasets and assessing satisficing. This paper compares (1) the push-to-web data with validated official statistics, (2) ptw and ESS 9 data using means and standard deviations, (3) ptw and ESS 9 data assessing the respective empirical cumulative distributions for each response option, and, (4) finally, assessing satisficing comparing the two modes in the ptw data and the ptw data to ESS 9 data using a so-called differentiation index. The empirical investigation so far yielded promising results that a switch to a self-completion push-to-web mode is a viable strategy to cope with most of the problems the “classical” F2F-approach is plagued with. The sample composition of the PtW data is pretty in line with the ESS 9 data except the overrepresentation of older cohorts which may vanish as the “new” technologies become a common tool for all age groups. Age is still a significant driver of selection effects, i.e. as a predictor for either using a web-based questionnaire or a paper questionnaire, in a self-completion mode.

Author

• Peter Grand, Institute for Advanced Studies