Study proves: Telephone surveys are the »Gold Standard of Public Opinion Polls«

A study conducted by the Konrad Adenauer Foundation (Berlin, January 2024) has found that telephone and face-to-face interviews remain the most effective methods for conducting representative surveys.

The study, titled »Online, Offline or Both? Survey Methods Tested in Practice«, investigates the response behaviors of participants using five surveys that featured almost identical questionnaires but employed different survey methods:

• One purely telephone sample
• Two purely online samples
• Three mixed-mode samples (a combination of telephone and online samples)

Each survey was conducted among 1,000 individuals in Germany on the topic of the animal welfare competence of major German political parties. The surveys took place from June to July 2023, with the study being published in January 2024.

The results revealed significant deviations in response behavior among the different survey methods. »Given that all the surveys were conducted around the same time, the deviations should have fallen within the expected margin of error if the results were truly representative of the population. However, this was not the case. In some instances, the results varied by up to 43 percentage points from the telephone survey, with other deviations reaching up to 20 points.«²

The key factor: Sampling Methodology

The study’s authors, Sabine Pokorny and Dominik Hirndorf, attribute the significant differences in response behavior to the varied sampling methods. While online studies often rely on self-recruitment, suitable telephone samples can include almost everyone. »Telephone surveys have the advantage of randomly reaching nearly every individual through landlines and mobile phones.«³

Self-recruitment in online studies can lead to blind spots, excluding individuals who do not actively use the internet. »According to the Federal Statistical Office (Destatis), 3.1 million people in Germany have never used the internet, representing about 5 percent of the population.«⁴

In contrast, Germany’s telecommunications infrastructure supports representative telephone surveys: 99% of German households have a telephone connection,⁵ and there are 169 million registered SIM cards.⁶ CATI surveys, based on random sampling, ensure that every person with a telephone connection in Germany has the opportunity to be included in surveys.

»Purely random selection is the only way to achieve a sample that is representative of the population. Currently, this can primarily be achieved through telephone or face-to-face surveys.«⁷

Telephone: The Gold Standard of Public Opinion Polls

Although it is only »[…] a matter of time before population-representative mixed mode and online surveys [are] also possible«, »according to Pokorny and Hirndorf […] telephone or face-to-face surveys remain the gold standard of public opinion polls.«⁸ The study concludes: »Online surveys based on non-random sampling cannot replace random surveys such as telephone surveys«.⁹

The publication of the study can be found on the website of the Konrad Adenauer Foundation: https://www.kas.de/de/analysen-und-argumente/detail/-/content/online-offline-oder-beides 

Sources