Who is this CATI, and why is she calling me at work? – Interview with Johanna Krumbach

The dossier »Quality of samples« on marktforschung.de has been exploring challenges facing the industry, with the focus being mainly on samples from online panels. Which begs the question: Why is there so little mention of CATI in this context nowadays? Are there no sample quality issues with CATI?

Johanna Krumbach: It may seem that the quality of samples from CATI-based studies isn’t a hot topic because, for instance, we use an RDD dual frame for population surveys or a stratified random sample from a business register. Formally, these methods meet the criteria for a good sample: Every element in our gross sample has a non-zero selection probability. Anyone with a German telephone number, or any company listed with a number, has the chance to participate in the survey.

So, does CATI face fewer sample quality issues than online panels?

Johanna Krumbach: As a CATI provider, we don’t face the same challenges as online panel providers. For example, by closely monitoring our interviewers, we can almost guarantee that responses are not generated by AI algorithms. With well-trained interviewers, respondents are encouraged to fully engage in the answering process, avoiding patterned responses. Our interviewers are our first line of quality control and act as motivators to increase engagement.

What is the interviewer’s role in ensuring the quality of a survey?

Johanna Krumbach: Interviewers are tasked with maintaining the respondent’s attention, assessing the plausibility and seriousness of responses, and creating a conducive interview environment. They must identify and mitigate disruptive factors, like excessive noise or interruptions, and suggest rescheduling if needed.

Though interviewers are crucial for maintaining data quality, they are part of a broader strategy that includes rigorous training and strict quality controls that complement data verification processes. Overall, CATI allows us to better manage the factors affecting sample quality than online surveys, thanks to decades of methodological research and refinement in survey research.

It seems that the CATI method is a reliable tool for generating quality samples. Is that true?

Johanna Krumbach: Absolutely, when used correctly. However, producing a high-quality CATI sample requires significant time and resources, which is a disadvantage compared to online panels. This is why CATI samples are often combined with online panel samples in mixed designs, like the blended calibration used by infas, to leverage the strengths of both approaches.

How do you use CATI in the right way to ensure the quality of the sample?

Johanna Krumbach: The challenge in CATI surveys lies in addressing the causes of non-response errors. While a well-designed gross sample can largely quantify coverage and sampling errors, the quality of the final sample hinges on fieldwork planning and execution. The goal is to maximize the accessibility and cooperation of potential participants.

What’s the challenge in reaching all targeted individuals? Don’t you have their phone numbers?

Johanna Krumbach: Failures often stem from the inability to reach contacts initially, perhaps due to outdated numbers or unanswered calls after multiple attempts. When contact is made, refusal to participate is the main obstacle. Public skepticism towards unknown callers is high, exacerbated by frequent telephone scam reports in the media.

Therefore, our interviewers must differentiate our legitimate research from such scams, possibly by directing respondents to easily accessible and transparent information sources online. Unfortunately, many people are already lost before this can happen – other target groups, especially older ones, do not have a way to access this information.

Are there any other factors that can bias a telephone sample?

Johanna Krumbach: Skepticism towards calls can lead to deliberate non-answers, with people often researching the calling number online to gauge its legitimacy. This skepticism can skew the sample if not properly addressed and taken into account.

In the B2B sector, what difficulties exist in reaching the desired audience?

Johanna Krumbach: In B2B surveys, the skepticism factor is less significant. A culture of pre-announced calls has emerged, where respondents prefer an email or SMS notification before the call happens. We can still leverage the readiness of business contacts to provide information, though it’s crucial to develop targeted strategies to activate them.

How can you maximize the activation of your target audience?

Johanna Krumbach: Activation can be maximized by not only recruiting participants by phone but also pre-announcing the survey through email or mail. This strategy improves accessibility and willingness to cooperate, facilitating the use of mixed-mode approaches, combining telephone and online surveys.

Does mixing methods reintroduce online research issues?

Johanna Krumbach: Incorporating different contact and survey methods, like adding email addresses to a CATI framework, doesn’t negatively impact the gross sample’s integrity. The purpose of a mixed-mode survey is to enhance accessibility for different groups by identifying the most effective survey method for each beforehand.

It’s vital to understand the communication preferences and reachability of each target group. For example, email invitations might be more successful for engaging individuals from smaller or newer companies in a CAWI survey, whereas direct phone recruitment is likely more effective for those in established firms. This approach is about strategic targeting to ensure the survey reaches and resonates with the intended audience.

What should be done if, during the field phase, it’s found that a group is not reliably reachable through the chosen channel?

Johanna Krumbach: If the initial approach isn’t yielding the expected results, we can switch to cross-channel strategies, such as contacting individuals who were initially invited via email, now by phone. This can serve as a reminder for the CAWI survey or we might directly switch to conducting a CATI survey. These methods aim to improve accessibility and encourage participation by offering alternative ways to engage.

To mitigate non-response bias, it’s crucial to communicate with participants in a manner that they find accessible and comfortable. Employing various recruitment and survey methods helps achieve this. When participants are in a self-administered mode, removing the interviewer as a control element, other quality assurance strategies must be implemented, particularly when participation incentives are involved.

Ultimately, is it the combination of methods that guarantees quality?

Johanna Krumbach: Mixed survey designs, including the utilization of mixed modes, undeniably contribute to enhancing sample quality! I eagerly anticipate the ongoing advancement and adoption of such methodologies to maximize the potential of every gross sample.