Guidance

How to assess the use and risks of telephone surveys for monitoring, evaluation and research during COVID-19

Published 7 September 2021

© Crown copyright 2021

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/how-to-assess-the-use-and-risks-of-telephone-surveys-for-monitoring-evaluation-and-research-during-covid-19/how-to-assess-the-use-and-risks-of-telephone-surveys-for-monitoring-evaluation-and-research-during-covid-19

1. About phone surveys and this guidance

[footnote 1]

Remote surveying activities are characterised by interviews, not conducted in person ie. the enumerators (if any) do not sit face to face with the respondent. They can be of 3 broad types:

This guideline focuses on phone surveys as they allow us to retain more human contact with the respondents than the others, but most of the considerations we make for phone surveys are applicable to the other forms of remote surveying too [footnote 3].

In phone surveys or Computer-Assisted Telephone Interviews (CATI) the enumerator calls the respondent and administers the questionnaire on the phone, recording responses on a digital survey tool. Phone surveys have 2 main advantages:

  • because they cut down the logistic and transportation costs associated with enumerators physically reaching the respondent, they are cheaper than traditional field surveys
  • they enable data collection in contexts where it might be difficult for research, monitoring or evaluation teams to travel around and administer interviews (such as fragile and/ or conflict settings, or pandemic situations).

In the context of the COVID-19 pandemic, phone surveys are probably one of the best tools in our toolbox, and many organisations have switched from face-to-face to phone survey to ensure continuity of monitoring, evaluation and research activities. Exploiting the growing technological penetration, researchers have been using this approach for years to collect data even in the most challenging contexts; as a result, there is a wealth of resources available on how to conduct phone surveys and what best practice looks like.

However, the context generated by the COVID-19 pandemic on a global scale is unprecedented, and practical guidance to support decisions around whether and how phone surveys can be applied in the current COVID-19 context is missing. This guidance is an attempt to fill in this gap.

The intended audience for this guidance are commissioners, producers and users of monitoring, evaluation and research products, particularly in the international development sector. By providing a collection of considerations, the guidance aims to:

I) Stimulate thinking and discussion to make better informed decisions on whether and how to use phone surveys for data collection in a COVID-19 affected world

II) Encourage commissioners and producers to respectively demand and put in place mitigation strategies to address and correct predictable risks and biases related to phone surveys

III) Provide users of monitoring, evaluation and research products the critical tools to better interpret evidence gained through phone surveys

It is outside the scope of this guidance to instruct on how to manage or conduct a phone survey; the Innovations for Poverty Action (IPA) Remote Surveying in a Pandemic: Handbook, as well as other useful resources on how to practically implement a phone survey are listed in Section 5.

New experiences and considerations on using phone surveys in the context of COVID-19 are emerging on an ongoing basis. Therefore, the set of considerations outlined in this document should be interpreted as contributing to an evolving discussion and not as comprehensive and/or conclusive.

2. Is a phone survey suitable for you?

When considering if and how to conduct a phone survey there is not a right or wrong answer, it depends on the context you are operating in. The considerations are many, and often case-specific. However, here we try to summarise the key issues you should reflect on before commissioning or implementing a phone survey, and we recommend you read the DFID Ethical Guidance for Research, Evaluation and Monitoring Activities before you decide to start or continue any research, evaluation and monitoring activity.

Does it pose a risk for the respondent/s (or the enumerator/s)?

“Do no harm” always comes first. In a context where all the main organisations dealing with violence against women are raising the alarm for heightened risk of domestic violence due to COVID-19 (see UN Women report brief and WHO brief), think carefully of the risk you are exposing your enumerators and your respondents to and about what the principle of ‘do no harm’ means in your operating context. In lock-down, it is virtually impossible for respondents to get enough privacy to make the interview completely confidential and phone calls can leave a trail that perpetrators can find. See this UN Women brief about Violence Against Women and Girls Data Collection during COVID-19 for more considerations on this specific aspect [footnote 4].

Are you confident that mobile phones are available to the population you want to reach out to?

A phone survey will be - at best - representative of the population with a phone. According to the 2020 GSMA report, 30% of the world population still lacks access (at least at the individual level). In this 30% of the population, vulnerable and marginalised categories (including but not limited to poor households, women and people with disability) are over-represented. A phone survey might exclude these vulnerable strata of the population from reach and so you might be losing their voice. Think about access to technology comprehensively: it is not only about access to a phone, but also to electricity to recharge it (access to electricity is often measured by official nationally representative household surveys), a stable network and data bundles in case survey calls are happening through applications [footnote 5]. Whether mobile phones are available individually or on a shared basis is also an important consideration to make: asking the respondent to speak with any other member of the household is an approach often used in landline surveys in developed countries but rare in phone surveys in developing countries [footnote 6].

Including and identifying people with disability remains a challenge in the context of phone surveys, making their inclusion more difficult to achieve. The preferred method of disability self-assessment in large scale surveys is the Washington Group Questions [footnote 7]; however, consider that there is low overall evidence on how well these can be administered over the phone and, more generally, relying on self-reporting will likely lead to a significant under-estimation of the prevalence [footnote 8]. To ensure inclusion of persons with disabilities (as well as of minorities and harder to reach populations) you may need to consider purposeful sampling (e.g. through disabled persons’ organisations), although note that purposeful inclusion of people with disabilities is usually subject to higher scrutiny by ethical review boards (see Section 3 for more details).

Are your outcomes of interest correlated with variables determining access to mobile phones?

Often mobile phone availability is determined by characteristics like overall level of social and economic wellbeing of the households, urban or rural location etc. If the information you are interested in is affected by, or has a relationship with, these characteristics, your information is likely to be not representative of the whole population but rather only of a subsection of it. For example, evidence from studies in Nigeria and Ghana suggests that the sample for a phone survey tends to be biased towards males who are wealthier, younger, more literate and who live in urban sites compared to face-to-face survey’s sample.

Is information collectable through a phone survey?

Some data simply cannot be collected through a phone interview: think for example of biometric information, reduced or full form consumption or income aggregates and time-use data questions, or other questions that might require visuals, calculations by the respondent, etc. In general, anything that requires the respondent to think for more than a few seconds is likely not to work very well. Are you confident that the data you need can be collected by asking verbal questions on the phone? Also think in terms of “accessibility” of the survey: how would persons with certain types of impairments, including cognitive, comprehend the survey?

How sensitive is the information you want to collect?

Would the receiver of the call feel comfortable replying to your questions? Consider that you have no control over where the respondent might be or who they might be surrounded by during the call. Try to go beyond the obvious and go deeper into the cultural sensitivity of the information you need - power dynamics within the household might affect not only response rate but also how genuine and representative responses are. For example, in certain contexts, people might feel reluctant to even speak about “what crops did you cultivate on your plot in the last rainy season?” in front of other family members, so you might either receive no response or an inaccurate response influenced (directly or indirectly) by the presence of others.

How much information do you need?

Generally speaking, phone surveys are not particularly suitable for long questionnaires. How long is long depends on many contextual factors, but most practitioners consider, as a rule of thumb, an average duration of 10-15 minutes and a maximum one of 30-40 minutes (with the option to break the interview up over several short calls if needed). This is key for older persons, and persons with certain impairments, as they may lose concentration [footnote 9]. Do consider that this is not only a “data quality” issue, but also a safeguarding one: think carefully of the implications that keeping a family member unnecessarily long at the phone might have in terms of stress and the wellbeing of the respondent.

Is contact information available?

It might sound like stating the obvious, but a phone survey is possible only if phone numbers and contact information are available. This is potentially less of an issue if your phone survey is a follow-up or an end-line survey where respondents’ contact information has been collected relatively recently [footnote 10]. There is room to be creative and look for existing samples, but you need to understand and consider carefully whether i) the use of these existing contact numbers constitutes a security breach under the General Data Protection Regulation or other local laws and ii) for what purpose this existing sample was assembled and who it represents. Random Digit Dialling (RDD) might also be an option in cases where phone contacts are not available. However, be aware that RDD can be inefficient. In this study on Ebola in Liberia, the research team had to make 214,823 calls to achieve 12,761 completed interviews. The points so far listed are not only reflection points about the feasibility of a phone survey, but also key sources of risk you might want to take into consideration if, after reflecting on the operating context and culture, you conclude that a phone survey is suitable for you. If this is the case, there are some practical aspects to keep in mind to ensure that all the mitigation strategies are in place to address (as well as possible) the risks listed above.

Does it fit your budget?

Costs are highly context specific and depend on the cost of labour, cost of use of phone, sample size, duration of the survey, whether you are giving incentives or not and number of call attempts. This World Bank blog post, usefully summarises some costs (averaging around $5 per CATI survey), however, there is recent anecdotal evidence that costs could be much higher:

  • Uganda (2020) - $18 per household for a 20 minutes CATI interview
  • Pacific area (2020) - $24 per CATI interview
  • Cross country (Ethiopia, India, Peru, Vietnam - 2021) longitudinal study [footnote 11] – $26.5 for a 40 min CATI interview

It is worth bearing in mind that productivity of call centres might be lower during the COVID-19 pandemic because they run on reduced staff.

3. Practical considerations to help mitigate risks

The risk of data loss and/or leakage might be higher when remote surveying

Is the enumerator conducting the interview in a quiet environment? Do enumerators have regular access to the internet, electricity and the possibility to upload interviews on a server (and so delete them from the device) in real time? Are survey devices safe enough when working from home? Depending on the replies to these questions, you might want to consider “upgrading” data security systems and protocols accordingly. Also, it’s important to strike a good balance between asking personal information e.g. someone’s name, to build a connection with them (and reduce the risk of break off) and not collecting unnecessary personal identifiers. Collecting additional information that is not critical not only makes respondents more easily identifiable in the case of data leakage or loss, but also puts respondents at potentially higher risk and may affect the quality of the data collected. This is critical for persons with disabilities but will need to be weighed against the need to ensure that they are included in the survey.

Institutional Review Board (IRB) approval must be obtained for additional aspects

Approval by ethical committee boards are a mandatory requirement for all research and evaluation activities involving human participants, regardless of whether interaction happens face to face or remotely. However, be aware that in the context of a phone survey, more (or different) aspects require IRB approval: oral consent – which must be provided before the interview starts, usually through recorded oral consent, audio audits and additional incentives for respondents [footnote 12]. At the time of writing, a number of IRBs have announced that if the research modality already given ethical approval for is changed from face-to-face research to remote techniques then there is no requirement to go through IRB approval again [footnote 13]. However, this should be checked each time as individual institutions may vary in their guidance.

Sensitisation of approach

It’s good practice for any survey to engage with respondents before survey activities start. The aim is twofold, on one hand to introduce the activities to respondents and make them an engaged part of the project and on the other hand to capture in advance any potential risks for the survey approach. This aspect is especially relevant where respondents are approached for the first time (a baseline survey) and in conflict afflicted situations. As much as possible, engage with the communities about major risks related to the proposed data collection and about who may have difficulty engaging and why. This might be done by speaking with some members of the community or a local NGO or other community organisation, or through a quick literature review on the mobile phone landscape (e.g. mobile phone ownership and usage, social and gender norms) in the country.

Protocols for non-response (and refusal to respond) should be in place

An approach to managing non-response depends on the context as well as time and budget availability. Two useful principles are:

i) ensure the calls are spaced (not just calling at the same time of day every day and hoping for a response) ii) the number of call attempts should be as high as reasonably possible

There is a fine line between diligently chasing a respondent and harassment; the need to obtain a response should be balanced against the need to avoid putting people under too much pressure. It is advisable to test this aspect during the pilot to find the culturally appropriate number that maximises responses without wasting resources and causing stress. In a webinar on adaptions with phone surveys, Tavneet Suri (J-Pal) uses 9 attempts, maximum 3 per day, 3 hours apart, one day in between as a protocol for one of her studies; Innovations for Poverty Action (IPA) guideline suggests 5 attempts, with the second attempt 4 hours after the first attempt and third, fourth and fifth attempts 18 hours apart. The Listening to Young Lives at work survey manual (University of Oxford) suggests to schedule the appointment at the most convenient time for the respondent. Experiences in other sectors (like political polling) have found that people are more likely to pick up unknown “local numbers” as compared to unknown, customer care and long-distance numbers. Appropriate sensitisation as described in the point above might also be a good strategy to mitigate this risk.

Attrition

Recent experience from partners suggests that the switch from face-to-face to phone survey because of COVID-19 led to sometimes dramatic drops in response rate. This can challenge the representativeness and power of the study, so it’s important that attrition is monitored carefully by collecting basic demographic information to understand what type of respondents are dropping out of the survey (balanced with the need to keep duration of surveys and risk of collecting not-needed identifiers limited) to enable post-survey bias adjustments (see point below on “Plan strategies to correct sample bias”). Providing monetary incentives is an effective strategy to encourage response (see point below).

Enhanced protocols for data quality should be in place

Due to the bias risks discussed above, and of enumerators working from home, a phone survey requires some additional measures to be included to ensure data quality. First of all, it’s important to have identification protocols in place to ensure the right person is interviewed. Implementing high-frequency data quality checks can help you spot a variety of issues (for example about the instruments, the programming of the digital tools to administer them, or even surveyor performance) immediately and give you the opportunity to rectify them promptly. In addition, you might want to consider whether back-checks [footnote 14] should be carried out on the entire sample and/or questionnaire? If audio audits should be included in the protocols and on what portion of the sample? Is it possible to perform random spot-checks during the administration of the call to verify how the enumerator interacts with the respondents [footnote 15].

Remote training of enumerators requires extra care and attention

Training enumerators to administer the survey is more challenging in the context of COVID-19, where training in-person might not be an option if lock-down restrictions are in place [footnote 16]. The research, monitoring or evaluation team must adapt the logistics and content of the training to a remote training situation. Pre-recorded training sessions are an often (successfully) used strategy; but think carefully about how you can ensure that enumerators are properly prepared for follow up questions from respondents, in particular to respond to requests for assistance, resources or health advice, especially as the economic impacts of the pandemic worsen.

Language diversity issues are exacerbated: Speaking a language perceived as “foreign”, “unfamiliar” or “unusual” on the phone is more difficult than in person. Lack of understanding, confidence or even willingness to converse on the phone with a person speaking with a different accent (or dialect, language or sex) are issues that need to be well thought out. You should carefully consider how, for example, calls from enumerators speaking an official national language might be received from older persons with low or no literacy skills speaking a local language or dialect. Would they be able to understand? Would they feel confident replying to their questions? How are they planning to test language skills of enumerators to match those of the respondents? In the authors’ experience, having the interview administered by an enumerator speaking the same language or dialect as the respondent makes the interview process more efficient and the quality of the information collected better; we think that this is even more the case for phone surveys. Similarly, it is important to consider how the interview will be understood by persons with a variety of impairments (e.g. use of simplified language for persons with intellectual impairments etc). It’s considered good practice to create strict protocols to manage the survey in a multiple language and impairments context: it’s crucial to pilot questionnaires before the data collection round starts to test they are easy to administer for enumerators and to understand for respondents, and to ensure there are data quality checks in place. For example, data auditors speaking the language the survey is administered in are often utilised not only to ensure that the questionnaire is administered properly, but also to check on the correct translation of qualitative information collected.

How is the team planning to incentivise respondents?

Depending on the funding arrangements and operating context, it may be worth considering providing a small gift to respondents to thank them for their time and to incentivise response. Ethical considerations should be applied to assess whether it is possible to do this without causing possible harm for the respondent [footnote 17] and ethical review boards will usually require an amendment submission to the IRB if the compensation is changing. For example, if the survey protocol shifts from an “in-kind” gift handed to the respondent to airtime or gift cards, the IRB might want reassurance that the electronic compensation is a safe option and is received by the right person.

There is wide evidence – mostly from developing countries – that providing monetary gifts incentivises response but there is also evidence that it does so with diminishing returns. For example, this study from Peru suggests that “considerably higher incentive (5$) did not prove much more successful in reducing attrition than a smaller amount (1$)” [footnote 18]. As a minimum, the compensation in a phone survey context should cover recharging and usage costs for the respondents (this is even more relevant in the context of a reduced household income as a result of the pandemic) but in deciding what an appropriate incentive might be, you need to use your best knowledge of the operating context.

There are a number of ways to issue incentives remotely. For example, mobile money transfers or issuing scratch-off cards to enumerators who then read the numbers out to the participant after the interview is done, giving them access to mobile phone credit [footnote 19].

Plan strategies to correct (in addition to prevent) sample bias

If the general characteristics and structure of the population of interest are known and documented (for example by a previous survey or census), statistics and econometrics techniques can be used to ensure that those who are under-represented have additional weights given to their responses, to rectify this bias. Planning for this from the beginning not only gives you a strong mitigation strategy against selection bias but might also provide good reassurances when assessing phone survey feasibility. Here is a good example from the Ebola experience of how researchers managed to adjust for attrition in a phone survey and this article shows the corrective power of post-stratification techniques applied to a sample in Uganda.

Having an understanding of the demographics of the general population will also help you identify disability issues, including prevalence, and whether the country included the Washington Group Questions in any recent census or household level surveys. For further details on this strategy, see this World Bank blog post on using phone surveys to understand the impact of COVID-19.

4. Challenge assumptions

When considering if and how to conduct a phone survey there is not a right or wrong answer, it depends on the context you are operating in. Therefore, it’s important to have a thorough understanding of the local setting and culture, without falling into the trap of just relying on assumptions to inform your decision. To give an example: the COVID-19 pandemic has caused job loss and distress for many, so we should treat people’s time with consideration and sensitivity while they are struggling to find alternative means to provide for them and their families. This a right and sensible consideration and shows sensitivity towards the human subjects we are working with. Nevertheless, evidence from certain contexts challenges this assumption: to quote a New York Times article “people are dealing with anxiety, and they haven’t seen their family and friends [..], they just want to talk to someone”. The article also mentions that executives from several firms confirmed that not only are people more willing to talk, but they are also more likely to stay until the end of the conversation. This seems to be corroborated by some preliminary evidence from partners at the World Bank who are reporting similar patterns, with response rates largely exceeding the expectations formulated based on historical success [footnote 20]. Can something similar be true also in your own context?

5. Additional resources and support

  1. This guidance has been produced through a collaboration between the FCDO Evaluation Unit and Growth Research Team. It is a collection of authors’ considerations and experiences working with phone surveys, compiled by Alex Christopher, Federica Di Battista and Benedetta Musillo. We thank Jessica Vince (FCDO); Maria Kett (Senior Adviser to the Disability Inclusion Helpdesk); Sarah Kopper (J-PAL); Markus Goldstein, Kristen Himelein, and Michael B. O’Sullivan (World Bank); Simone Schaner (USC); Rohini Pande and Charity Moore (Yale University); Dave Runganaikaloo (Cardno); and Marta Favara (University of Oxford) for their useful comments and contributions. We also thank FCDO Evaluation Unit colleagues for their support and inputs. ↩

  2. In web surveys, respondents receive a link to a programmed questionnaire. In recorded surveys, the questionnaire is pre-recorded in either written or oral form, as for example interactive voice response surveys. ↩

  3. In this World Bank blog post, this remote survey toolkit by 60 Decibels, and in the Innovations for Poverty Action’s “Remote surveying in a pandemic” handbook the characteristics of the three remote survey methods are compared and summarised, with specific attention to the COVID-19 context. ↩

  4. A practical way of mitigating potential harm is by asking the respondent upfront if they are in a private place and able to speak with you for the survey duration. This is also an area where call masking can be really useful, and training enumerators, particularly if it’s a survey on domestic violence, on how to respond if the male partner takes the phone or calls back. ↩

  5. For example, Skype, WhatsApp, etc. This of course is a primary consideration if on the other hand you are thinking about a web or recoded survey. ↩

  6. Also, refer to the safeguarding and sensitivity points to think about the risks it might raise for respondent and the power dynamics in the households respectively. ↩

  7. The Washington Group on Disability Statistics promotes and coordinates international cooperation in the area of health statistics focusing on the development of disability measures suitable for census and national surveys, with the main aim of promoting generation of information on disability that is comparable world-wide. ↩

  8. It would be useful to document and share any experience of identifying disabled people in phone surveys; we encourage readers to share examples with the authors and/or with the Disability Inclusion Helpdesk. ↩

  9. Margret Buchholz, Ulrika Ferm & Kristina Holmgren (2020) Support persons’ views on remote communication and social media for people with communicative and cognitive disabilities, Disability and Rehabilitation, 42:10, 1439-1447. ↩

  10. It is worth flagging that practitioners are experiencing several issues in contacting respondents even in follow-up survey due to poor quality contact number information collected at baseline. ↩

  11. Note that survey costs associated with longitudinal studies are on average more expensive as keeping the cohort intact is a key priority. This usually requires going the extra mile to reach out to respondents. ↩

  12. IRB requirements depends on the institution hosting it and on the type of study you are seeking ethical clearance for. Common requirements are informed consent (communicating to respondents about the objective and scope of the study, whether they should expect any compensation, if and how they are going to benefit from the study , the duration of the call, whether the data will be matched with other existing dataset, and any issue related to anonymity), questionnaire, study design and some cover submission form that is specific to the institution. Consider that IRB application can require additional documents and reassurances if the study is dealing with vulnerable categories, particularly children, prisoners and victims of domestic abuse: make sure you have strong protocols in place to deal with these scenarios. ↩

  13. For example, IPA’s IRB protocol is that of not requiring an amendment submission to the ethical board if i) study protocols are only adjusting to reflect shift from face-to-face to phone survey, ii) the only change in the informed consent is to reflect the change in the data collection mode and iii) the only questionnaire change is to eliminate questions. See the “IRB and data protection” section in the IPA’s “Phone Surveying in a Pandemic: Handbook” for more information. ↩

  14. https://dimewiki.worldbank.org/wiki/Back_Checks ↩

  15. To the best of our knowledge we are not aware of a survey platform allowing you to do this unless the implementing partner is relying on 3-ways calling capability, however it might be interesting to monitor this space to observe and encourage partners to think creatively about possible solutions. ↩

  16. It’s worth reporting that some partners operating surveys in Sub-Saharan Africa have reported lock-down measures relaxed to the point where in-person training was operable with due precautions – although face-to-face interviews are not. So, we might expect this point to be relaxed sooner. ↩

  17. See the DFID ethical guidance for research, evaluation and monitoring activities under the data collection and analysis section for more useful pointers. ↩

  18. See IPA’s “Phone surveying in a Pandemic: Handbook” for more details and references. ↩

  19. Note that with this last option, it is good practice to have an audit mechanism to ensure that respondents actually received the phone credit. ↩

  20. This is still very preliminary evidence, and more formal studies are needed as results come in from a wider variety of context. ↩

Back to top