Looking back but moving forward--the opportunities and challenges of a mixed mode approach to survey research.
|Abstract:||The purpose of this paper is to present a brief history of survey modes, encourage the use of multi mode surveys and identify inherent challenges in using a multi mode approach. While multi mode surveys are not a new phenomenon recent technological advancements such as the increased use and reliance of computers, online access, internet, and cell and smart phones as well as social media have expanded the opportunities to improve the quality and cost effectiveness of survey research in health and other fields.|
|Publication:||Name: American Journal of Health Studies Publisher: American Journal of Health Studies Audience: Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2011 American Journal of Health Studies ISSN: 1090-0500|
|Issue:||Date: Spring, 2011 Source Volume: 26 Source Issue: 2|
|Topic:||Computer Subject: Internet|
Data collection of samples or populations certainly is not a modern
phenomenon. Greeks and Romans conducted censuses thousands of years ago.
For centuries methodologies for data collection have continually evolved
along with available technologies. In more recent times surveying has
been a major means of collecting data on and about a sample or
population of interest. Survey researchers are continually challenged to
decide which method or mix of methods is optimal for their purposes. In
this case optimal may not be the best conceivable method or mix of
methods but what is available and practical given considerations such as
time and cost constraints and quality of the data to be obtained.
The purpose of this paper is to present a brief history of survey modes, encourage the use of multi mode surveys and identify inherent challenges in using a multi mode approach. By no means are multi mode surveys a new concept. They have been used for several centuries. However, technological advances such as the use of computers, expanded online access, the internet, and cell and smart phones make multi mode surveys both possible and attractive for those doing survey research. Multi mode surveys offer increased opportunities to make surveys larger and more affordable, increase the efficiency of the data collection process, improve response rates and enhance data quality. However, they are not a panacea. They, like any other individual survey mode, should be used judiciously while realizing and addressing potential pitfalls.
SURVEY MODES FROM PAST TO PRESENT
The earliest documented survey dates from 1788 when Sir John Sinclair used a mail survey to all ministers in all parishes of the Church of Scotland. Sinclair was persistent in his desire for a good response rate. He followed-up his survey with 23 reminder letters and also used 'statistical missionaries' to personally visit non-responders. His tenaciousness paid off with a 100% response rate (Whithers, 2010 at www.edina.uk). Mail surveys, with adequate follow-up, are still considered an efficient data collection method (Dillman, 2000).
The first scientific face-to-face survey took place in 1912 when Sir Arthur Bowley conducted a study of working-class conditions in five British cities in which samples of citizens were interviewed using a structured interview schedule. (De Leeuw, 2005). Until the 1960s face-to-face interviewing was considered the gold standard among survey researchers.
The use of telephoning as a major means of collecting survey data became even more prominent in the 1980s with the rapid growth of the computer and the introduction of computer-assisted telephone interviewing (CATI) which integrated and facilitated data collection and data processing. CATI provided for greater efficiency and improved data quality and became a useful tool to screen potential respondents and follow-up nonrespondents (Dillman, 2000).
The next advance of integrated programs was computer-assisted personal interviewing (CAPI) used in face-to-face interviewing and computer-assisted self interviewing (CASI), usually conducted in concert with CAPI. CAPI-CASI have been used especially with interviews on sensitive topics where the interviewer hands over a hand-held computer to the respondent so he or she can answer sensitive items privately (De Leeuw, 2002).
THE EMERGENCE OF THE INTERNET AND E-MAIL
The latest modes include a host of new techniques made available in the 1990s by the Internet and e-mail and their rapid adoption in the population. Today, Web-based surveys are commonplace. In e-mail surveys the questionnaire may be contained in the e-mail or included as an attachment. The respondent can complete the questionnaire by simply replying to the original e-mail. In Web-based surveys the questionnaire is on a Web site. To complete the survey the respondent can either click on a hyperlink in the e-mail inviting their participation or by typing, or copying/pasting the Web address into the browser window and then completing the questionnaire. Both e-mail and Web based surveys have become very popular (Sue and Ritter, 2007; Smyth and Pearson, 2011). Internet surveys are very cost effective, can reach large populations and make large surveys affordable. They can also be done at the convenience of the respondent, used for complex questionnaires by automatically skipping to appropriate questions based on respondent replies, may be better than phone or face-to face questions for sensitive topics, and provide quick turnaround.
However, as with all modes, there are limitations. Underecoverage is a concern (Couper, 2000; Sue and Ritter, 2007). Not everyone has a computer or knows how to use wireless technology to access the Internet. The Pew Internet and American Life project estimated that in 2010 79% of American adults use the Internet (http://www.pewinternet. org/Static-Pages/Trend-Data/Whos-Online.aspx). Nor is Internet usage spread evenly over all groups. Non-Hispanic blacks, the elderly, those with a lower education and income, and those living in rural places are underrepresented. However, cell phone and wireless laptop internet use have each grown more prevalent and gaps between groups have narrowed. The Pew Internet and American Life project report indicates that nearly half of all adults (47%) go online with a laptop using a Wi-Fi connection or mobile broadband card (up from the 39% who did so as of April 2009) while 40% of adults use the internet, email or instant messaging on a mobile phone (up from the 32% of Americans who did this in 2009). Fifty-nine percent of adults now access the internet wirelessly using a laptop or cell phone compared to 51% who used a laptop or cell phone wirelessly in April 2009.
Encouraging also is the finding that African-Americans and English-speaking Latinos continue to be among the most active users of the mobile web. Cell phone ownership is higher among African-Americans and Latinos than among whites (87% vs. 80%). Minority cell phone owners take advantage of a much greater range of their phones' features compared with white mobile phone users. Also, and not surprisingly, young adults (those ages 1829) are also avid users of mobile data applications, but older adults are gaining fast. Compared with 2009, cell phone owners ages 30-49 are significantly more likely to use their mobile device to send text messages, access the internet, take pictures, record videos, use email or instant messaging, and play music. (http://www.pewinternet.org/Reports/2010/ Mobile-Access-2010.aspx).
Another chronic problem and limitation of online and Web-based surveys is a poor response rate which may result in a systematic bias. Online or Web-based surveys are notoriously known for their low response rates. Most respondents receive multiple messages a day and can quickly and easily delete the invitation to participate or the questionnaire itself. Abandonment, where the respondent quits in the middle of the questionnaire, also is commonplace. Other limitations to be discussed later in the paper are dependence on software to create and deploy questionnaires. Software products vary greatly in cost, ease of use, flexibility and compatibility with the respondent's computer. What is seen on the screen by the survey developer or one respondent may well be seen differently by another respondent. However, despite these limitations, online and Web-based surveys can be an integral and valuable part of a mixed mode survey approach.
MIXED OR MULTI MODE SURVEYS
The use of various methods of data collection are called mixed or multi mode surveys. The underlying rationale for using mixed modes is that it is well known that some people respond better to one method than another. For example, in the 1960s and the 1970s telephone follow-up was used if face-to-face interviews were not completed or mail questionnaires were not returned. However, there are implications of collecting data from different modes. For example, self administered modes such as mail or Web-based ones may result in different responses than in interviewer involved modes such as face-to-face or by telephone, especially for sensitive topics. In the case of a face-to-face interview this problem can be addressed by giving the respondent a paper questionnaire or hand held computer.
One reason for using mixed modes is to improve response rates. In using a single mode such as a mail survey or telephone mode using repeated follow-up attempts generally increases the response rate. However, reliance on a single mode to increase the response rate may also exacerbate a bias by only adding more of a certain demographic such as gender. Similarly, follow-up using a single mode on a survey about cancer may only result in adding more health conscious respondents. Using a multi mode approach does not eliminate this possibility but may lessen the possibility since some people are more likely to respond more and better to one method than another.
CHALLENGES IN USING MIXED MODES
Modes are means of data collection. Biemer and Lyberg (2003) define an optimal method of data collection as the best method given the research question and certain restrictions. Among others, restrictions may include time and cost, privacy and ethical considerations, as well as the inherent challenges of a given method. Sometimes the optimal method is the use of multiple modes by utilizing the advantages of each mode while compensating for the limitations of each. By doing so the researcher should try create an optimal mix that maximizes cost effectiveness while reducing errors due to coverage, nonresponse and measurement errors. Often this involves tradeoffs. Use of multi modes involves an understanding and acceptance of tradeoffs in the pursuit of optimizing research goals of the particular study. Multi modes used in one study may have only modest applicability in another study.
One tradeoff of using mixed modes is the real possibility of mode effects. Many comparisons of mode effects have been made for traditional methods such as face-to face-interviews, mail-self administered and telephone studies. In a meta analysis of 67 articles and papers, DeLeeuw (1992, Chapter 3) found consistent but small differences between modes while noting a dichotomy between studies with and without an interviewer. Not surprisingly, less is known about the more recent use online and Web-based modes. An excellent recent coverage of multiple mode articles, including a wealth of useful references, can be found in the online journal issue Survey Practice at http://surveypract.ice. org/2010/10/. Articles include the timing of multiple modes and response rate, comparing different modes and response rates as well as mixed mode surveys and the Internet.
Another inevitable challenge in using mixed modes is the issue of data equivalence. Unlike single modes where there is no threat of mode effects on data collection, using multiple modes may influence the data and thus threaten validity. DeLeeuw, 2005 correctly surmises that this places the research between a rock and hard place. One the one hand multiple modes are used to reach respondents that may not be reached otherwise in order to reduce coverage and nonresponse error. But doing that opens up the distinct possibility that respondents may answer differently to different modes. Research to date indicates general equivalency between test data collected over the Internet and by paper and pencil. Subjects appear to use the same processes in responding to each mode. Far less is known in comparing Internet surveys with either telephone or face-to face-interviewing. To date, there is no comprehensive meta-analysis on this topic. One early and somewhat persistent finding is that Internet surveys are more likely to report less socially desirable responses. In comparing a web survey to a telephone survey, Link and Mokdad (2005) found web respondents more likely to self report heavy drinking compared to those in a telephone interview. Similarly, Kreuter et al (2008) found web survey respondents to report more accurate information to factual questions on educational achievement such as GPA. Fortunately, as previously mentioned, strategies exist to reduce but certainly not eliminate this possibility in other modes.
DATA REDUCTION AND ANALYSIS
In single mode surveys data are collected using a single format. In multi mode surveys data may be collected under different restrictions and formats that later need to be integrated into a common format. When using mixed modes Dillman and Christian (2003) emphasize constructing equivalent questions across each mode. At times, this presents challenges relating to cost, time and quality issues. For example, including graphs or charts in an e-mail survey may create a file that is subsequently blocked due to its large size. E-mail surveys limit the types of questions since they can't automatically skip questions. Web surveys require the use of specific software that may not afford asking the same question in the same format as in another mode or provide for easy transfer to the data file of other modes. Also, respondents may not be able to see the questions in the same format as other modes due to computer or device screen size.
SIMULTANEOUS OR SEQUENTIAL MODE ADMINISTRATION
After a researcher decides to use multiple modes of data collection, there is another issue: should they be simultaneous or sequential? For example, should the respondent first be sent an email invitation to an Internet-based survey and then followed up with a mailed questionnaire or telephone interview--or --should all of the modes be offered to respondents at the same time? If the decision has to be based on cost then try the least expensive method first. On the other hand, because some types of potential respondents will be more likely respond to one method than another, offering them simultaneously may increase response rate. Consequently there is no one answer to this question.
Mixed-mode surveys have many potential advantages. Those contemplating doing survey research should seriously consider utilizing a multi-mode approach. There is a wealth of methodological literature on established survey modes such as mail, face to face, telephone and self administered questionnaires. However, technological advances have expanded the available modes. The ubiquitous use of computers, expanded online access, the internet, and cell and smart phones create new and attractive opportunities for those doing survey research in health and other areas. Likewise, these same technological advances create new challenges. There is little doubt that some of inherent challenges of using multiple modes will be addressed as the technology and knowledge base improves.
Biemer, P. P. and Lyberg, L. E. (2003). Introduction to Survey Quality. New York: John Wiley.
Couper, M. P. (2000). Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly, 64, 464-494.
De Leeuw, E. D. (1992). Data Quality in Mail, Telephone, and Face-to-face Surveys. Amsterdam: TT-Publicaties.
De Leeuw, E. D. (2002). The Effect of Computer-assisted Interviewing on Data Quality: A Review of the Evidence [CD-Rom]. In J. Blasius, J. Hox, E. de Leeuw, and P. Schmidt (eds), Social Science Methodology in the New Millennium, Opladen, FRG: Leske + Budrich.
De Leeuw, E. D. (2005). "To Mix or Not to Mix Data Collection Modes in Surveys." Journal of Official Statistics (JOS), 21, 233-255. Also available at www.jos.nu:21 June 2010 http://www.jos.nu/Articles/ abstract.asp?article=212233.
Dillman, D. A. (2000). Mail and Internet surveys: The Tailored Design Method. New York: John Wiley.
Dillman, D. A. and Christian, L.M. (2003). Survey Mode as a Source of Instability in Responses Across Surveys. Field Methods, 15(2), 1-22.
Kreuter, F, Presser, S. and Tourangeau, R. (2008). "Social Desirability Bias in CATI, IVR, and Web Surveys. The Effect of Mode and Question Sensitivity." Public Opinion Quarterly, 2008, 72, 5, 847-865. Also available online: 21 June, 2010 http://poq.oxfordjournals.org/content/vol72/issue5/#ARTICLES
Link, M. W. and Mokdad, A H. (2005). "Effects of Survey Mode on Self-reports of Adult Alcohol Consumption: A Comparison of Mail, Web and Telephone Approaches." Journal of Studies on Alcohol, March 2005, pp239-245.
Smyth, J. D., and Pearson, J. E. (2011). "Internet Survey Methods: A Review of Strengths, Weaknesses, and Innovations." Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies. Ed. Marcel Das, Peter Ester, Lars Kaczmirek. New York: Taylor & Francis Group, pp. 11-44.
Sue, V. M. and Ritter, L. A. (2007). Conducting ONLINE Surveys. Thousand Oaks, CA: Sage Publications Inc.
Whithers, C. W. J. (2010) "Scotland Accounted For: An Introduction To The Old (1791-1799) And The New (1834-1845) Statistical Accounts Of Scotland." EDINA, The Statistical Accounts of Scotland. Available online 23 November 2010: http://edina.ac.uk/stat-acc-scot/reading/.
Thomas O'Rourke, Ph.D., MPH, CHES
Thomas O'Rourke, Ph.D., MPH, CHES, is Professor Emeritus of Community Health at the University of Illinois. Please address all correspondence to Thomas O'Rourke, Ph.D., MPH, CHES 1206. S. Fourth St. Rm. 129, University of Illinois, Champaign, IL 61820; PHONE: (217) 840-7036; FAX: (217) 333-2766; EMAIL: email@example.com
|Gale Copyright:||Copyright 2011 Gale, Cengage Learning. All rights reserved.|