Survey confidentiality vs. anonymity: young men's self-reported substance use.
Substance abuse (Surveys)
Health surveys (Evaluation)
Moore, Roland S.
Ames, Genevieve M.
|Publication:||Name: Journal of Alcohol & Drug Education Publisher: American Alcohol & Drug Information Foundation Audience: Academic; Professional Format: Magazine/Journal Subject: Health; Psychology and mental health; Social sciences Copyright: COPYRIGHT 2002 American Alcohol & Drug Information Foundation ISSN: 0090-1482|
|Issue:||Date: Wntr, 2002 Source Volume: 47 Source Issue: 2|
|Geographic:||Geographic Scope: United States Geographic Code: 1USA United States|
A key concern in the literature on survey validity is how the absence of anonymity affects responses to questions regarding sensitive personal issues. This paper reports on an experiment conducted to see if respondents who provided their identification to researchers would be as forthcoming regarding substance use as anonymous respondents from the same population. A sample of 1811 male entrants into a U.S. military branch (mean age= 18.9 years), using self-administered questionnaires, provided self-reports of alcohol, tobacco, and other drug use. Confidential identification codes were visible on 1507 instruments, and absent of 304 instruments. No statistically significant differences were found between the 2 groups' self-reported substance use over the previous 12 months. The findings suggest that the lack of total anonymity in the confidential mode of survey administration does not necessarily impede the same kind of self-reports of alcohol, tobacco and other drug consumption given anonymously.
[Keywords: Survey Methods, Data Collection, Alcohol Consumption, Tobacco, Illicit Drugs]
How does the absence of anonymity affect the quality of responses to survey questions regarding sensitive personal issues, or issues that if exposed, could put an individual respondent at risk? This paper seeks to answer that question by reporting on the outcome of an experiment conducted in a population of young adult males entering a military workforce to assess whether the presence or absence of anonymity within a confidential survey would influence disclosure of rates of alcohol, tobacco, and other drug use.
Survey expert Don Dillman has described confidentiality as "an ethical commitment not to release results in a way that any individual's responses can be identified as their own." Dillman adds that "Only when the sponsor cannot identify each person's response, even momentarily, is it appropriate to promise that a response is anonymous" (Dillman, 2000, p. 163). In other words, confidential respondents are known only to the researchers, and anonymous respondents are not identifiable even by the researchers.
A review of the literature comparing modes of survey administration indicated that self-reported alcohol or drug use varies in accuracy, depending upon how questions are administered (Harrison & Hughes, 1997; Gmel, 2000; Kraus & Augustin, 2001). For example, Hamid, Deren, Beardsley, & Tortu, 1999) found that respondents were much more likely to report drug use after rather than before undergoing a urine test.
It is beyond the scope of this article to review the inconsistencies of estimating substance use including methods other than self-reports (such as collateral accounts, sales records, etc.), but such studies are plentiful (Longford, Ely, Hardy, & Wadsworth, 2000). A classic review article on the validity of alcohol consumption self-reports (Midanik, 1982) suggests that under-reporting may result from denial or failure to recall (especially in response to questions on longer timeframes). However, Midanik (1982) also mentions youthful bragging as the source of over-reporting of some drug use. Other data sources for establishing validity of self-reported substance use, such as collateral reports, official or archival records, or alcohol sales data, are not necessarily more accurate than self-reports to establish consumption rates or the number of substance-related problems (Midanik, 1982).
Even in anonymous surveys, social acceptability can influence responses (Embree and Whitehead, 1993; Kraus and Augustin, 2001). In sensitive domains, including self-reported substance use and sexual behavior, comparisons of anonymous and confidential survey administration have indicated some differences in response rates, even if the responses themselves did not differ significantly by mode (Seigal et al., 1988).
Leonhard and colleagues (1997) found that anonymity accounted for less that 1% of the variance in an experimental comparison of anonymous and confidential self-reports among respondents undergoing treatment for substance abuse. Similarly, a survey of self-reported drug consumption and attitudes found few significant differences between a confidential group and an anonymous group of adolescent respondents (Malvin ,& Moskowitz, 1983; O'Malley, Johnston, Bachman, & Schulenberg, 2000).
Total anonymity is impossible in panel, studies, in which it is necessary to track respondents and match their data between successive waves of surveys. The experiment reported in this paper was designed to compare differences in responses between a confidential group in the first wave of a panel study with young military recruits and an anonymous group drawn from the same population. All of the respondents in the confidential group were asked to provide their identification numbers with the understanding that these numbers would be matched with addresses for the follow up second and third wave of surveys.
The methodological significance of this experiment to alcohol, tobacco, and other drug research is fundamental: As self-reports are the primary form of gaining information about substance use, reviewing evidence for the accuracy of responses is essential. Although a military population may not be representative of the general population in the United States, clearly the military workforce is a substantial and important population in the U.S. comprising nearly 1.4 million personnel on active duty (United States Department of Defense, 2002).
In this experiment, responses to questions about alcohol, tobacco and other drug use were compared between a larger group taking part in the first wave of a 3-wave panel study and a subset of respondents from the same population who were asked to answer the same survey questions anonymously using instruments with no identifiers. In 1998, data were obtained from a sample of 1,811 male entrants into a branch of the United States military (mean age =18.9 years), using self-administered questionnaires presided over by trained non-military researchers in randomly selected regular classroom sessions at a basic training camp.
Identifier codes were visible on the covers of 1,507 of the instruments (the confidential group), and absent on 304 instruments (the anonymous group). For the confidential group, a tear-off section of the survey instrument requested names and social security numbers. This tear-off form bore the same code as the one on the cover. These forms were collected and sealed in an envelope in view of the respondents. Respondents were informed that the sealed envelopes would remain with the researcher until he or she placed them in a locked cabinet at the research center.
To the confidential group, the presiding researchers explained how confidentiality would be protected. Specifically, the respondents were told about the purpose of the study, the use of the survey data, the procedures for connecting codes on the questionnaires to the respondents' names and addresses for follow-up surveys, separation of identifying information from survey responses, and that the researchers' professional ethics (backed by a National Institutes of Health-issued Certificate of Confidentiality) obliged the researchers to maintain the privacy and confidentiality of individual survey responses.
The respondents were given the opportunity to ask questions about the confidentiality of their answers. Questions to which the survey team responded included how the survey data would be used and requests for further assurances that military superiors would not have access to individual responses.
The anonymous group received similar instructions but were told to leave the tear-off form blank and to refrain from writing their names or other uniquely identifying information on the surveys.
Response rates were calculated by dividing the number of usable surveys by the total number of people who were registered in each class, including those who were absent because of illness or guard duty. Response rates were 93% for the confidential group and 94% for the anonymous group.
Outcome measures included a variety of self-reports on alcohol and other drug use in the year preceding entry to basic training. After being told what constituted a standard drink of four different kinds of alcohol (beer, wine, liquor , and other alcoholic beverages), respondents were asked separate questions on the usual quantity of those beverages consumed per day, and on how many days they drank in the preceding year (everyday--none on a 9-point scale). They were asked the maximum number of drinks they had consumed in one day, the number of days they had consumed that maximum number of drinks. They were also asked on how many days they drank five or more drinks. A series of questions asked the frequency with which 12 drugs (illicit, or licit drugs without a doctor's prescription) were consumed (again, using a 9-point scale from everyday--none). Cigarette consumption was also measured with questions regarding the frequency and quantity of cigarettes smoked in the 30 days prior to entering basic training.
Frequency measures were converted from scaled responses to the numbers of days they represented. For example, the response "once or twice a week during the past 12 months" was converted to 78 days (1.5 x 52). To normalize the distribution of the responses, (in order to more closely conform to the assumptions of the comparative parametric tests used), the self-reported alcohol, tobacco and other drug quantity and frequency measures were log transformed. Simple bivariate analyses (t-tests for equality of means) were used to compare use among the two groups. Levene's test was used to determine if the t-tests should be calculated with or without equal variances assumed. Although the statistical tests were performed on long-transformed values, mean values for all analysis are presented in their untransformed state to allow for easier assessment.
Tables 1-4 show comparisons between the confidential group (N=1507) and the anonymous group (N=304) on various alcohol, tobacco and other drug consumption measures.
In all of the measures of quantity and frequency of alcohol consumption (Table 1 and Table 2), the logged means of the responses from the anonymous and confidential groups did not differ significantly.
In both quantity and frequency measures of cigarette use, the differences were non-significant between the two groups (Table 3).
Other Drug Consumption
None of the logged self-reported drug consumption means differed significantly between the groups (Table 4). Relatively few self-reports of high frequency of use in the anonymous group raised the means in some of the drug categories. However, log-transforming the data normalized their distribution for comparison.
Findings from this experiment indicate that the lack of total anonymity (i.e., names and social security numbers are given) in the panel study sample does not appear to impede the same kind of self-reports of alcohol and drug consumption as those given by an entirely anonymous subset of the same population. No statistically significant difference was found between the means of the groups' self-reports on a variety of measures concerning alcohol and drug consumption over the previous 12 months. This finding echoes the conclusions of Bjamason and Adalbjarnardottir (2000), who found that identifier numbers necessary for longitudinal research do not unduly bias self-reported substance use by adolescents.
It could be argued that log-transforming the measures had a flattening effect, increasing the likelihood that the two groups would not differ significantly. As a check, the means of the anonymous and confidential group responses were also compared in their unlogged form. With one exception (the confidential group reported a mean 7.25 days drinking the maximum self-reported drinks per day, vs. 14.12 days by the anonymous group, p<.013), the anonymous and confidential groups did not differ significantly on any of the measures.
This experiment's sample consisted of men who had just entered the military. It is difficult to use any particular population sample to generalize to a larger group. The findings may not necessarily be generalizable to other populations, such as college, or treatment samples, or the U.S. population in general. Given the size of the military workforce, these results can be used to generalize to an important sub-population in this country.
At the time of the survey, the United Sstates military policy on drug use was one of zero tolerance. This is, if drug use is detected, the enlisted individual must undergo treatment and rehabilitation, after which the individual is discharged from the military. Although the respondents were asked about their substance use prior to entering basic training in the military rather than current use, the military's strong anti-drug climate did not appear to affect the validity of responses in the confidential group. It is worth underscoring that the recruits were being asked about their substance use in the year prior to entering the military and when they were still classified as civilians. Thus, they were not necessarily jeopardizing their new jobs in the Navy by disclosing use of drugs prior to their entering the service.
As Hamid et al. (1999) note in their review of the literature on self-reported drug use, the circumstances in which interview and drug tests are conducted strongly affect the reliability and validity of such self-reports. The confidential survey administration in this experiment benefited from experienced researchers answering questions, separating identifiers from questionnaires in the presence of respondents, and giving assurance that only data analysts would ever connect the data across the three surveys, and that they would report the results in aggregate rather than individual form.
The lack of significant differences in the present study's responses support the conventional wisdom that the presence or absence of identifiers do not make much of a difference if respondents are convinced that the data will be kept confidential. On the other hand, if respondents do not trust the logistics of the survey, they will hesitate to give entirely truthful responses to sensitive questions even if the identifier is left off.
Bjarnason, T., & Adalbjamardottir, S. (2000). Anonymity and confidentiality in school surveys on alcohol, tobacco, and cannibis use. Journal of Drug Issues, 30(2), 335-344.
Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method. (2nd ed.). (p. 163) New York: John Wiley & Sons.
Embree, B.G., & Whitehead, P.C. (1993). Validity and reliability of self-reported drinking behavior: Dealing with the problem of response bias. Journal of Studies on Alcohol, 54, 334-344.
Gmel, G. (2000). The effect of mode of data collection and of non-response on reported alcohol consumption: A split-sample study in Switzerland. Addiction, 95, 123-134.
Hamid, R., Deren, S., Beardsley, M., & Tortu, S. (1999). Agreement between urinalysis and self-reported drug use. Substance Use and Misuse, 34, 1585-1592.
Harrison, L., & Hughes, A. (Eds.). (1997). The Validity of Self-Reported Drug Use: Improving the Accuracy of Survey Estimates. NIDA Research Monograph 167. Rockville, MD: National Institute on Drug Abuse.
Kraus, L., Augustin, R. (2001). Measuring alcohol consumption and alcohol-related problems: Comparison of responses from self-administered questionnaires and telephone interviews. Addiction, 96, 459-471.
Leonhard, C., Gastfriend, D.R., Tuffy, L.J., Neill, J., & Plough, A. (1997). The effect of anonymous vs. nonanonymous rating conditions on patient satisfaction and motivation ratings in a population of substance abuse patients. Alcoholism: Clinical and Experimental Research, 21, 627-30.
Longford, N., Ely, M., Hardy, R., & Wadsworth, M.E.J. (2000). Handling missing data in diaries of alcohol consumption. Journal of the Royal Statistical Society, Series A, 163, 381-402.
Malvin, J.H., & Moskowitz, J.M. (1983). Anonymous vs. identifiable self-reports of adolescent drug attitudes, intentions and use. Public Opinion Quarterly, 47, 557-566.
Midanik, L. (1982). The validity of self-reported alcohol consumption and alcohol problems: A literature review. British Journal of Addiction, 77, 357-382.
O'Malley, P.M., Johnston, L.D., Bachman, J.G., & Schulenberg, J. (2000). A comparison of confidential versus anonymous survey procedures: Effects on reporting of drug use and related attitudes and beliefs in a national study of students. Journal of Drug Issues, 30, 335-54.
Siegel, H.A., Leviton, L.C., Cole, P.A., Wang, J., Bachmann, L., & Hook, E.W., 3rd. (1988). Feasibility of obtaining sexual risk and STD history in the context of a drinking drivers' program. Public Health Reports, 113, 341-345.
United States Department of Defense. (2002). Population Representation in the Military Services. Fiscal Year 2000. Arlington, VA: Department of Defense.
This research was funded by National Institute on Alcohol Abuse and Alcoholism Grant #AA06282-17. The authors appreciate the assistance of Barry Hoag in arranging the surveys and the valuable methodological suggestions offered by Joel Grube, Robert Lipton, Andrew Ho, Patricia Madden and William Ponicki.
ROLAND S. MOORE, Ph.D *.
GENEVIEVE M. AMES, Ph.D.
Prevention Research Center,
Pacific Institute for Research and Evaluation
Roland S. Moore, Please send all correspondence to first author:
Roland S. Moore, Ph.D.
Prevention Research Center
2150 Shattuck Ave., Suite 900
Berkeley, CA 94704-1314
Phone 510-883-5770, Fax 510-644-0594
Table 1. Usual and Maximum Quantity of Alcohol Use in Past 12 Months (in standard drinks per drinking day) Confidential Anonymous mean logged mean logged t-test of Significance values values means * (2-tailed) Usual 3.64 3.58 .143 .886 quantity beer Usual .75 .70 .049 .961 quantity wine Usual 2.93 2.68 .828 .408 quantity liquor Usual 1.62 1.85 -1.030 .303 quantity other alcohol Maximum # of 8.80 8.40 .340 .734 drinks in 1 day * For purposes of parametric assumptions of normality, logged means were used for the t-test. Table 2. Frequency of Alcohol Use in Past 12 months (number of days per week) CONFIDENTIAL ANONYMOUS mean logged mean logged t-test of Significance values values means * (2-tailed) Days 64.51 62.00 -.006 .995 drinking beer Days 8.25 7.81 -.324 .746 drinking wine Days 35.80 37.57 -.445 .657 drinking liquor Days 25.25 29.19 -1.084 .279 drinking other alcohol Days 7.25 14.12 -1.810 .071 drinking max. # of drinks Days 16.09 13.53 1.055 .291 drinking 5+ drinks * For purposes of parametric assumptions of normality, logged means were used for the t-tests. Table 3. Quantity and Frequency of Cigarette Use in Past 30 Days (number of cigarettes per day and number of smoking days per month) Confidential Anonymous mean logged mean logged t-test of Significance values values means * (2-tailed) 30-day 8.61 8.81 .033 .974 frequency of cigarette use 30-day 7.76 7.81 -.349 .727 quantity of cigarette use * For purposes of parametric assumptions of normality, logged means were used for the t-tests. Table 4. Frequency of Illicit Drug Use: Days in Past 12 Months CONFIDENTIAL ANONYMOUS mean logged mean logged values values Marijuana 25.07 31.16 Amphetamines 3.29 4.24 Cocaine .95 2.98 Heroin, .40 1.44 narcotics PCP .15 1.88 Other 1.57 4.21 hallucinogens Sedatives 2.21 2.05 GHB, Rohypnol .35 1.41 Other .90 1.86 tranquillizers Inhalants .63 1.25 Ecstasy .56 1.47 Steroids .61 1.26 t-test of Significance means * (2-tailed) Marijuana -1.110 .267 Amphetamines .233 .816 Cocaine -.394 .693 Heroin, -.609 .543 narcotics PCP -1.570 .117 Other -1.523 .129 hallucinogens Sedatives 0.55 .956 GHB, Rohypnol -1.421 .156 Other .107 .915 tranquillizers Inhalants 1.151 .250 Ecstasy -.890 .374 Steroids -.366 .715 * For purpose of parametric assumptions of normality, logged means were used for the t-tests.
|Gale Copyright:||Copyright 2002 Gale, Cengage Learning. All rights reserved.|