Document Detail

Development of the adult and child complementary medicine questionnaires fielded on the National Health Interview Survey.
Jump to Full Text
MedLine Citation:
PMID:  24267412     Owner:  NLM     Status:  Publisher    
The 2002, 2007, and 2012 complementary medicine questionnaires fielded on the National Health Interview Survey provide the most comprehensive data on complementary medicine available for the United States. They filled the void for large-scale, nationally representative, publicly available datasets on the out-of-pocket costs, prevalence, and reasons for use of complementary medicine in the U.S. Despite their wide use, this is the first article describing the multi-faceted and largely qualitative processes undertaken to develop the surveys. We hope this in-depth description enables policy makers and researchers to better judge the content validity and utility of the questionnaires and their resultant publications.
Barbara J Stussman; Christina D Bethell; Caroline Gray; Richard L Nahin
Publication Detail:
Type:  JOURNAL ARTICLE     Date:  2013-11-23
Journal Detail:
Title:  BMC complementary and alternative medicine     Volume:  13     ISSN:  1472-6882     ISO Abbreviation:  BMC Complement Altern Med     Publication Date:  2013 Nov 
Date Detail:
Created Date:  2013-11-25     Completed Date:  -     Revised Date:  -    
Medline Journal Info:
Nlm Unique ID:  101088661     Medline TA:  BMC Complement Altern Med     Country:  -    
Other Details:
Languages:  ENG     Pagination:  328     Citation Subset:  -    
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): BMC Complement Altern Med
Journal ID (iso-abbrev): BMC Complement Altern Med
ISSN: 1472-6882
Publisher: BioMed Central
Article Information
Download PDF
Copyright © 2013 Stussman et al.; licensee BioMed Central Ltd.
Received Day: 22 Month: 4 Year: 2013
Accepted Day: 21 Month: 11 Year: 2013
collection publication date: Year: 2013
Electronic publication date: Day: 23 Month: 11 Year: 2013
Volume: 13First Page: 328 Last Page: 328
PubMed Id: 24267412
ID: 3882880
Publisher Id: 1472-6882-13-328
DOI: 10.1186/1472-6882-13-328

Development of the adult and child complementary medicine questionnaires fielded on the National Health Interview Survey
Barbara J Stussman1 Email:
Christina D Bethell2 Email:
Caroline Gray3 Email:
Richard L Nahin1 Email:
1National Center for Complementary and Alternative Medicine, National Institutes of Health, Bethesda, MD, USA
2School of Medicine, Child and Adolescent Health Measurement Initiative, Oregon Health & Sciences University, Portland, OR, USA
3Palo Alto Medical Center, Health Policy Research Institute, Palo Alto, CA, USA


The National Health Interview Survey (NHIS) is an annual in-person household survey of the health of the U.S. civilian, non-institutionalized population conducted by the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). In 2002, 2007 and 2012 supplemental questions on complementary medicine were added to the NHIS. These questionnaires filled a void for large-scale, nationally representative datasets on the out-of-pocket costs, prevalence, and reasons for use of complementary therapies in the U.S. All three surveys were sponsored primarily by the National Center for Complementary and Alternative Medicine (NCCAM), part of the National Institutes of Health (NIH). The first NHIS complementary medicine survey, fielded in 2002, was initiated in-part by the NCCAM 5-Year Strategic Plan 2001–2005, which included the creation of a Special Populations Program to, among other goals, support research on complementary medicine use in racial and ethnic minority populations [1]. In the early 2000’s, existing datasets for complementary medicine use in the U.S. lacked sufficient sample sizes to analyze use by minority populations. Given this deficit, NCCAM made the decision to fund a large national survey of complementary therapies. The NHIS was chosen as the vehicle for the complementary medicine surveys because of its large sample size and its oversampling of non-Hispanic black and Hispanic persons (and Asian persons beginning in 2006).

Analyses of the 2002 and 2007 surveys have resulted in more than 100 publications in peer-reviewed journals [2]. It is expected that the 2012 supplement will have the same significant impact on research and policy-making communities as did the 2002 and 2007 versions, which provided the most comprehensive nationally representative data on complementary medicine use available for the United States. Although many research priorities and specific content identified during the development of the 2002 questionnaire remained as part of the 2007 and 2012 surveys, each questionnaire contained unique topics. For example, the 2012 survey included a series of items related to using complementary therapies for wellness-related reasons. NCCAM decided to include this topic during development of its third strategic plan [3], which placed an emphasis on use of complementary therapies to promote health, wellness and well-being.

This article describes the multi-faceted and largely qualitative approach taken to develop the three questionnaires, particularly detailing the results of expert input and testing of cognitive salience and clarity of questionnaire items. This article will guide researchers and policy makers who plan to use data generated from the surveys, as well as inform quantitative assessments of questionnaire validity and of the research value provided by the resulting datasets.

Overall development process

The development of the three NHIS complementary medicine questionnaires was not performed within the framework of a planned research study. Instead, questionnaire design and development generally followed steps documented in the literature [4-8] and in an NCHS report on questionnaire design [9]. The NCHS protocol included four phases: 1) planning, 2) exploratory studies, 3) developmental methods, and 4) testing methods. Planning is particularly important for government surveys which generally have tight timeframes, require multiple clearances, and may involve staff from multiple organizations. For each questionnaire, development began 18–24 months prior to data collection in collaboration with survey methodologists from NCHS. The second phase, exploratory studies, is useful for identifying the response process and response problems unique to a given set of questions. This step applies to specific aspects of complementary medicine modalities such as the relationship between self-identity and use of nonvitamin, nonmineral dietary supplements (NVNMDS) [10] discussed later in this article. The third phase, developmental methods, involves iterative rounds of questionnaire evaluation through participant interviews as the main approach for discovering problems with questionnaires. Although the number of respondents need not be large (20–50 respondents), ideally they should differ widely in demographic characteristics to maximize generalizability. The final phase described in the NCHS report, testing methods, refers to large-scale field testing. Although budget constraints did not allow for large-scale field pre-testing, input from field staff was solicited prior to fielding and resultant modifications were made.

Development of the three NHIS questionnaires applied the definition of complementary medicine established by NCCAM: “a group of diverse medical and health care systems, practices, and products not generally considered part of conventional medicine” [1]. Although the specific therapies included varied slightly across survey years, the following were included in all three questionnaires: acupuncture, Ayurveda, biofeedback, chelation therapy, chiropractic care, energy healing therapy, hypnosis, massage, naturopathy, NVNMDS, homeopathic treatment, diet-based therapies, yoga, tai chi, qi gong, and meditation and other relaxation techniques. The 2002 questionnaire collected complementary medicine use for adults only, whereas the 2007 and 2012 included children and adults. All three surveys were approved by the NCHS Research Ethics Review Board.

Specific steps detailed in this article include: (a) literature reviews, (b) feedback from scientific and practitioner communities, (c) expert input and workshops, and (d) participant interviews (cognitive testing). Additionally, for the 2012 child questionnaire, quantitative analysis of data from the 2007 child complementary medicine questionnaire (frequencies analysis of prevalence of therapies by age, race/ethnicity, disease status, and use for health conditions) and focus groups with health professionals and the lay public were conducted. Although these steps took place sequentially within the overall development process (literature review, then expert panels, then draft initial questions, then cognitive testing, then refinement), the synthesis of these components to yield the final content was an iterative process. For example, after the expert panels, during the drafting of initial questionnaire items, we often discussed a particular series of questions with the expert panel member who focused the corresponding discussion during the meeting. Likewise, while refining the questionnaire based on results of cognitive testing, we often referred back to notes from workshops, consulted with expert panel members, or sought additional research articles to decide on the most appropriate solution. Specific steps used in the development of each questionnaire varied (Table 1) and are described in discrete terms below.

Literature reviews

The goal of each literature review was to identify previous surveys on complementary medicine use in order to help determine which therapies to include in the NHIS questionnaire. These surveys were reviewed by the expert panels when making recommendations to NCCAM and NCHS staff on what complementary therapies should be included in the NHIS. Literature reviews for all three surveys employed the search paradigm utilized by Wootton and Sparber [11]. Development of the 2002 questionnaire began by reviewing smaller-scale national and regional surveys identified by Wooton and Sparber through January 5, 2001. Development for the subsequent surveys examined U.S. national or regional surveys on complementary medicine that were published after January 5, 2001. Only U.S. surveys were considered during the literature review since the terminology, definitions and regulations vary considerably from country to country. For instance, although in the U.S. dietary supplements are regulated as food-based in the Dietary Supplement Health and Education Act [12], in other countries, such as Germany, they are often regulated as drugs [13] or, as in Japan, covered by national health insurance [14].


The systematic review by Wooton and Sparber [11] identified seven population-based surveys published in peer-reviewed journals through January 5, 2001. Of these, five reported data from national population-based surveys on complementary medicine use in the U.S. [15-19] and two reported data from state-level population-based surveys – Florida [20] and South Carolina [21]. The report by Druss and Rosenheck [19] was based on data from the 1996 Medical Expenditures Panel Survey supported by the U.S. Agency for Healthcare Research and Quality. These seven surveys served as a starting point by the expert panel for identifying content to be added to the 2002 NHIS. Item wording from these surveys was often changed to be consistent with the NHIS format.


In preparation for the 2007 NHIS complementary medicine survey an updated search of the scientific literature published up to January 2006 was conducted, and five additional national population-based surveys in adults [22-26] and one state-level - Michigan [27] were identified. Two of these were initial reports from the 1999 and 2002 NHIS surveys [22,24]. When reviewing the four non-NHIS surveys, particular attention was paid to complementary approaches not included in the NHIS surveys, as well as to reasons individuals said they used complementary therapies. Among the complementary therapies included on these surveys but not on the 2002 NHIS were traditional healers and movement therapies, both of which were added to the 2007 survey.


From published reports through January 2011, the search update for the 2012 NHIS identified three new national-population based surveys [28-30], three state-level population-based surveys – California [31], Hawaii [32], and Minnesota [33] - and one survey of active duty military personnel in the U.S. Navy and marine corps [34]. These were reviewed for complementary approaches not contained in the 2007 NHIS, as well as wellness-related reasons individuals used these therapies. No previously unidentified complementary therapies were found in these surveys.

In addition, for the 2012 questionnaire, literature searches were employed specifically to inform questionnaire design related to reasons and motivations for using complementary therapies, such as for wellness or well-being, as well as questions asking about insurance coverage [35-40]. PubMed and Google Scholar were searched, and additional references were suggested by outside experts attending a workshop on wellness (Section Wellness workshop) and members of a think tank (Section Research topic think tank) to identify research priorities for the 2012 questionnaire. Wellness-related reasons identified in the literature for why individuals use complementary therapies include physical, spiritual, and emotional well-being, nutrition and lifestyle, and to avoid pharmaceuticals [35-40].

To identify additional dietary supplements for inclusion in the 2012 NHIS, dietary supplement sales data were reviewed [41]. Among the many products added to the NHIS survey were Acai, bee pollen and other bee products, digestive enzymes and methylsulfonylmethane (MSM). In total, 119 nonvitamin, nonmineral dietary supplements were queried in the 2012 NHIS versus 44 in the 2007 NHIS.

Feedback from scientific and practitioner communities

Both the 2002 and 2007 surveys received considerable attention by the public, clinical and research communities. Staff at NCCAM received unsolicited feedback from complementary medicine researchers and practitioner organizations that was used to inform the development of the 2007 and 2012 surveys. Much of this feedback came during meetings held for other purposes and was not collected systematically. Because this feedback was unsolicited and non-systematic, data was not kept on who provided specific feedback and when.


Based on unsolicited feedback on the 2002 questionnaire and updated literature reviews, the following changes were made for the 2007 survey: 1) Addition of questions on complementary medicine use by children and the condition being treated by these therapies; 2) addition of out-of-pocket expenditures on complementary therapies; 3) addition of movement therapies (Feldenkrais, Alexander technique, Pilates, Trager psychophysical integration); 4) addition of traditional healers to better capture complementary medicine use in Hispanic and Native American populations; 5) deletion of the megavitamin section; 6) addition of questions to find out why some adults did not use five types of complementary therapies - acupuncture, chiropractic or osteopathic manipulation, NVNMDS, yoga, and meditation - chosen based on the most commonly used therapies from each of the major categories of complementary medicine as defined by NCCAM at the time of questionnaire development [1]; 7) deletion of folk medicine; and 8) addition of questions about the relationship between complementary and conventional treatments (adult version only). Content for 2002, 2007, and 2012 questionnaires are shown in Table 2. Other changes, such as the addition of questions on vitamins and minerals, and an expanded section on NVNMDS, grew out of consultations with other parts of the NIH.


Based on unsolicited community feedback about the 2007 survey several items were modified for the 2012 version. For example, follow-up questions on yoga were restricted to only those using meditation and/or breathing exercises as a component of yoga. Additionally, the list of NVNMDS was expanded beyond that of 2007.

Expert panels and workshops

Expert panels were convened for the development of the 2002 and 2012 surveys, but not for the 2007 survey. Panel members were invited based on their published work and expertise, and were considered experts in their own field. No sampling method was utilized. For each of the three in-person panels described below an open-dialogue process was used, with iterative summaries of high level input and priorities set throughout the meetings. The 2002 panels consisted of half-day phone calls and numerous email exchanges, while the 2012 panels were day-long meetings. Each meeting began with introductions, background presentations and an explanation of goals and objectives for the session. Input was solicited from each member of the panel and careful attention was given to keep discussions focused and on-target. Time was allocated at the end of each meeting to reach consensus on conclusions discussed throughout the meeting. For the in-person 2012 panels, scientific notetakers were utilized to ensure thorough documentation.


In February 2001 a panel was convened made up of 13 national experts in survey design and complementary medicine research and practice, along with NCCAM and NCHS staff, to prioritize content for the 2002 questionnaire. A second panel made up of 13 experts in women’s health, mental health in minority populations, complementary medicine use by African Americans and Native Americans, use of Chinese medicine, and cultural relevance was assembled to review draft questionnaire items. Table 3 shows a complete listing of expertise on each panel. The second panel did not convene as a group, but expert opinion was solicited in writing from each individual member related to the person’s area of expertise and additional modifications were made based on this input.

The first expert panel suggested specific therapies to include in the 2002 NHIS. The expert panel also identified four areas of research priority: 1) Prevalence of complementary medicine use; 2) purpose of complementary medicine use (whether for a specific condition, general wellness, or both); 3) disclosure of complementary medicine use to conventional providers; and 4) inventory of natural supplements and prescription medications (not fielded due to logistical reasons). The highest priority was to identify the specific conditions or diseases individuals report using complementary therapies to treat, which is distinct from assessing the prevalence of complementary medicine use among individuals who experience one or more health conditions. Analysis of this later construct is possible by linking the NHIS complementary medicine questionnaires to other NHIS core questionnaire data files. Additional conditions and diseases hypothesized to correlate with complementary medicine use were added to those already in the NHIS core questionnaire. The second panel identified ways to improve the cultural sensitivity of the questionnaire. For example, the use of the word “traditional” to refer to Western medical practices was replaced with “conventional” based on the recommendation from a cultural expert concerned that “traditional” has different meanings to individuals with regard to various culturally related practices.

Based on review of the literature and input from the panels of outside experts, a questionnaire was constructed, edited and refined by NCHS and NCCAM staff. Underlying this process was a set of criteria for questionnaire development identified from the literature [42-44] (Table 4). Once the questionnaire was constructed, the expert panels were given an opportunity to provide additional feedback. The draft questionnaire was also circulated to experts throughout the NIH for comment.

Wellness workshop

As evidenced in NCCAM’s strategic plans, the Center’s priorities have evolved during the past decade from a focus on treatment of disease to symptom management and promotion of optimal health [1,3]. In August 2009, NCCAM staff sponsored a one-day workshop on how to best capture the concept of wellness and optimal health via survey items. Twenty-seven individuals participated in the workshop including behavioral scientists from universities and non-profit organizations, as well as representatives from NCCAM and other parts of NIH, CDC, and the Food and Drug Administration (Table 3). Because it was the sole internationally accepted definition at the time, NCCAM utilized the World Health Organization’s definition of wellness to generate discussion: “wellness is the optimal state of health of individuals and groups. There are two focal concerns: the realization of the fullest potential of an individual physically, psychologically, socially, spiritually, and economically, and the fulfillment of one’s role expectations in the family, community, place of worship, workplace and other settings” [45]. In preparation for the workshop, the planning committee identified criteria to help guide discussion [46-56].

Workshop attendees discussed several topics including how best to define wellness and differences between wellness and well-being. Discussion at the workshop focused on psychological, physical, spiritual, and social measures of wellness, as well as the relationship between wellness and use of complementary therapies. Workshop results guided the development of wellness items for the 2012 survey, particularly the interviewer protocol for cognitive testing of the wellness concept.

Interviewer protocols are guides to assist in the conduct of qualitative interviews and generally contain verbal probing and follow-up questions to aid in providing some standardization across interviews [57]. The interviewer protocol used for the testing of the wellness concept contained questions to elicit reasons or motivations for using complementary therapies without presupposing a “treatment” reason and respondents were asked about changes noted in “mind or body” when using the therapy (see section Concepts of wellness and well-being).

Research topic think tank

In September 2010 NCCAM staff convened a panel of experts to review the previous two NHIS complementary medicine questionnaires and identify items to retain as well as key research areas not previously covered. The panel was comprised of 24 individuals with expertise in health services research, minority health, economics, and psychometrics, as well as complementary therapies (Table 3). Based on feedback from the research topic think tank, an extensive set of questions was included in the draft questionnaire about insurance coverage and out-of-pocket expenses. This draft set of questions captured detailed information about deductible amounts, number of visits and other dollar amounts covered by insurance, and whether the respondent’s complementary medicine use was reduced due to limited insurance coverage. This section was ultimately pared down due to complexity and time restraints (see Section Draft questionnaire).

Members of the think tank also recommended that deep breathing be removed as an independent intervention in the survey given the wide variability in how participants interpret the question as identified during cognitive interviews. Instead, queries on the use of deep breathing exercises were imbedded as follow-up questions for participants using yoga, tai chi, qi gong, meditative techniques, hypnosis and biofeedback. Finally, the think tank recommended that the survey provide some specificity on the types of meditation being used. Thus, instead of asking a global question as to whether a participant used meditation, the 2012 survey asked about three broad classes of meditation identified in an Agency for Healthcare Research and Quality systematic review: mantra, mindfulness and spiritual [58].

NCCAM staff members reviewed successive drafts for face validity and item phrasing, and an NCCAM staff member trained in cognitive interviewing techniques administered the draft questionnaire to nine NCCAM staff members with various cultural and academic backgrounds to better refine the questionnaire before it underwent cognitive testing with volunteers from the general public.

Cognitive interviewing

Cognitive interviewing, a technique to learn respondents’ thought processes as they answer survey questions, has become standard practice for survey development [59]. The aim is to uncover how respondents interpret the intent and meaning of survey questions, and whether or not these match those of the researcher. The interview generally involves one interviewer and one respondent, usually takes between 30 to 90 minutes, and respondents are often compensated for their time. The interviewer will read the draft questions to the respondent and then use probing techniques to determine how the respondent is interpreting the questions and what the respondent is thinking while answering. The interviews were videotaped and their verbal content entered into Q-notes, a software system created by NCHS’s Questionnaire Design Research Laboratory (QDRL) staff to organize and help with the analysis of cognitive interview data. Interpretations of key terms and concepts in each question were examined and compared among participants in order to identify inconsistent interpretations. Ideally, interviewing continues until “saturation” is reached, that is, no new data are emerging and all the concepts are well defined and explained [60]. However, because the NHIS must begin data collection in January each year, real-world deadlines did not allow for saturation for all sections of the questionnaire; rather, cognitive interviewing focused mainly on new topic areas added in each survey year, or existing sections that were substantially expanded (e.g. NVNMDS in 2007).

After receiving IRB approval, QDRL staff used a variety of recruitment methods to obtain volunteers for cognitive testing including advertisements in the Washington Post and alternative medicine magazines, flyers placed at various co-ops and yoga studios, and the offices of various providers of complementary therapies. Informed consent was obtained from all participants prior to the start of the interview. For each survey year, a total of 25–48 in-depth semi-structured or open-ended interviews were conducted with volunteers from the general public with a range of socio-demographic characteristics (Table 5). During these interviews, questionnaire concepts were tested for cognitive salience and clarity. Possible actions taken as a result of cognitive testing included accepting the original question, accepting the original question with minor edits, accepting the original question with major edits, dropping the original question, and writing a new question (Table 6). The questionnaire development process was fluid and iterative so that changes were made and the questionnaire re-tested throughout the development process. Table 7 provides additional examples of problems found during cognitive review of all three survey questionnaires and the resulting modification made in order to reduce response error.


Twenty-five in-depth semi-structured interviews were conducted; except for chelation therapy, collectively, interviewees had used every complementary medicine identified by literature review and outside experts for inclusion in the questionnaire. Two main findings quickly emerged: 1) modalities were practiced differently with respect to seeing a practitioner or using on one’s own; and 2) many of the initial modalities tested were not discrete and overlapped with other modalities tested. To mitigate the first problem, NCHS staff, with input from the expert panel, identified 10 therapies that are mainly practitioner-based. For these therapies, the scope was narrowed to ask “Have you ever seen a provider for…” rather than “Have you ever used…” This better matched the way respondents described using complementary therapies during interviews. To reduce overlap among therapies, “Chinese medicine” was dropped because it was found to encompass many of the more specific therapies such as acupuncture and NVNMDS. In subsequent versions of the questionnaire, “home remedies” was tested as a self-care therapy and was deleted because it was found to mean different things to different respondents, as well as overlap with NVNMDS, vitamins, and homeopathy. Ultimately, a wide range of modifications and restructuring took place based on the results of cognitive testing including: 1) practitioner-based modalities were separated from self-use modalities; 2) similar modalities were consolidated (e.g. yoga, tai chi, and qi gong were combined) or deleted (e.g. Chinese medicine); 3) questions that lacked meaningful responses or consistent interpretations were deleted (e.g. age of first use, “bad reactions,” and “home remedies.”) [61].


Thirty-two cognitive interviews were conducted on seven different versions of the 2007 draft questionnaire. Results from testing reaffirmed two issues that surfaced during cognitive testing of the 2002 questionnaire. The first relates to home remedies, which was re-tested for the 2007 questionnaire as a practitioner-based therapy, with the goal of capturing visits to complementary medicine practitioners in minority and ethnic populations. However, testing revealed that respondents often do not consider home remedies to be practitioner-based, even when the question text includes the phrase “Have you ever seen a practitioner for…” Although respondents spoke of both home remedies and folk medicine as practices passed down through the family, as found during cognitive interviewing for the 2002 questionnaire, the two terms meant different things to different participants and were therefore deleted. Second, heavy users of complementary therapies had difficulty recalling how old they were when they first used a particular therapy even when provided age categories. Because testing showed that estimates would be less reliable for long-time users of complementary therapies compared to recent users, the question was deleted [62].

In addition to these findings, cognitive testing also revealed that the question on “deep breathing exercises” produced numerous false positive responses. Many respondents had very little knowledge of this technique and essentially guessed at its meaning. One respondent said yes to having used deep breathing because “this is when you let out a sigh of being home to get your second wind to cook dinner.” [62]. To try to mitigate this problem, a definition was added to the term, but it was ultimately dropped from the 2012 survey based on Think Tank recommendations (Section Research topic think tank). Particular attention was also given to three sections that were new and/or expanded from the 2002 questionnaire (see below).

Child section

For the 2007 NHIS, NCCAM expanded the questionnaire to include a limited set of items on complementary medicine use by children, a topic that had not been previously studied on a national level. The NHIS child section is answered by proxy, usually the child’s parent. As expected, testing revealed that parents do not always know the answers to items on motivations and behaviors related to their children, such as whether a child prays for his or her health. Questions on motivations and prayer were, therefore, not asked in the child section.

Mineral and NVNMDS sections

Based on interest and funding by the NIH Office of Dietary Supplements, the 2007 questionnaire included expanded sections on vitamin and mineral supplements and NVNMDS. These sections collected detailed follow-up information for up to two specific vitamins or minerals or NVNMDS supplements and an expanded list of reasons for using these. Although the sections on vitamins and minerals and NVNMDS were identical, cognitive testing revealed problems unique to the NVNMDS items [10,62]. First, it became apparent that respondents lacked a consistent, agreed-on definition of an herbal supplement. Respondents held wide-ranging and varying definitions of NVNMDS and efforts to craft a definition that met researchers’ criteria and was understandable to respondents fell short. Ultimately, the best approach was to simply ask respondents about taking specific supplements listed on a flashcard. This made the survey response task concrete and less reliant on shared definitions, although, the tradeoff was the loss of information about supplements not listed on the card. Second, some respondents viewed themselves as having or not having a “complementary medicine identity” and this perception affected the way they interpreted and answered questions, such that response error occurred even when the term “natural herbs” was understood as intended. To mitigate this, the description of “a typical herb user’s” lifestyle was removed from the lead-in statement [10,62].

Perceptions of complementary therapies

A section on perceptions of complementary therapies was explored for the 2007 questionnaire. The questions were to capture respondents’ perceptions about the scientific evidence, safety, effectiveness, and holistic aspects of complementary therapies, as well as items concerning locus of control and how decisions are made to use or not use complementary therapies. The initial plan was to ask a long series of items about each individual complementary therapy, but time constraints made this unfeasible. QDRL staff then tested a version asking about complementary therapies as a whole, and determined that respondents’ views vary by specific therapies such that they could not arrive at a single answer. Ultimately the section was drastically reduced and embedded within each therapy rather than as a stand-alone section.

Concepts of wellness and well-being

In fall 2010, staff from NCHS’s cognitive lab performed inductive interviews with 24 volunteers to learn more about individuals who use complementary therapies to promote their wellness, general health and well-being. The interviewer protocol contained questions to elicit reasons or motivations for using complementary therapies without presupposing it was to treat a specific condition. In particular - and in keeping with the published literature [35,36,38-40] - respondents were asked about changes noted in “mind or body” when using complementary therapies. Interviewing focused on how respondents who identified “wellness” and “well-being” as their main reason for using complementary therapies understood and conceptualized these terms. Because the intent was to learn more about a general concept or concepts rather than test draft questionnaire items, a more open-ended approach was utilized. When analyzing and interpreting the interviews, researchers aimed for “thick description,” that is, to provide a deeper, more nuanced understanding of the ways that individuals interpret events and phenomena in their social worlds. New participants were recruited and interviewed until no new conceptual insights were identified (saturation). Testing revealed that although respondents reported benefits to their physical health, they primarily associated complementary medicine use with emotional and psychological benefits. The most frequently mentioned benefits were the positive effect on one’s mood, stress level, and overall mental and emotional health. In addition to these emotional and psychological benefits, users also described many different physical benefits, most often mentioned in conjunction with their use of acupuncture, massage, or chiropractic care, and in particular, to help treat and eliminate back pain. The primary components of respondents’ conceptualizations of well-being included being centered, finding balance, and not simply being disease free, but optimally healthy [63].

Based on these qualitative research findings and related literature, a series of items for the 2012 questionnaire was drafted by NCCAM and NCHS staff to capture wellness-related reasons for using complementary therapies. Examples include “because it is natural,” and “because it focuses on the whole person, mind, body, and spirit.” Additionally, some respondents mentioned having benefited from wellness-related outcomes as a result of using complementary medicine. Hence, the questionnaire includes several items asking if using complementary medicine led to outcomes such as “improve your overall health and make you feel better,” “motivate you to eat healthier,” and “help to reduce your stress level or to relax” (Table 8).

Draft questionnaire

Twenty-four cognitive interviews were conducted on several different versions of the 2012 draft questionnaire in an iterative process during which revisions were made and retested. Testing focused particularly on two sections: 1) A large set of questions about insurance and payment; and 2) the expanded list of reasons for using the therapy, especially for general wellness and wellbeing. Cognitive testing and focus group feedback (see below) determined that the questions on insurance coverage of, and payment for, complementary therapies were too complex to adequately capture valid information in the limited survey time available. They were replaced with a simple question asking if health insurance covered any of the costs; if yes, did it cover all, some, or none, and how much the respondent paid in total or per visit for the particular therapy [64].

CAMHI conducted focus groups and quantitative analyses

To provide additional input on the 2012 questionnaire, investigators from the Child and Adolescent Health Measurement Initiative (CAMHI) based at the Oregon Health & Sciences University performed quantitative analysis of data from the 2007 child complementary medicine questionnaire including frequencies analysis of the prevalence of each therapy by age group (0–5, 6–11, 12–17), race/ethnicity (Hispanic, white non-Hispanic, black non-Hispanic, other non-Hispanic), disease status, and whether the child used any complementary medicine for reported health conditions [65]. Also explored was how these prevalence rates varied depending on which definition of complementary medicine was used (e.g., whether vitamins and minerals were included in the definition). These analyses led to the identification of sample size deficiencies, appropriateness of given therapies among different age groups, and construct validity of items related to complementary medicine use [65]. Resulting modifications to the 2012 survey included: 1) restricting the child survey to children aged 4 and older; 2) for both the child and adult questionnaires, follow-up questions were not asked for individual traditional healers, movement therapies, and special diets. Rather, individual complementary approaches within each category were grouped to increase cell sizes; and 3) follow-up questions on the reasons children or adults used Ayurveda and chelation therapy were not asked due to small cell sizes on earlier surveys.

In addition, CAHMI organized and conducted key informant interviews and focus group discussions with 21 volunteers from the Family Voices Network [66], as well as from the CAHMI network of expert advisors, pediatric provider partners and family leaders partnering on an NIH/NCCAM grant related to pediatric use of complementary therapies. Informed consent was obtained from all interview and focus group participants. Informant interview and focus group recommendations included: (1) further refinement of the cost and insurance questions, (2) inclusion of child-specific health and well-being reasons for using complementary medicine, (3) the addition of questions about complementary medicine information-seeking, (4) communication with other health care providers about complementary medicine use, and (5) the addition of an item on satisfaction of complementary medicine use for a given reason for use [65]. Although CAMHI-conducted focus groups and quantitative analyses focused on the child component, the provided input contributed to refinement of both the 2012 child and adult questionnaires.


In this paper, we detail the multifaceted, largely qualitative, iterative process based on literature reviews, expert panel input, and cognitive interviewing methods used to develop and refine the NHIS complementary medicine questionnaires. For the 2012 questionnaire, we also used inductive cognitive review to conceptualize the concept of “well-being” and focus groups with family leaders and subject matter experts to further refine the child questionnaire. By soliciting feedback from subject matter experts, family leaders, and the lay public, we improved the likelihood that the questionnaire items will be understood and interpreted as intended. We describe the systematic evaluation of questionnaire items, the decision-making processes, and rationale for how items were retained, modified, or deleted. By providing this in-depth description of how these influential national surveys were developed, we hope to enable policy makers and researchers to better judge the content validity and utility of the questionnaires and the resultant findings. We also illustrate how the questionnaires have adapted over time to new scientific information and societal shifts so that modifications can be better understood by data users.

There are several limitations to the approach we used to develop the NHIS complementary medicine questionnaires. First, due to time constraints, in many cases, we were not able to continue cognitive testing until we reached saturation and may have missed some problematic issues with the questionnaires that could potentially reduce validity or generalizability. Second, although great care was taken to select individuals for cognitive testing who differed widely in demographic characteristics, the volunteers may not have been representative of the U.S. population in all important respects, which could limit generalizability of the testing results. Third, we did not include lay people on the expert panels convened to determine content areas for the questionnaires and thus could have missed salient topics of particular interest to users of complementary medicine. However, lay public were included in the CAHMI-organized focus groups that identified additional survey topics not prioritized by the think tank experts. Fourth, there is no universally accepted definition of what constitutes a complementary medicine approach. As a result, there is no definitive list of which therapies should be included in the questionnaire. For instance, whether prayer for health reasons should be considered within the realm of complementary therapies has received considerable discussion [67,68]. For these surveys, we have been as inclusive as possible given time constraints, thus allowing greatest flexibility to data users. Fifth, the NHIS has limitations common to all cross-sectional studies collecting participant-reported data: it is not possible to make inferences about cause and effect, and data are not independently verified. This latter limitation may be particularly important for the section on out-of-pocket expenditures in which respondents likely provide estimates versus exact amounts. Finally, in the 2007 survey and for some respondents in the 2012 survey, although the cost per visit or purchase was collected as continuous data within the range of $0-$500 per expenditure, the number of visits or purchases per person was collected as categorical data (based on recommendations from cognitive testing). This mixing of continuous and categorical data makes the computation of actual expenditures on complementary therapies very complex [69]. For the 2012 survey, we hope to provide better guidance and technical documents explaining how to analyze the expenditure data.

The three NHIS complementary medicine questionnaires were developed to assess the U.S. public’s use of complementary health approaches and factors thought to be associated with this use. It is our expectation that quantitative analyses on the validity and reliability of the questionnaires will supplement the qualitative development approaches describe here, and will inform future survey design and provide guidance to users of existing survey data. Through changes over time, the NHIS complementary medicine surveys have attempted to meet the challenge of providing information to the NCCAM/NIH, national academies, researchers, clinicians, and policy makers. The NHIS complementary medicine datasets remain the primary national source of data on complementary medicine use in the United States.


NHIS: National Health Interview Survey; CDC: Centers for disease control and prevention; NIH: National institutes of health; NCCAM: National center for complementary and alternative medicine; NVNMDS: Nonvitamin, nonmineral dietary supplements; NCHS: National center for health statistics; QDRL: Questionnaire design research laboratory; CAMHI: Oregon Health & Sciences University’s Child and Adolescent Health Measurement Initiative (CAMHI).

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

BJS participated in survey design, data analysis, and drafted substantial portions of the manuscript. CDB participated in survey design, data analysis, and drafted substantial portions of the manuscript. CG participated in data analysis and revised substantial portions of the manuscript. RLN participated in survey design, and drafted substantial portions of the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:


We thank the participants in NCCAM’s September 2010 Think Tank for their invaluable insights and guidance. We thank D. Lee Alekel, Alexandra Belcher, Kirsten Bondalapati, Laura Lee Johnson, Wendy Weber, and Jennifer Peregoy for their helpful feedback on this article. We thank Cambria Wilhelm for her excellent assistance.

BJS and RLN were employees of the federal government during the planning, analysis, and writing of this manuscript and performed their work as part of their official duties. No outside financial support was provided. CDB’s work on this project was supported by NIH/NCCAM grant 1 R21 AT004960. CG was employed by the federal government during the planning and analysis of this manuscript and received no financial support during the writing of the manuscript. The findings and conclusions of this report are those of the authors and collaborators and do not necessarily represent the views of the National Institutes of Health, National Center for Complementary and Alternative Medicine, the Oregon Health & Sciences University, School of Medicine, or the Palo Alto Medical Center, Health Policy Research Institute.

National Center for Complementary and Alternative Medicine Expanding Horizons of HealthcareFive-Year Strategic Plan 2001–2005Year: 2000Bethesda, Maryland, USA: US Department of Health and Human Services, National Institutes of Health, National Center for Complementary and Alternative Medicine NIH Publication number 01–5001.
Publications Using NHIS CAM Data
National Center for Complementary and Alternative Medicine Exploring the Science of Complementary and Alternative MedicineThird Strategic Plan: 2011–2015Year: 2011Bethesda, Maryland, USA: US Department of Health and Human Services, National Institutes of Health, National Center for Complementary and Alternative Medicine NIH Publication number 11–7643, D458.
Veale B,Questionnaire design and surveysAust Fam PhysicianYear: 1998134995029648317
Passmore C,Dobbie AE,Parchman M,Tysinger J,Guidelines for constructing a surveyFam MedYear: 20021328128612017142
Rattray J,Jones MC,Essential elements of questionnaire design and developmentJ Clin NursYear: 200713234243 Review. 17239058
Cella D,Riley W,Stone A,Rothrock N,Reeve B,Yount S,Amtmann D,Bode R,Buysse D,Choi S,Cook K,Devellis R,DeWalt D,Fries JF,Gershon R,Hahn EA,Lai JS,Pilkonis P,Revicki D,Rose M,Weinfurt K,Hays R,PROMIS Cooperative GroupThe Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks:2005–2008J Clin EpidemiolYear: 2010131179119410.1016/j.jclinepi.2010.04.01120685078
DeWalt DA,Rothrock N,Yount S,Stone AA,PROMIS Cooperative GroupEvaluation of item candidates: the PROMIS qualitative item reviewMed CareYear: 200713Suppl 51221
Vital and Health StatisticsQuestionnaire Design in the Cognitive LaboratoryYear: 198913Hyattsville, Maryland, USA: NCHS DHHS Publication No. (PHS) 89–1076.
Willson S,Stussman B,Maitland A,Nahin RL,Role of self-concept in answering survey questions on complementary and alternative medicine: challenges to and strategies for improving data qualityJ Altern Complement MedYear: 2009131319132510.1089/acm.2008.058019954343
Wootton JC,Sparber A,Surveys of complementary and alternative medicine: part I. General trends and demographic groupsJ Altern Complement MedYear: 20011319520810.1089/10755530175016430711327525
The Dietary Supplement Health and Education Act of 1994 (Public Law 103–417, DSHEA)
Ajazuddin,Saraf S,Legal regulations of complementary and alternative medicines in different countriesPharmacogn RevYear: 20121315416023055642
Watanabe S,Imanishi J,Satoh M,Ozasa K,Unique place of Kampo (Japanese traditional medicine) in complementary and alternative medicine: a survey of doctors belonging to the regional medical association in JapanTohoku J Exp MedYear: 200113556310.1620/tjem.194.5511556734
Eisenberg DM,Kessler RC,Foster C,Norlock FE,Calkins DR,Delbanco TL,Unconventional medicine in the United States. Prevalence, costs, and patterns of useN Engl J MedYear: 19931324625210.1056/NEJM1993012832804068418405
Paramore LC,Use of alternative therapies: estimates from the 1994 Robert Wood Johnson Foundation National Access to Care SurveyJ Pain Symptom ManageYear: 199713838910.1016/S0885-3924(96)00299-09095565
Astin JA,Why patients use alternative medicine: results of a national studyJAMAYear: 1998131548155310.1001/jama.279.19.15489605899
Eisenberg DM,Davis RB,Ettner SL,Appel S,Wilkey S,Van Rompay M,Kessler RC,Trends in alternative medicine use in the United States, 1990–1997: results of a follow-up national surveyJAMAYear: 1998131569157510.1001/jama.280.18.15699820257
Druss BG,Rosenheck RA,Association between use of unconventional therapies and conventional medical servicesJAMAYear: 19991365165610.1001/jama.282.7.65110517718
Burg MA,Hatch RL,Neims AH,Lifetime use of alternative therapy: a study of Florida residentsSouth Med JYear: 1998131126113110.1097/00007611-199812000-000069853724
Oldendick R,Coker AL,Wieland D,Raymond JI,Probst JC,Schell BJ,Stoskopf CH,Population-based survey of complementary and alternative medicine usage, patient satisfaction, and physician involvement. South Carolina Complementary Medicine Program Baseline Research TeamSouth Med JYear: 20001337538110798505
Ni H,Simile C,Hardy AM,Utilization of complementary and alternative medicine by United States adults: results from the 1999 National Health Interview SurveyMed CareYear: 20021335335810.1097/00005650-200204000-0001112021691
Mackenzie ER,Taylor L,Bloom BS,Hufford DJ,Johnson JC,Ethnic minority use of complementary and alternative medicine (CAM): a national probability survey of CAM utilizersAltern Ther Health MedYear: 200313505612868252
Barnes PM,Powell-Griner E,McFann K,Nahin RL,Complementary and alternative medicine use among adults: United States, 2002Adv DataYear: 20041311915188733
Upchurch DM,Chyu L,Use of complementary and alternative medicine among American womenWomens Health IssuesYear: 20051351310.1016/j.whi.2004.08.01015661582
Honda K,Jacobson JS,Use of complementary and alternative medicine among United States adults: the influences of personality, coping strategies, and social supportPrev MedYear: 200513465310.1016/j.ypmed.2004.05.00115530580
Rafferty AP,McGee HB,Miller CE,Reyes M,Prevalence of complementary and alternative medicine use: state-specific estimates from the 2001 Behavioral Risk Factor Surveillance SystemAm J Public HealthYear: 2002131598160010.2105/AJPH.92.10.159812356602
Carlson MJ,Krahn G,Use of complementary and alternative medicine practitioners by people with physical disabilities: estimates from a National US SurveyDisabil RehabilYear: 20061350551310.1080/0963828050021206216513583
Wilson KM,Klein JD,Sesselberg TS,Yussman SM,Markow DB,Green AE,West JC,Gray NJ,Use of complementary medicine and dietary supplements among U.S. adolescentsJ Adolesc HealthYear: 20061338539410.1016/j.jadohealth.2005.01.01016549299
Bair YA,Gold EB,Zhang G,Rasor N,Utts J,Upchurch DM,Chyu L,Greendale GA,Sternfeld B,Adler SR,Use of complementary and alternative medicine during the menopause transition: longitudinal results from the Study of Women’s Health Across the NationMenopauseYear: 200813324310.1097/gme.0b013e31813429d618090037
Hsiao AF,Wong MD,Goldstein MS,Yu HJ,Andersen RM,Brown ER,Becerra LM,Wenger NS,Variation in complementary and alternative medicine (CAM) use across racial/ethnic groups and the development of ethnic-specific measures of CAM useJ Altern Complement MedYear: 20061328129010.1089/acm.2006.12.28116646727
Harrigan R,Mbabuike N,Efird JT,Easa D,Shintani T,Hammatt Z,Perez J,Shomaker TS,Use of provider delivered complementary and alternative therapies in Hawai’i: results of the Hawai’i Health SurveyHawaii Med JYear: 200613130132 134–139, 151. 16774141
Cheung CK,Wyman JF,Halcon LL,Use of complementary and alternative therapies in community-dwelling older adultsJ Altern Complement MedYear: 200713997100610.1089/acm.2007.052718047447
Smith TC,Ryan MA,Smith B,Reed RJ,Riddle JR,Gumbs GR,Gray GC,Complementary and alternative medicine use among US Navy and Marine Corps personnelBMC Complement Altern MedYear: 200713716
McCaffrey AM,Pugh GF,O’Connor BB,Understanding patient preference for integrative medical care: results from patient focus groupsJ Gen Intern MedYear: 2007131500150510.1007/s11606-007-0302-517846846
Green AM,Walsh EG,Sirois FM,McCaffrey A,Perceived benefits of complementary and alternative medicine: a whole systems research perspectiveOpen Comp Med JYear: 2009133545
Wolsko PM,Eisenberg DM,Davis RB,Ettner SL,Phillips RS,Insurance coverage, medical conditions, and visits to alternative medicine providers: results of a national surveyArch Intern MedYear: 20021328128710.1001/archinte.162.3.28111822920
Vincent C,Furnham A,Willsmore M,The perceived efficacy of complementary and orthodox medicine in complementary and general practice patientsHealth Educ ResYear: 19951339540510159673
McGregor KJ,Peay ER,The choice of alternative therapy for health care: testing some propositionsSoc Sci MedYear: 1996131317132710.1016/0277-9536(95)00405-X8913002
Swartzman LC,Harshman RA,Burkell J,Lundy ME,What accounts for the appeal of complementary/alternative medicine, and what makes complementary/ alternative medicine “alternative”?Med Decis MakingYear: 20021343145012365485
Nutrition Business JournalSupplement Business Report 2010Year: 2010Boulder, CO: New Hope Natural Media, Penton Media, Inc
Fowler F,Improving Survey QuestionsYear: 1995Thousand Oaks, California: Sage Publications
Tourangeau R,Rips L,Rasinski K,The Psychology of Survey ResponseYear: 2000Cambridge, New York: Cambridge University Press
Meadows KA,So you want to do research? 5: questionnaire designBr J Community NursYear: 20031356257014688664
Smith BJ,Tang KC,Nutbeam D,WHO Health Promotion Glossary: new termsHealth Promot IntYear: 20061334034510.1093/heapro/dal03316963461
Diener E,Subjective well-beingPsychol BullYear: 1984135425726399758
Ryff CD,Happiness is everything, or is it? Explorations on the meaning of psychological well-beingJ Pers Soc PsycholYear: 19891310691081
Bufford RK,Paloutzian RF,Ellison CW,Norms for the spiritual well-being scaleJ Psychol TheologyYear: 1991135670
Ware JE Jr,Sherbourne CD,The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selectionMed CareYear: 19921347348310.1097/00005650-199206000-000021593914
McHorney CA,Ware JE Jr,Raczek AE,The MOS 36-item short-form health survey (SF-36). II. Psychometric and clinical tests of validity in measuring physical and mental health constructsMed CareYear: 1993132472638450681
McHorney CA,Ware JE Jr,Lu JF,Sherbourne CD,The MOS 36-item short-form health survey (SF-36). III. Tests of data quality, scaling assumptions, and reliability across diverse patient groupsMed CareYear: 199413406610.1097/00005650-199401000-000048277801
Adams T,Bezner J,Steinhardt M,The conceptualization and measurement of perceived wellness: integrating balance across and within dimensionsAm J Health PromotYear: 19971320821810.4278/0890-1171-11.3.20810165100
Hatch RL,Burg MA,Naberhaus DS,Hellmich LK,The spiritual involvement and beliefs scale. Development and testing of a new instrumentJ Fam PractYear: 1998134764869638112
Ryan RM,Deci E,On happiness and human potentials: a review of research on hedonic and eudaimonic well-beingAnnu Rev PsycholYear: 20011314116610.1146/annurev.psych.52.1.14111148302
Gomez R,Fisher JW,Domains of spiritual well-being and development and validation of the spiritual well-being questionnairePersonal Individ DifferYear: 2003131975199110.1016/S0191-8869(03)00045-X
Gallager MW,et al. The hierarchical structure of well-beingJ PersYear: 2009131025104910.1111/j.1467-6494.2009.00573.x19558444
Harris-Kojetin LD,Fowler FJ Jr,Brown JA,Schnaier JA,Sweeny SF,The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Consumer Assessment of Health Plans StudyMed CareYear: 199913Suppl 31021
Ospina MB,Bond TK,Karkhaneh M,et al. Meditation Practices for Health: State of the Research. Evidence Report/ Technology Assessment no. 155Year: 2007Rockville, MD: Agency for Healthcare Research and Quality AHRQ publication no. 07-E010.
Beatty P,Willis G,Research synthesis: the practice of cognitive interviewingPublic Opin QYear: 20071328731110.1093/poq/nfm006
Corbin J,Strauss A,Basics of Qualitative Research 3eYear: 2008Thousand Oaks, CA: Sage Publications, Inc
Miller K,Cognitive Interview Evaluation of the Complementary and Alternative Medicine Supplement for the National Health Interview Survey: Results of Interviews conducted June-July, 2001Year: 2001Hyattsville, MD: National Center for Health Statistics
Willson S,Cognitive Interviewing Evaluation of the 2007 Complementary and Alternative Medicine Module for the National Health Interview Survey. Interviews conducted February 13-March 1 and May 1-May 12 2006Year: 2006Hyattsville, MD: National Center for Health Statistics
Gray C,Chepp V,Irby A,Report on Complementary and Alternative Medicine and Well-Being Study conducted at the National Center for Health Statistics, QDRL. Interviews conducted September 1-October 6 2010Year: 2010Hyattsville, MD: National Center for Health Statistics
Gray C,Chepp V,Cognitive Interview Evaluation of the Complementary and Alternative Medicine Supplement for the National Health Interview Survey: Results of Interviews conducted February-April, 2011Year: 2011National Center for Health Statistics: Hyattsville, MD
Bethell C,Gombojav N,Stumbo S,Brown C,Blumberg S,Carle A,Newacheck PW,Using the National Health Interview Survey and Medical Expenditures Panel Survey to Assess the Use and impact of Complementary and Alternative Medicine for Children in the US: A Meta ‒Data Methods Key Issues ReportYear: 2013Portland, Oregon, USA: The Child and Adolescent Health Measurement Initiative
Family Voices Network Last accessed October 25, 2013.
Tippens K,Marsman K,Zwickey H,Is prayer CAM?J Alt Comp MedYear: 20091343543810.1089/acm.2008.0480
Kaptchuk TJ,Eisenberg DM,Varieties of healing. 2: a taxonomy of unconventional healing practicesAnnals Int MedYear: 20011319620410.7326/0003-4819-135-3-200108070-00012
Nahin RL,Barnes PM,Stussman BJ,Bloom B,Costs of complementary and alternative medicine (CAM) and frequency of visits to CAM practitioners: United States, 2007Natl Health Stat ReportYear: 20091311419634296

[TableWrap ID: T1] Table 1 

Iterative development process for complementary medicine questionnaires

Questionnaire development steps
  2002 2007 2012
 Literature review
 Feedback from scientific communities on previous questionnaires
 Expert input/panel
 Conceptual workshop
 Inductive cognitive review
 Cognitive interviews with staff members
 Cognitive interviews with volunteers from general public
 Focus groups with health leaders and subject matter experts
 Feedback from Census Bureau interviewers
 Quantitative analysis of previous child survey data     X

[TableWrap ID: T2] Table 2 

Content of 2002, 2007, and 2012 NHIS complementary medicine questionnaires

Topic 2002 Enhancements/changes for 2007 Enhancements/changes for 2012
Practitioner-based therapies included in survey
Chiropractic care replaced with chiropractic or osteopathic manipulation
Added craniosacral therapy
Rreiki no longer distinguished from other energy healing therapies
Modified list of traditional healers
Added movement therapies (Feldenkreis,
Allowed participants to report whether chiropractor or osteopathic physician provided manipulation therapy
Alexander Technique, pilates, Trager
Psychophysical integration)
Added Traditional Healers (Curandero, Espirtista,
Chelation therapy
Hierbero or Yerbera, Shaman, Botanica,
Native American Healer, Sobador)
Chiropractic care
Removed folk medicine
Energy healing therapy/Reiki
Folk medicine (such as Curanderismo or Native American healing)
Self-care therapies included in survey
Nonvitamin, nonmineral dietary supplements (NVNMDS) (35 total)
Included 44 NVNMDS on flashcard
Pared down set of questions on vitamin and mineral supplements
High dose or megavitamin therapy
Added vitamin and mineral supplements
Expanded list of NVNMDS to 21 on flashcard and 98 in lookup table (119 total)
Homeopathic treatment
Removed high dose or megavitamin therapies
Classified meditation into 3 broad categories (mantra, mindfulness, and spiritual)
Special diets (vegetarian, macrobiotic, Atkins, Pritikin, Ornish, zone)
Added South Beach to special diets
Removed breathing exercises as separate therapy but included it within other therapies (hypnosis, biofeedback, meditation, guided imagery, progressive relaxation, yoga, tai chi, qi gong)
Yoga, tai chi, qi gong
Pared down questions on prayer for your own health
Relaxation techniques (meditation, guided imagery, progressive relaxation, deep breathing exercises)
Removed zone and South Beach from special diets
Prayer for your own health
Removed prayer for your own health
Information collected for each therapy
12 month prevalence
Added lifetime prevalence
Added whether all or some of the costs were covered by health insurance
Frequency of use in past 12 months for practitioner-based therapies
Added frequency of use in past 12 months for self- care therapies
Modified questions on out-of-pocket costs to allow for different ways of reporting
Use for specific health problem or condition
Added out-of-pocket costs in past 12 months (per visit/purchase)
Cost of self-help materials expanded to include both practitioner-based and self-help therapies
Reasons for use (conventional care would not help, conventional care was too expensive, combination of conventional care and complementary medicine would help, recommended by conventional provider, thought it would be interesting to try)
Added whether also using conventional care for the same health problem as using complementary therapy
Narrowed questions on reasons for use to the 3 therapies most important for health; did not ask for chelation and Ayurveda; asked collectively for movement, traditional healers, and special diets
Added additional reasons for using complementary therapy (energy, wellness, immune function, recommended by family, friends, or coworkers)
Greatly expanded reasons for use to include wide range of wellness-related reasons, motivators, and outcomes
How much therapy helped health problem
Added how much therapy helped most important reason for using it
Importance of use of therapy
Added how much use of therapy helped the specific health problem
Disclosure to conventional provider
Whether insurance covered any of the Costs
Added importance of use of therapy
Modified disclosure to apply to personal care provider
Added reason for not disclosing use
Added sources of information about therapy
Additional information collected Numerous health conditions added to NHIS core Added additional health conditions to NHIS core
Modified list of health conditions added to NHIS core based on prevalence data from prior surveys
Collected detailed follow-up information for up to two NVNMDS
Added questions about having a personal health care provider and whether he/she is the same practitioner for complementary therapies
Expanded list of reasons for using NVNMDS and vitamin and mineral supplements
Expanded child questionnaire to be almost identical to adult; limited to children age 4+
Added reasons didn’t use certain therapies (acupuncture, chiropractic, NVNMDS, yoga, and meditation)
Added child questionnaire with subset of items from adult survey (12 month prevalence, use for specific health problem, use of dietary supplements for sports)

[TableWrap ID: T3] Table 3 

Expertise of panel members

Initial development
Minority population
Wellness workshop
Research topic think tank
(n = 13) (n = 13) (n = 27) (n = 24)
Epidemiology (n = 2)
Aging, women’s health and use of complementary therapies
Aging, dementia, and health rehabilitation (n = 4)
Pediatrics and complementary therapies (n = 2)
Health policy
Diabetes and complementary therapies
Chronic disease and health behaviors (n = 3)
Healthcare utilization and complementary therapies (n = 2)
Health psychology
Ethnic and racial minorities and complementary therapies
Developmental disabilities, and parent and child behaviors
Pain conditions. epidemiology and complementary therapies
Health services research
Health outcome research in Hispanic populations
Health education and complementary therapies
Health services research and complementary therapies (n = 2)
Minority populations and complementary therapies
Healthcare disparities
Health outcomes research
Public health, Hispanic populations, and complementary therapies
Nutrition and physical activity
Minority populations and complementary therapies
Health psychology and spirituality
Quantitative psychology and complementary therapies
Pain conditions and complementary therapies
Public health and African Americans
Human values and moral development
Health Psychology and complementary therapies
Public health (n = 2)
Public health and the environment
Patient reported outcome measures
Questionnaire design
Use of complementary therapies among African Americans
Nephrology, clinical research, and complementary therapies
Health education and complementary therapies
Use and effectiveness of complementary therapies in health and illness
Pain conditions, epidemiology and complementary therapies
Geriatrics, rural healthcare, and complementary therapies
Survey design and complementary therapies Women’s health and complementary therapies (n = 3) Pediatric psychology and illness (n = 2)
Health economics and complementary therapies
Analysis of complex datasets
Positive psychology
Complementary therapy use among racial and ethnic minorities
Quality of life research
Survey Methodology (n = 2)
Questionnaire design, survey methodology and health statistics (n = 3)
Medical Care statistics
Social and psychiatric epidemiology and social support
Naturopathy (n = 2)
Social Psychology and mental health
Sociology and qualitative research
Substance abuse
Survey design (n = 2)
Wellness, and psychological issues of dealing with chronic illness

[TableWrap ID: T4] Table 4 

Criteria used to develop items for complementary medicine questionnaires1

1 Literacy level of questions at 8th grade or below
2. Specific questions are generally easier to answer than broad ones
8. Questions covering multiple concepts are decomposed into single questions
3. Question captures what researcher intended (i.e. every respondent answering the same question)
9. Questions are designed to avoid social desirability effects
4. Question interpreted as intended by persons in a range of socio-demographic groups
10. Avoid:
   a. questions worded in the negative
5. Response categories fit the question and are non-overlapping, clear and unambiguous
   b. complex questions
   c. questions combining two items in one (double barreled)
6. Reference periods are clear and match respondent’s ability to recall
   d. jargon
   e. ambiguous terms
7. Terms and definitions are defined    f. questions that lead respondents toward a particular answer

1This criteria were identified from the literature: References [42-44].

[TableWrap ID: T5] Table 5 

Demographic characteristics of participants in cognitive interviewing, 2002, 2007, 2012

2002 (n = 25)
2007 (n = 32)
2012 (n = 48)
  Draft Q Draft Q W-being (n = 24) Draft Q (n = 24)
  Under 35


  Non-Hispanic White
  Non-Hispanic Black
  High school or below
  More than high school
  Under $20,000


Marital status
  Never married

  Currently married



  Widowed 0 2

Data unavailable.

1The total for the 2012 race/ethnicity category is greater than the number of participants for 2012 because the data were collected such to allow overlap between Hispanic and other categories.

21 participant in 2002 and 1 participant in 2007 had unknown income data.

[TableWrap ID: T6] Table 6 

Examples of actions taken based on results of cognitive review and focus groups, 2012

Question Action taken Justification
Did you see a practitioner for/use [therapy] because it was recommended by a medical doctor?
Accept original question
Cognitive testing confirmed that respondents were primarily thinking of “Western,” “mainstream” medical doctors when they responded to this question, as intended.
Original question: Did you do breathing exercises as part of [certain therapies]?
Accept original question with minor edits
Testing revealed that respondents understood “breathing exercises” differently so that some thought it included merely taking a deep breath whereas others thought it was a more formal technique. Therefore, a definition was included in the text of the question.
Final question: Did you do breathing exercises as part of [certain therapies]? Breathing exercises may involve actively controlling the way air is drawn in, or the rate or depth of breathing.
Original question: During the past 12 months, did you see practitioner for/use [therapy] for any of these reasons…”
Accept original question with major edits
This question was divided into two questions and re-worded so that only respondents using medical treatments were asked whether experiences with medical treatments influenced their use of complementary therapies, and the type of medical treatment is explicit. Cognitive testing of the original question revealed that respondents found the reference to “medical treatments” unclear. For respondents who were not receiving or using any medical treatments, these questions appeared not to be applicable. In fact, a number of respondents specifically commented that these questions were not relevant to their circumstances.
…Medical treatments were not helping you?
…Medical treatments were too expensive?
…[modality] combined with medical treatments would help you?
Revised and restructured into 2 new questions:
Did you receive any of the following medical treatments for
[condition using complementary therapy for]…”
Prescription medications?
Over-the-counter medications?
Physical therapy?
Mental health counseling?
[next question asked for respondents who said yes to any items in question above]?
DURING THE PAST 12 MONTHS, did you see a practitioner for/use [modality] for any of these reasons…
These medical treatments do not work for the health problem you want to treat or prevent?
[next question asked for respondents using prescription medications and/or over the counter medications]
Prescription medications/Over the counter medications cause side effects?
Do you currently see a practitioner for [therapy] more, less or about the same as you did one year ago?
Question dropped
Question was problematic for respondents who see a practitioner only sporadically or on an “as needed” basis. For instance, one respondent who had used homeopathic treatment in the past 12 months said she only uses it when she has a flare up and therefore found it difficult to compare her current use to her use 1 year ago. Similarly, a respondent who did not practice yoga with any regularity had difficultly answering.
Original question: During the past 12 months, did you see a practitioner for/use [therapy] because it is how you were raised?
New question written
Although the meaning is similar, this item was re-worded to better capture the way some respondents are influenced to use complementary therapies by cultural factors. When hearing the original question, several respondents hesitated and seemed to find the original wording awkward, evidenced by one respondent who said, “That sounds very odd.” The phrasing “how you were raised” suggested a more deliberate action than merely being exposed to complementary therapies at an early age. “It was part of your upbringing” better captured unintentional or cultural exposure which was the intent of the item.
Final item: During the past 12 months, did you see a practitioner for/use [therapy] because it was part of your upbringing?
General wording of "disease or health problem" found in many questions
Changing wording to "health problems, symptoms, or conditions" is more comprehensive and inclusive
DURING THE PAST 12 MONTHS, did you or another family member get information about [fill1: modality] from any of the following sources?
Question written (didn’t exist previously)
Information sources about complementary therapies was identified as an important and missing topic during focus groups
The internet?
Books, magazines, or newspapers?
Television or radio?
Scientific articles?
Health food stores?
[fill1: Not including the practitioner [fill: S.C. name] saw for [fill2: modality] DURING THE PAST 12 MONTHS, did you let [fill S.C. name]’s personal health care provider know about [fill: his/her] use of [fill3: modality]?
Question written (didn’t exist previously) Communication about complementary medicine use with other health care providers was identified as an important and missing topic during focus groups
If no, why not:…
[fill: S.C. name] was not using it at the time?
They discouraged use of it in the past?
You were worried they would discourage it?
You were concerned about a negative reaction?
You didn’t think they needed to know?
They didn’t ask?
You don’t think they know as much about it as you do?
They didn’t give you enough time to tell them?

[TableWrap ID: T7] Table 7 

Examples of response error in cognitively tested questions for 2002, 2007, and 2012 questionnaires

Questionnaire item Problem identified during cognitive testing Type of error Resolution
Have you ever used high dose or megavitamin therapy for your own health or treatment?
Many respondents who took vitamin supplements and/or a daily multi-vitamin responded “yes” even though they were not taking megavitamins.
Question divided into two parts so respondents who took any kind of vitamins had a chance to say “yes.” Second question screened out the non high-dose users.
Have you EVER seen a provider or practitioner for movement therapies?
This question posed some definitional problems. Respondents had different definitions of movement therapy, including yoga and, more commonly, physical therapy.
Definition unknown
Specific movement techniques (Feldenkreis, Alexander Technique, Pilates, Trager Psychophysical Intergration) were asked about individually
How old where you when you first saw a practitioner for [therapy]?
It was nearly impossible for persons who were heavy users of complementary therapies for a long time to remember the age at which they started using various complementary therapies.
Failure to recall
Question dropped
Have you EVER used natural herbs or other non-vitamin supplements for your own health?
For some people, using herbal, non-vitamin supplements carries meaning beyond simply using an herb for a specific purpose. As one respondent put it, using alternative therapies is a “way of life.” As a result, people who do not see themselves in this light define their supplement use differently from those they view as “users” of supplements. Several respondents who had no complementary therapy identity wanted to be sure they weren’t labeled as “users.” And this concern affected the way they interpreted and answered the questions. In the end, false negatives were elicited from respondents with this perspective; that is, the question is not capturing all people who use herbal supplements. Furthermore, the data suggest herbal users may not be missed at random, representing a pattern that would lead to biased estimates [62].
Answer based on self-concept rather than actual behavior
Question limited to specific pills, capsules, tablets, or liquids labeled as a dietary supplement, listed on a flashcard shown to the respondent
DURING THE PAST 12 MONTHS, did [child] pray specifically for the purpose of his/her OWN health?
This question asked respondents about something they have no way of knowing for sure (“I don’t know”; “I don’t think so”; “I’m guessing no”; “not that I’m aware of”). Those with small children can make a better assessment because prayers are sometimes said out loud together, but those with older children usually have to guess [62]. Information unknown Question dropped

[TableWrap ID: T8] Table 8 

Examples of wellness-related questions deleted or modified based on cognitive review, focus groups and other input, 2012

Question (s) Outcome Rationale
Help you relax
Combined into “help to reduce your stress level or to relax”
Testing found the questions to be redundant
Help to reduce your stress level
To cleanse/detoxify your body
Respondents offered a wide variety of interpretations and meaning varied tremendously from person to person. Some considered it to be referring to “a cleanse” in the very literal sense, such as a colon cleanse. One respondent provided the example of the cayenne pepper and maple syrup diet. Others thought it referred to the period after an addict stops using drugs or alcohol and attempts to purge the body of these substances. For still others this was interpreted to be more symbolic, as in the case of a “mental or emotional cleans”’ as one respondent put it [63].
Because the practitioner spends more time with you than medical doctors
Several respondents explained that they pay for the time with the practitioner. As one respondent put it after hearing this question, “Well, yeah, [because] I paid for 20 minutes.” Another respondent similarly explained that you “pay for the time” with the practitioner and so “of course you get more time with them” (he was thinking specifically about massage practitioners) [63].
Because the practitioner treats the whole person and not just one part because it focuses on the whole person and not just one part
Combined/Modified to “it focuses on the whole person, mind, body, and spirit”
The question did not work well when focused on “the practitioner,” especially for massage. Several noted that they actually want their masseuse to focus on only one part of, for example, their backs, and not their entire bodies. Also, the phrase “mind, body, and spirit” resonated with respondents’ experiences with complementary therapies [63].
Because I can participate in decisions about my health with my practitioner
The question had varying interpretations and varied widely based on the type of practitioner being seen. Respondents using “hands-on” treatments such as massage said they can tell their practitioners where to focus and which parts of their bodies need more attention. Others thought it meant “sitting down with my doctor, presenting the symptoms and then working together to come up with a solution. [63]”
To stay healthy
A number of respondents commented that this either seemed “too broad” or ‘too obvious.” As one respondent put it, “You go to a doctor to stay healthy. It’s too obvious.” Another asked, “What’s healthy? That’s too general.” [63]
Thinking about seeing a practitioner for [modality], please tell me if any of these statements are true for you. Using [modality] has…
Modified to accommodate proxy reporting and include items salient to children
Question stem and individual items modified to ask parent proxy whether he/she “thinks” these items applied to his/her child, and items thought to be unknown by parent were dropped. Also, the items on alcohol and smoking were not asked for children. Attendance at school was added.
Given you a sense of control over your health?
Helped you to relax?
Helped you to reduce your stress level?
Motivated you to eat healthier?
Motivated you to eat more organic foods?
[for respondents who report drinking alcohol in core]
Motivated you to cut back or stop drinking alcohol?
[for respondents who report smoking in core]
Motivated you to cut back or stop smoking cigarettes, cigars, or pipes?
Motivated you to exercise more regularly?
Improved your overall health and made you feel better?
Given you more hope for the future?
Increased your ability to focus?
Made you feel better emotionally?
Made it easier to cope with health problems?
Improved your outlook on life?
Improved your relationships with others?
Improved your self-confidence?
During the past 12 months, did you see a practitioner for [modality] for any of these reasons?
Modified to accommodate proxy reporting and include items salient to children Question stem and individual items modified so that parent proxy could answer about his/her child, and items thought to be unknown by parents were dropped. Item on sexual performance not asked for children.
To stay healthy
To improve your energy
To Improve your immune function
To improve your physical ability
To improve your athletic or sports
To improve your sexual performance
To improve your concentration
To improve your memory
To improve your flexibility
To improve your muscle strength

Article Categories:
  • Correspondence

Keywords: Complementary therapies, Complementary medicine, Questionnaire development, Cognitive interviewing, National Health Interview Survey (NHIS).

Previous Document:  An Inflammatory Pathway Links Atherosclerotic Cardiovascular Disease Risk to Neural Activity Evoked ...
Next Document:  The use of proton pump inhibitors in treating and preventing NSAID-induced mucosal damage.