A comprehensive analysis of the quality of online health-related information regarding schizophrenia.
Abstract: Social workers are major mental health providers and, thus, can be key players in guiding consumers and their families to accurate information regarding schizophrenia. The present study, using the WebMedQual scale, is a comprehensive analysis across a one-year period at two different time points of the top for-profit and nonprofit sites that provide information regarding schizophrenia. Results demonstrate that the majority of sites offer comprehensive information and useful features. For-profit sites had higher ratings than nonprofit sites. Likewise, there was somewhat greater inconsistency for nonprofit sites across the two rating periods. The results demonstrate that systematic evaluation of online health information about schizophrenia can contribute to a "gold standard" for Internet health-related information. In addition, the results provide information that social workers can use with consumers and their families to guide them to quality online health information. A brief list of top sites is provided for social workers to give to consumers and families.

KEY WORDS: online health information; quality of health information; resources for consumers; schizophrenia
Article Type: Report
Subject: Online health care information services (Usage)
Social workers (Psychological aspects)
Schizophrenia (Care and treatment)
Schizophrenia (Research)
Authors: Guada, Joseph
Venable, Victoria
Pub Date: 02/01/2011
Publication: Name: Health and Social Work Publisher: National Association of Social Workers Audience: Academic; Professional Format: Magazine/Journal Subject: Health; Sociology and social work Copyright: COPYRIGHT 2011 National Association of Social Workers ISSN: 0360-7283
Issue: Date: Feb, 2011 Source Volume: 36 Source Issue: 1
Topic: Event Code: 310 Science & research Computer Subject: Online health care service
Geographic: Geographic Scope: United States Geographic Code: 1USA United States
Accession Number: 251086512
Full Text: In the past 15 years, the Internet has become a major source of health information, including information about major mental health disorders such as schizophrenia (Eysenbach, Powell, Kuss, & Sa, 2002; Murphy, Frost, Webster, & Schmidt, 2004). The Internet is a relatively open medium. Some sites provide medical information to sell a product; other sites offer inaccurate or outdated information that might complicate an already difficult situation (Kummervold et al., 2002; Murphy et al., 2004).

Studies have looked at various medical information sites (Anderson, Nikzad-Terhune, & Gaugler, 2009; Godin, Truschel, & Singh, 2005), but few have reviewed sites with specific information about psychiatric conditions such as schizophrenia. Consumers and family members have complained that mental health professionals provide little information regarding schizophrenia (Lefley, 1998; National Alliance on Mental Illness [NAMI], 2003). Consequently, the Internet might be a primary resource to obtain information on schizophrenia. The present study sought to do two things: (1) comprehensively assess the quality of information about schizophrenia on the Internet at two time points across a one-year period and (2) provide a simple, easy to use, rank ordering of quality sites for social workers to use with consumers and family members.

BACKGROUND

Three million (1 percent) people in the United States hve with schizophrenia (National Institute of Mental Health [NIMH], 2008; U. S. Department of Health and Human Services (HHS), 2001). If left untreated, schizophrenia can result in inadvertent harmful consequences such as lack of self-care (including medical care), homelessness, substance abuse problems, increased vulnerability, poverty, and suicide (NIMH, 2008). NIMH reported that the impact of schizophrenia for consumers, their loved ones, and the communities they live in is substantial (HHS, 2001; NIMH, 2008). Thus, care and intervention for those living with schizophrenia is a major public health concern.

A critical component of intervention is information about schizophrenia that is accurate, up-to-date, and understandable. This is particularly true regarding diagnosis, possible treatments (and risks involved), and resources available to consumers and family members. As noted, consumers and family members have complained of not receiving adequate information about schizophrenia and available treatments (Lefley, 1998; NAMI, 2003). Social workers can refer consumers and families to quality online medical information as one option to decrease the information gap.

Quality of Online Medical Information

A recent study surveying online users about their consumption of health-related information found that the majority do not consistently evaluate the quality of the information they get online. This suggests that the majority of Americans do not ensure that the online information they obtain is accurate (Pew Internet & American Life Project, 2006). Indeed, available studies have demonstrated that the quality of online health information varies widely (Eysenbach et al., 2002). At the same time, the state of the literature in assessing the quality of online medical information is at an early stage, especially for specific psychiatric disorders (Eysenbach et al., 2002; Murphy et al., 2004). Social workers can assist consumers and families by having a reliable resource that highlights quality online medical information.

Researchers and professional health organizations have tried to operationalize "quality" information retrieved from the Internet (Eysenbach et al., 2002). Typically, quality information has included data from the peer-reviewed literature that are presented in a clear and understandable way and provide a broad range of information about the medical condition (that is, information is comprehensive) (Eysenbach et al., 2002; Pew Internet & American Life Project, 2006; Provost, Koompalum, Dong, & Martin, 2006). The Health on the Net Foundation (HNF) created a list of principles for health sites to follow when providing health information (HNF, 2009). The principles are based on the following "gold standard": the currency of information (peer-reviewed journals, recency of updates), the credentials of writers and editors, and the comprehensiveness of the information. In addition, the HNF principles include policies regarding editorial content; the purpose of the information (that is, opinions versus facts); the presence of advertisers, sponsors, or both; and privacy policies (HNF, 2009).

These efforts, however, pertain to general online health information sites. The need remains to evaluate online information for specific serious disorders such as schizophrenia. Some researchers have evaluated the quality of online health information for other disorders, such as Alzheimer's and eating disorders (Anderson et al., 2009; Godin et al., 2005; Murphy et al., 2004). One study assessed the quality of information regarding mental health issues and disorders (Godin et al., 2005). To our knowledge, the present study is the first study to evaluate online health information specific to schizophrenia.

Purpose of the Study

The present study focused on Web sites that provide information about schizophrenia. The main purpose was to answer this question: What is the quality of information about schizophrenia on the Internet? The objective is to contribute to the relatively new field of research that analyzes online health information in a standardized and comprehensive fashion. The second purpose was to provide a simple guide for social workers to use with consumers and their loved ones.

METHOD

Research shows that the general public uses a search engine as opposed to a specific uniform resource locator to locate desired health information (Pew Internet & American Life Project, 2006). Research also shows that the general public uses search engines on the basis of their familiarity and past experiences with these, which results in certain search engines being favored over others (Pew Internet & American Life Project, 2006). The present study chose the three most popular search engines (Lewis, 2009) to develop a sample of health information sites about schizophrenia: Google (google. com), Yahoo (yahoo. com), and MSN (msn.com). The study used the search terms "schizophrenia, "schizoaffective," and "psychosis" on the basis of diagnostic categories established for schizophrenia spectrum disorders in the DSM-IV-TR (American Psychiatric Association, 2000). Only English-language sites were included. A major study also found that people chose only the top one or two search results they received from a search engine (iProspect, 2006); thus, the present study selected only sites on the first page of each search.

The search resulted in a total of 30 sites. Three of these sites were secondary pages to one of the previous sites already selected, so they were dropped from the analysis. The final total was 27 sites. Because of the fluid nature of information on the Internet, the study evaluated the sites within a narrow time frame to reduce the influence of content change. Sites were accessed two times across a one-year period, the first series of rating occurring between March 30 and April 9, 2009, and the second occurring between May 11 and May 25, 2010.

Measure

The literature discusses the relative benefits and drawbacks of a disease-specific assessment device versus a generically designed scale to assess online health-related information (Anderson et al., 2009; Bath & Bouchier, 2003; Provost et al., 2006). There is no known disorder-specific Web site evaluation scale available for schizophrenia; thus, this study used a generic scale that offered a comprehensive approach to assessing quality online health information: the WebMedQual scale (Provost et al., 2006).

The main purpose of the WebMedQual scale is to offer a comprehensive assessment of the quality of online health information, as suggested by organizations such as the HNF (2009). Provost et al. (2006) used the HNF principles to create the WebMedQual. They used several phases of item development, including a health expert review of items. The final content analysis resulted in 95 items. The authors did not provide other psychometric properties for the scale, and no further work has been done to assess these properties to date, although there were plans to do so sometime in the future (personal communication with M. Provost, clinical research associate, Novartis Pharmaceuticals Canada, Inc., Toronto, April 4, 2009). The scale has been used in other studies evaluating disease-specific online health information (Anderson et al., 2009).

The items are divided into eight subcategories: content (currency and accuracy of information), authority of source (disclosure of expertise, training, and possible financial interests of contributing authors), design of Web site (navigation experience of site), accessibility and availability of information (regular availability of Web site, ease in functionality),links (quality and number of links), user support (technical support information), confidentiality and privacy (privacy and confidentiality policies), and e-commerce (products or services offered by site) (Provost et al., 2006). The e-commerce subcategory was dropped as the focus of this study was information that was free and readily available online. The reader is invited to review Appendix A for more detailed information regarding each of the subcategories.

The items are answered in a "yes" or "no" checklist format. All items are summed into a final score, with a higher score indicating a higher quality of online health information. The scale is scored as follows: a site receives a score of 2 ("yes") if the information is present and a score of 1 ("no") if the information is not present. In addition, an item is scored as I for "not sure" (that is, the reviewer is uncertain whether the site has the requested information). Several questions required reversed scoring to avoid potential scoring bias. We dropped five items because of their apparent obsolescence. For example, we found no site that included the following features: a text-only option, easy-to-find information about the platforms and browsers that permit optimal viewing, and user onscreen notification when entering or leaving a secure site. This resulted in use of 84 of the original 95 items of the scale.

We found that a substantial portion of sites did not include any advertisements due to their nonprofit status. Sites that did not include advertisements would automatically receive lower scores on a number of items pertaining to advertisements and corporate sponsorships (for example, "Is funding or other sponsorship for any specific content clearly indicated?") and, thus, would receive lower total scores, suggesting lower quality of information. Thus, the present study divided sites into two groups: nonprofit and for-profit. Twelve items regarding the use of advertisements were dropped for nonprofit sites, resulting in 72 items for this group. We used the generic top-level domain, as used in the Internet's domain name system, to determine a site's nonprofit (.org) or for-profit (.com/.net) status. Nonetheless, one nonprofit site (familydoctor. org) was moved to the for-profit group because it included numerous ads typical of a for-profit site. This resulted in an n of 12 for nonprofit sites and an n of 15 for for-profits.

Joseph Guada, who has over 20 years of psychiatric social work and research experience with families and schizophrenia, was the main assessor. Victoria Venable, who has several years of practice experience with child and adolescent mental health, assessed the same sites to test for interrater reliability. The clinical backgrounds of both authors helped in identifying sites that targeted nonprofessionals for inclusion in the study. Finally, neither author had any professional relationship or financial interests with any of the organizations whose sites were assessed. Both authors rated all 27 sites during the specified periods noted earlier.

Analysis

Subcategory and full-scale summations of nonprofit and for-profit Web sites are presented separately, given the difference in number of items used from the WebMedQual scale. Descriptive statistics were run for each subcategory as well as for the full scale within each group of sites. An interrater reliability was run to test for consistency across the two raters at both time periods, and a test-retest reliability was run to test for consistency of the overall scale across both time periods. In addition, because each group had a different range of possible scores, we converted the full and subscale scores into "percentage of items endorsed." In this way, group and subscale scores were comparable (on a scale of 0 percent to 100 percent, similar to that of a grading system). This would help in interpreting and understanding the rank order of sites.

RESULTS

The results of interrater reliability for the full scale were good across both raters for the for-profit sites (.84). However, the results were less satisfactory for the nonprofit sites at time 1 (.59) and time 2 (.68). Review of the results for time 1 showed four sites for which the two raters had differences in ratings (MedlinePlus, NIMH, Substance Abuse and Mental Health Services Administration [SAMHSA], and Wikipedia: Schizophrenia). The raters reviewed their ratings for these four sites and conferred on items where there was disagreement. The interrater reliability was rerun for the nonprofit group at time 1, which provided a satisfactory reliability (.90). The same process was used at time 2. There was a greater distribution of which sites had substantial differences across raters, depending on the subscale in question. The three subscales with the lowest interrater reliability were links, user support, and confidentiality. After reviewing our ratings, the overall interrater reliability for nonprofits was recalculated and increased to. 87 at time 2. Test-retest reliabilities across the two time points for the for-profit sites were .85 and .74 for nonprofit sites, respectively, suggesting a reasonable amount of reliability for the scale (Anastasi, 1988).

For-Profit Sites

The mean score for the for-profit sites was 137 (SD = 12) at time 1 and 132 (SD = 13) at time 2, with a possible range of 84 to 168. The average scores indicated that approximately 82 percent of the items (time 1) and 79 percent endorsement rate (time 2), suggesting that the sites did a good job of providing information, features, and resources regarding schizophrenia. The item endorsement rate for subcategories was on average 75 percent (time 1) to 79 percent (time 2), with a range of 73 percent (confidentiality) to 90 percent (design) at time 1 and 60 percent (accessibility) to 97 percent (design) at time 2.

The rank order of the sites is presented in Table 1. Scores ranged from a high of 153 (WebMD) to a low of 116 (The Experience of Schizophrenia), with a 27-point difference between the highest and lowest ranking sites at time 1. The WebMD subsection on schizophrenia's (http://www.webmd. com/schizophrenia/default.htm) 91 percent endorsement rate suggests that it does a very good job of providing features and information. MedicineNet.com's section on schizophrenia (http://www.medicinenet.com/schizophrenia/ article.htm) had an 89 percent endorsement rate, and Psych Central's section called "Schizophrenia and Psychosis" (http://psychcentral.com/disorders/ schizophrenia/) had an 88 percent endorsement rate. The scores suggest that the sites offered good quality information and features on the disorder. This rank order changed somewhat at time 2. Scores ranged from a high of 147 for eMedicine.com to low of 107 for Docguide.com, with a 40-point spread. Essentially, the same top five sites from the first series of ratings were the top five in the second series of ratings, although the exact order changed somewhat (see Table 1). Once again, the endorsement rates for the top five sites (84 percent to 87 percent) (time 2) suggest an overall good quality of information and features regarding schizophrenia.

Nonprofit Sites

The overall mean score for nonprofits was 111 (SD = 6; range: 101 to 123) at time 1 and 111 (SD = 8; range: 94 to 123) at time 2. The average scores indicated that 77 percent of the items were endorsed (for both times), suggesting that the sites did an average job of providing information, features, and resources regarding schizophrenia. Scores on the subcategories ranged from 53 percent to 90 percent at time 1. Two of the subcategories (authority and confidentiality) had less than 75 percent of the items endorsed. Authority had only 67 percent of the items endorsed, and confidentiality had only 53 percent of the items endorsed. In comparison, the for-profits did noticeably better than the nonprofits in these subcategories (authority: 78 percent; confidentiality: 73 percent) at time 1. Nonetheless, the range of endorsement of items across subscales increased somewhat, such that the lowest endorsement rating was 67 percent (authority) and the highest was 93 percent (design) at time 2. Both confidentiality and accessibility had endorsement rates of 70 percent; all other subscales had an endorsement rate greater than 75 percent. Overall, these results suggest that nonprofits did a somewhat poorer job of providing information regarding the qualifications of authors, providing clear information about confidentiality policies, and providing accessibility features.

As shown in Table 1, scores at time 1 ranged from a high of 123 (NAMI) to a low of 101 (Wikipedia: Schizophrenia), with a 22-point difference between the highest and lowest ranking sites. The NAMI section on schizophrenia (http://www.nami.org/Template.cfm?Section=By_Illness& Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=54&ContentID=23036) was the optimal site, with an 85 percent endorsement rate, suggesting a good quality of information and features. Mental Health America's page on schizophrenia (http://www.nmha.org/go/schizophrenia) had an 82 percent endorsement rate, also suggesting a relatively good quality of information and features; Helpguide. org's page on schizophrenia (http://www.helpguide.org/mental/schizophrenia_symptom.htm) had an 80 percent endorsement rate, suggesting a similar level of quality information and features. Scores at time 2 ranged from a high of 124 for MedlinePlus to a low of 94 for Successful Schizophrenia. As listed in Table 1, the top five sites changed at time 2: Both the Mental Health America and SAMHSA sites dropped out, and the site for the National Alliance for Research on Schizophrenia and Depression (NARSAD) moved up (only one site moved up given that time 1's list included six sites in the top five, owing to a tie between two sites for the rank order of five). A brief summary of top=rated sites across both groups based on the 2010 ratings is presented in Appendix B.

DISCUSSION

An increasing number of people use the Internet as a main source of health information (Eysenbach et al., 2002; iProspect, 2006; Pew Internet & American Life Project, 2006); thus, there is a critical need to ensure that such information is accurate and based on the best available research. This is particularly salient for consumers of mental health services and their families, who historically have been dissatisfied with the lack of information they receive from professionals. The present study sought to add to the literature regarding quality online health information by assessing sites about schizophrenic spectrum disorders. In addition, we sought to provide a resource for both social workers and consumers regarding the best of these sites (see Appendix B).

Overall Findings on Quality of Information and Site Features

The study found that the majority of Web sites included most of the features and information assessed by the WebMedQual scale, which is based on criteria from the HNF. The results offer preliminary data that the majority of sites strive to provide accurate, comprehensive, and helpful information about a potentially devastating disorder (HNF, 2009; Pew Internet & American Life Project, 2006). This was found at both rating time points. These results are in marked contrast to those of recent studies done to evaluate the quality of information about eating disorders (Murphy et al., 2004) and self-help sites (Godin et al., 2005). A more recent study on the quality of information for caregivers of someone with Alzheimer's found similar results to this study's in that the overall quality was adequate to very good, with some variation across individual sites (Anderson et al., 2009).

There were a few common problems across sites. Most did not offer the flexibility to customize interactivity for individuals with a disability, did not offer another language choice, or did not offer text enlargement. In addition, the nonprofits did a poorer job of providing information about confidentiality. This may have been because these sites did not collect either personally identifying or aggregate information (see the For-Profits versus Nonprofits section). Nonetheless, nonprofits should strengthen their disclosure regarding the collection (or lack thereof) of any personal identifying information and how that information is used.

An interesting finding is that none of the sites actually provided a stand-alone site that offered information specifically geared to consumers and family members about schizophrenic spectrum disorders. Typically, the information was a secondary part of the site. A substantial number of for-profit sites were what we called "general purpose" medical information Web sites that offered a wide range of information, from preventive care and prescription medication information to sections specific to the needs of children, women, men, and older adults. Thus, schizophrenia was a subsection of a larger mental health section, which itself was a section among numerous other medical conditions. Although it is encouraging to see that mental health disorders are seen as a part of the larger notion of "health" on general purpose sites, it also made finding the information more difficult or, at least, less convenient.

For-Profits versus Nonprofits

In general, the study found that for-profit sites' ratings were better than nonprofit sites' and that this persisted from time 1 to time 2. Despite having sponsors and advertisers, the for-profits had a higher percentage of items endorsed across all of the subcategories. The ratings indicated that they did a better job with the currency and accuracy of health information and the disclosure of expertise and possible financial interests of authors, and they provided user-friendly navigational features, technical support, and clearer privacy and confidentiality policies. This suggests that the mere presence of advertisements or sponsors did not necessarily compromise the quality or availability of features on a site.

The results raise the question of why for-profit sites tended to do better than nonprofit sites. One possibility is that for-profit sites are motivated to keep their sites up-to-date and information easily accessible for potential consumers of products and services offered on the site. Functionality and user friendliness can provide a more positive experience for a visitor while increasing the likelihood of the person returning in the future. Likewise, for-profit sites are more motivated to streamline their sites to facilitate navigation to as many portions of the site as possible, which can mean more "clicks" on pages where advertisers place their ads. For-profit sites also extensively used "cookies" to track users. As explained by virtually all of the sites that used cookies, the information obtained is provided to potential advertisers in aggregate form. This practice helps to create a more personalized experience not only in features and information for the visitor, but also in targeted advertising based on a visitor's previous navigation of the site (Ha, Al Shaar, Inkpen, & Hdeib, 2006). However, nonprofits probably did not have the resources to more closely manage their sites: The existence of their sites was independent of any possible income earned through them. Likewise, there was a larger amount of change regarding which sites were rated in the top 5 across the two time periods. This was mostly due to a substantial revamping of the NARSAD, the Brain and Behavior Research Fund site (http://www.narsad.org), which rebranded itself and included several new pages, features, and policies about using the site.

Limitations

The present study selected sites that, at the time of the ratings, were listed as top search results for two time periods. It is likely that many of these sites will remain at the top of search engine results at least in the near future. However, given the ongoing, changing nature of the Internet, the present study's results could quickly become obsolete. As previously noted, this proved in part to be the case with the nonprofit sites at time 2. We hope that the results from the present study will serve as a starting point for the ongoing evaluation of the quality of health-related information online regarding schizophrenia.

Interrater reliability was good in regard to the for-profit sites, but it is unclear why it was poorer for the nonprofit sites, although the rating improved somewhat at time 2. The nature of the nonprofit sites raises questions regarding what it was about these sites that made it difficult for us to endorse in like fashion at least when independently rated. Both raters reported that it was difficult to determine what features and information was located on some nonprofit Web sites because they had atypical designs. The Wikipedia page in particular has a unique design as well as a rapidly changing format for information updates. Likewise, although Medline Plus had many of the design features typical of others Web pages (for example, help, frequently asked questions, contact information), the topical information was provided across numerous separate pages so that locating a specific topic or site policy was sometimes difficult. This implies that the nature of some sites may make it difficult to locate information, at least initially.

In addition, at present, the WebMedQual scale has limited psychometric properties. Any findings based on the scale are limited until further psychometric testing is done. Although the test-retest reliabilities were good for both the for-profit and nonprofit sites, the consistently lower interrater reliabilities for the nonprofits suggest that the scale may have limited utility for such sites. Thus, the present study's findings are provisional at best.

CONCLUSION

The results provide the first evaluation of the quality of online health information about schizophrenic spectrum disorders. The information can be used as a resource guide for social workers serving individuals seeking information about this often debilitating and serious disorder. In addition, the present study offers a starting point for the continuation of a process for evaluating Web-based information regarding schizophrenic spectrum disorders. Systematically evaluating the online health information about specific mental illnesses can contribute to further clarification regarding a gold standard for Internet-based health information, in particular for schizophrenia. And, as previously noted, it provides another source of information for social workers to provide to consumers and their loved ones.

Given the constantly changing nature of the Internet, it is probable that the sites included in any study of this nature may substantially change in content and site organization. It is likewise possible that other new sites, unavailable at the time of this study, will appear, necessitating ongoing evaluation of the quality of information related to schizophrenia on the Internet. We hope that this will be the first step toward an ongoing process of evaluating online health information regarding one of the major psychiatric disorders affecting people today so that social workers have yet one more valuable resource to assist consumers and their families.

APPENDIX B: INTERNET SITES ON SCHIZOPHRENIA FOR FAMILIES, CONSUMERS, AND PROFESSIONALS

For-Profit Web Sites (In No Particular Order)

WebMD Schizophrenia http://www.webmd.com/schizophrenia/default.htm In-depth information on schizophrenia plus discussion boards and blogs. Registration may be required for some features.

MedicineNet.com: Schizophrenia http://www.medicinenet.com/schizophrenia/article.htm Provides content on schizophrenia, medications, and treatment. Discussion pages are also available.

Psych Central: Schizophrenia and Psychosis http://psychcentral.com/disorders/schizophrenia/ Fact sheets, chat rooms, and in-depth information about schizophrenia. Registration may be required to access some features.

Nonprofit Web Sites (In No Particular Order)

National Alliance on Mental Illness (NAMI) http://www.nami.org/

Provides general information on schizophrenia as well as other major mental health disorders. Also provides information about getting help and doing advocacy at the local and national levels.

NARSAD--The Brain and Behavior Research Fund http://www.narsad.org/

Provides general information about schizophrenia as well as other major mental health disorders. Also provides the opportunity to donate toward its general fund to fund promising research regarding these disorders..

Helpguide.org http://www.helpguide.org/

Provides general information regarding mental health issues and coping strategies.

Note: This is a partial listing of Web sites reviewed for content, source authority, and confidentiality. For a full listing, please contact Joseph Guada (guada.1@osu.edu) at Ohio State University, College of Social Work.

Original manuscript received October 19, 2009

Final revision received June 17, 2010

Accepted July 28, 2010

REFERENCES

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Anastasi, A. (1988). Psychological testing (6th ed.). New York: Macmillan.

Anderson, K. A., Nikzad-Terhune, K. A., & Gaugler, J. E. (2009). A systematic evaluation of online resources for dementia caregivers. Journal of Consumer Health on the Internet, 13, 1-13.

Bath, P. A., & Bouchier, H. (2003). Development and application of a tool designed to evaluate Web sites providing information on Alzheimer's disease. Journal of Information Science, 29, 279-297.

Eysenbach, G., Powell, J., Kuss, O., & Sa, E. (2002). Empirical studies assessing the quality of health information for consumers on the World Wide Web: A systematic review. JAMA, 287, 2691-2700.

Godin, S., Truschel, J., & Singh, V. (2005). Assessing quality assurance of self-help sites on the Internet. Journal of Prevention & Intervention in the Community, 29, 67-84.

Ha, V., Al Shaar, F., Inkpen, K., & Hdeib, L. (2006). An examination of user perception and misconception of Internet cookies. In CHI extended abstracts (pp. 833-838). New York: SIGCHI.

Health on the Net Foundation. (2009). The HONCode in brief. Retrieved from http://www.hon.ch/HONcode/Webmasters/Conduct.html

iProspect. (2006). Search Engine Behavior Study [Electronic version]. Retrieved from http://www.iprospect.com/premiumPDFs/WhitePaper_2006_ SearchEngineUserBehavior.pdf

Kummervold, P. E., Gammon, D., Bergvik, S., Johnsen, J. K., Hasvold, T., & Rosenvinge, J. H. (2002). Social support in a wired world: Use of online mental health forums in Norway. Nordic Journal of Psychiatry, 56, 59-65.

Lefley, H. P. (1998). Families, culture, and mental illness: Constructing new realities. Psychiatry: Interpersonal & Biological Processes, 61, 335-355.

Lewis, E. (2009). Top ten search engines. Retrieved from http://www.seoconsultants.com/search-engines/

Murphy, R., Frost, S.,Webster, P., & Schmidt, U. (2004). An evaluation of Web-based information. International Journal of Eating Disorders, 35, 145-154.

National Alliance on Mental Illness. (2003). Family to family: Recent studies of family-to-family. Retrieved from http://www.nami.org/Template.cfm?Section=Family-to- Family&template=/ContentManagement/ContentDisplay.cfm&ContentID=36366

National Institute of Mental Health. (2008). National Institute of Mental Health strategic plan. Washington, DC: Author.

Pew Internet & American Life Project. (2006). Online health search 2006. Retrieved from http://www.pewinternet.org/~/media//Files/Reports/2006/ PIP_Online_Health_2006.pdf.pdf

Provost, M., Koompalum, D., Dong, D., & Martin, B. C. (2006). The initial development of the WebMedQual Scale: Domain assessment of the construct of quality of health Web sites. International Journal of Medical Informatics, 75, 42-57.

U.S. Department of Health and Human Services. (2001). Mental health: Culture, race, and ethnicity--A supplement to mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.

Joseph Guada, PhD, MSW, is assistant professor, and Victoria Venable, MSW, LISW-S, is research assistant, College of Social Work, Ohio State University, Columbus. Address correspondence to Joseph Guada, College of Social Work, Ohio State University, 1947 College Road, 325G Stillman Hall, Columbus, OH 43210; e-mail: guada.1@osu.edu.
APPENDIX A: WEBMEDQUAL SUBCATEGORY DESCRIPTIONS

Subcategory       Description                       No. of items

Content           Items deal with the currency           19
                  and accuracy of health
                  information provided by the
                  site. How timely is the
                  information? How thorough and
                  complete is the information?

Authority of      Items deal with the disclosure         18
source            of expertise, training, and
                  possible financial interests
                  of contributing authors. Who
                  are the site's authors, and do
                  they have potential conflicts
                  of interest if the site has
                  advertisers or corporate
                  sponsorship?

Design of Web     Items deal with the user's             17
site              navigation experience of the
                  site. How appealing and
                  intuitive are the site's
                  features so that information
                  can be easily found?

Accessibility     Items deal with the regular            6
and               availability of Web site
availability of   content and ease in
information       functionality (including for
                  those with sight impairments
                  or whose first language is not
                  English).

Links             Items deal with the quality            4
                  and number of links offered by
                  the site so that a user can
                  get more information if
                  desired.

User support      Items deal with both technical         8
                  support and requests for more
                  information. How easy is it to
                  locate and contact technical
                  support? How easy is it to
                  locate and contact site
                  administrators with question
                  regarding the health
                  information provided by the
                  site?

Confidentiality   Items deal with the site's             17
and privacy       privacy and confidentiality
                  policies. Does the site
                  clearly explain what personal
                  information, if any, is
                  collected and how this
                  information might be used in
                  the future?

E-commerce        Items deal with the user's             6
                  experience of buying products
                  or services offered by the
                  site (Provost et al., 2006).


Table 1: Rank Order Ratings of Sites Based on Overall Score on
WebMedQual Scale

                          2009         2010
                          Rank         Rank
For-Profit                Order        Order
Web Site (a)             (Score)      (Score)

WebMD Schizophrenia      1 (153)      2 (146)

MedicineNet.com:         2 (150)      2 (146)
Schizophrenia

Psych Central:           3 (147)      3 (145)
Schizophrenia and
Psychosis

eMedicine:               4 (146)      1 (147)
Schizophrenia

MayoClinic.com:          5 (145)      4 (141)
Schizophrenia

Familydoctor.org:          (145)      5 (139)
Schizophrenia

EverydayHealth.com:      7 (142)      6 (137)
Schizophrenia Center

Schizophrenia.com        8 (138)      9 (132)

Medical News Today:      9 (134)     10 (129)
Schizophrenia

Merck Manual Home       10 (133)      7 (135)
Edition:
Schizophrenia

Health Information--    133          12 (118)
Schizophrenia

Mentalhelp.net:         12 (129)      8 (133)
Schizophrenia

BehaveNet:              13 (120)     11 (124)
Schizophrenia

Docguide.com:           14 (119)     14 (107)
Schizophrenia

The Experience of       15 (116)     13 (110)
Schizophrenia

For-Profit
Web Site (a)           URL

WebMD Schizophrenia    http://www.webmd.com/schizophrenia/
                       default.htm

MedicineNet.com:       http://www.medicinenet.com/schizophrenia/
Schizophrenia          article.htm

Psych Central:         http://psychcentral.com/disorders/
Schizophrenia and      schizophrenia/
Psychosis

eMedicine:             http://emedicine.medscape.com/article/
Schizophrenia          805988 overview

MayoClinic.com:        http://www.mayoclinic.com/health/
Schizophrenia          schizophrenia/DS00196

Familydoctor.org:      http://familydoctor.org/online/famdocen/
Schizophrenia          home/common/mental health/ treatment/
                       266.html

EverydayHealth.com:    http://www.everydayhealth.com/emotional-
Schizophrenia Center   health/schizophrenia/index.aspx

Schizophrenia.com      http://www.schizophrenia.com/index.php

Medical News Today:    http://www.medicalnewstoday.com/
Schizophrenia          sections/schizophrenia/

Merck Manual Home      http://www.merck.com/mmhe/sec07/
Edition:               ch107/ch1076.html
Schizophrenia

Health Information--   http://www.cmellc.com/topics/
Schizophrenia          schiz.html

Mentalhelp.net:        http://www.mentalhelp.net/poc/center-
Schizophrenia          index. php?id=7

BehaveNet:             http://behavenet.com/capsules/
Schizophrenia          disorders/schiz.htm

Docguide.com:          http://www.docguide.com/news/
Schizophrenia          content.nsf/
                       PatientResAllCateg/
                       Schizophrenia?OpenDocument

The Experience of      http://www.chovil.com/
Schizophrenia

                           2009       2009
                          Rank       Rank
Nonprofit                 Order      Order
Web Site (b)             (Score)    (Score)

NAMI                     1 (123)    3 (119)

Mental Health            2 (118)    8 (111)
America:
Schizophrenia

Helpguide.org:           3 (115)    4 (114)
Schizophrenia

Neuroscience for         4 (114)    4 (114)
Kids--Schizophrenia

MedlinePlus:             5 (112)    1 (123)
Schizophrenia

SAMHSA:                    (112)    7 (112)
Schizophrenia

AACAP: Schizophrenia     7 (111)    9 (108)
In Children

NARSAD--The Brain
Fund and Behavior
Research (formerly       8 (110)    2 (120)
known as National
Alliance for
Research on
Schizophrenia and
Depression)

NIMH--Schizophrenia      9 (109)    6 (113)

Mental Health: A
Report of the           10 (105)   11 (100)
Surgeon General--
Chapter 4
(Schizophrenia)

Successful              11 (103)   12  (94)
Schizophrenia

Wikipedia:              12 (101)   10 (107)
Schizophrenia

Nonprofit
Web Site (b)           URL

NAMI                   http:www.nami.org/Template.cmf?Section-By_
                       Illness&Template=/TaggedPage/TaggedPageDisplay.
                       cfm&TPLID=54&ContentID=23036

Mental Health          http://www.nmha.org/go/schizophrenia
America:
Schizophrenia

Helpguide.org:         http://www.helpguide.org/mental/
Schizophrenia          schizophrenia_symptom.htm

Neuroscience for       http://faculty.washington.edu/
Kids--Schizophrenia    chudler/schiz.html

MedlinePlus:           http://www.nlm.nih.gov/medlineplus/
Schizophrenia          schizophrenia.html

SAMHSA:                http://mentalhealth.samhsa.gov/
Schizophrenia          publications/allpubs/KEN98-0052/
                       default.asp

AACAP: Schizophrenia   http://www.aacap.org/cs/root/facts
In Children            for families/schizophrenia-in-
                       children

NARSAD--The Brain      http://www.narsad.org/
Fund and Behavior
Research (formerly
known as National
Alliance for
Research on
Schizophrenia and
Depression)

NIMH--Schizophrenia    http://www.nimh.nih.gov/health/
                       topics/schizophrenia/index.shtml
Mental Health: A
Report of the          http://www.surgeongeneral.gov/
Surgeon General--      library/mental health/chapter4/
Chapter 4              sec4.html
(Schizophrenia)

Successful             http://www.successful
Schizophrenia          schizophrenia.org/

Wikipedia:             http://en.wikipedia.org/wiki/
Schizophrenia          Schizophrenia

Note: URL= uniform resource locator; NAMI = National Alliance on
Mental Illness; 5AMHSA= Substance Abuse and Mental Health
Services Administration; AACAP =American Academy of Child and
Adolescent Psychiatry: NARSAD = National Alliance for Research on
Schizophrenia and Depression; NIMH = National Institute of Mental
Health.

(a) Possible range: 84-168.

(b) Possible range: 72-144.
Gale Copyright: Copyright 2011 Gale, Cengage Learning. All rights reserved.