Pragmatic evaluation for large events with youth: use of a brief visual analog scale measure.
|Abstract:||Health educators and program evaluators struggle to balance collecting meaningful data while addressing pragmatics of evaluation data collection, particularly for large youth events. A local day-long community-based youth summit was held with 289 middle school students. Youth attended combinations of mass and break-out sessions. Planners and evaluators assessed proximal student outcomes throughout the day. A 2-question visual analog scale was developed and utilized. Each sessions-specific evaluation form was color-coded and distributed to participants prior to, and collected at the end of, each session. The evaluation tool assessed students' perceptions of learning and enjoyment. This method enabled evaluators to accurately calculate response rates and refusals. Importantly, the method was efficient, inexpensive, and simple.|
McKyer, E. Lisako J.
Smith, Matthew Lee
|Publication:||Name: American Journal of Health Studies Publisher: American Journal of Health Studies Audience: Professional Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2009 American Journal of Health Studies ISSN: 1090-0500|
|Issue:||Date: Fall, 2009 Source Volume: 24 Source Issue: 4|
|Product:||Product Code: E121930 Youth|
An emerging body of research reveals multiple benefits of integrating youth participation in community health efforts (Breitbart & Kepes, 2007; Cahill, 2007; Checkoway, Dobbie, & Richards-Schuster, 2003; Goodyear, & Checkoway, 2003). Targeting youth as part of health and other risk-reducing and asset promoting community-based social capital enhancement programs show great promise. Organized youth activities have been shown to be popular and effective in delivering health education content and providing experiences associated with positive youth development (Hansen, Larson & Dworkin, 2003). Youth summits enable health professionals and volunteers to collaborate with youth to accurately identify relevant psychological and socioecological health concerns, express opinions about health programs and initiatives, and offer recommendations for program improvement.
One-day youth summits attract several hundred participants who convene in a single location. Often, large youth summits cater to participants in different grade levels and from different schools. Even with the most careful of planning, the nature of these youth gatherings may be chaotic, challenging to manage, and a logistical nightmare for event planners and volunteers. Imagine several hundred children and adolescents on a field trip! To complicate this potentially overwhelming environment, youth summit formats frequently incorporate multiple brief sessions hosted throughout the day, with participants rotating in attendance across sessions and venues.
YOUTH SUMMIT EVALUATIONS
Various evaluation approaches have been used by summit planners. Summits hosted for high school-aged youth often utilize questionnaires given to participants at the end of each session. While detailed questionnaires may work with older youth, these methods may not be practical with younger groups (e.g., primary, middle school-ages) due to event-related time constraints and literacy issues. Other approaches have included post-summit follow-ups with youth participants (i.e., retrospective approaches; Smith, Genry, Ketring, 2005). Although retrospective approaches allow for a more in-depth collection of information, such methods are vulnerable to session recall bias, small sample sizes resulting from limited resources, and may include data from sub-samples that are neither representative nor generalizable to the greater sample (Smith, Genry, Ketring, 2005). In short, retrospective evaluation formats may threaten evaluators' ability to determine the internal validity of youth summits, which make any hope of determining external validity to similar populations nearly impossible.
Ideally, data should be collected from summit participants at the end of each session. Evaluators with less stringent restrictions on funding and resources have accomplished this by providing youth participants with hand-held computer devices (Proescholdbell, Scott & Placona, 2005). Using hand-held computer devices to collect information is quick and effective; however, costs associated with this highly technical approach are prohibitive for most communities, which primarily rely on local contributions to meet basic summit needs. Although beneficial in some situations, these evaluations may detract from the effectiveness of youth programs due to designating a large proportion of allocated funds toward overly-sophisticated methods of data collection.
In consideration of various challenges to youth summit evaluation, a need exists for a simple, quick, inexpensive, yet practical and useful outcome evaluation tool, which can be used with large numbers of participants--regardless of participant literacy levels. This paper describes a successful, timely, simple and inexpensive method implemented by the Youth Summit Evaluation Team to assess summit participants' perceptions of how much they learned and their level of enjoyment during each session.
TARGET PROGRAM: BRYAN/COLLEGE STATION 2008 YOUTH SUMMIT
The Bryan/College Station 2008 Youth Summit (BCS Youth Summit) was based on the principles of America's Promise Alliance (1997), and aimed to deliver the Alliance's "five promises" to targeted disadvantaged youth. The Alliance's research-based framework for youth development focuses on every child having: 1) Caring Adults, 2) Safe Places, 3) A Healthy Start, 4) Effective Education, and 5) Opportunities to Help Others. The 2008 BCS Youth Summit included five sessions based on the aforementioned promises (i.e., one mass session and four break-out sessions; see Table 1). Each session lasted between 30 minutes to one hour. Sessions were held in different rooms throughout the summit's venue. Sessions integrated lecture, discussion, and activities to elicit participant engagement and allow them to apply the content and concepts presented. The 289 summit participants and volunteers were given approximately ten minutes to transition between sessions.
Participants were identified as eligible to attend the BCS Youth Summit by their middle school teachers. Teachers were provided a checklist using objective criteria to identify disadvantaged students. Based upon teachers' selections, students' parents were sent recruitment letters and consent forms to enroll their child in the summit. Active parental consent was required for students to attend the event. A total of 289 6th, 7th, and 8th grade participants from 4 local middle schools attended the BCS Youth Summit. Additionally, approximately 40 community volunteers, and 20 teacher volunteers donated their time to the day-long event. Of these middle schools, all four were rated as academically acceptable based on each campus' 2007-2008 TAKS performance scores.
[FIGURE 1 OMITTED]
Youth summit planners recognized the need for collecting evaluation data beyond merely tallying the number of participants who attended the event. Of equal or greater importance was the need to collect (1) information to assist in the improvement of subsequent summits, (2) data to document goal achievement; and (3) evidence which may be parlayed into securing additional event sponsorship, support, and funding in future years.
Brief Visual Analog Measure for Youth (Proximal Outcome Measure). A Brief Visual Analog Survey Measure for Youth (VAS) instrument was developed and utilized to assess participants' perceptions following each of the four youth summit break-out sessions. The instrument consisted of two questions: (1) How much did I learn this session? and (2) How much did I enjoy this session? Response options were on a 3-point Likert-type scale and enhanced by a vi sual analog to facilitate comprehension by children with wide-ranging reading abilities (see Figure 1). Nine forms fit on a standard 8 1/2 x 11 sheet of paper, thereby keeping printing expenses to a minimum.
Items needed to conduct the evaluation are low cost and easily obtainable. Materials needed include (1) Brief VAS Measure for Youth, (2) small colored gift bags (one differing color for each session type), and (3) small "golf" pencils.
Prior to the youth summit, volunteers attended an orientation to familiarize them with the BCS Youth Summit timeline, the event venue, and evaluation protocol. Event evaluators were instructed about the most effective times and locations to disseminate and collect evaluation materials to maximize efficiency. Following the orientation, volunteers organized Brief Analog Measures for Youth into color-specific stacks of twenty for easy dissemination. Each session's VAS were printed on different colored paper (i.e., Job Skills on blue paper, Safe Places in yellow paper, Healthy Start on white paper, and Ready for 21 on green paper). This enabled evaluators to quickly conduct visual checks to prevent cross-session data contamination.
On the day of the event, youth were handed the VAS upon entering each session. Disseminating VAS at the beginning of each session enabled evaluators to conduct a quick count of the number of evaluation forms distributed. At the conclusion of each session, youth were reminded to complete the VAS and submit it upon exiting the room. Participants were instructed to drop the completed VAS form into the colored bags, which were held by the evaluator(s) stationed in strategic locations. After all VAS forms were collected, the collection bags were sealed and labeled (see Table 2). Participants were then physically directed to the next session. Comparing the number of VAS forms disseminated with the number of forms collected enabled evaluators to simply calculate evaluation completion rates.
All analyses were performed in SPSS (version 16). Frequencies were calculated for participant perceptions of learning and enjoyment. Crosstabulations were conducted to identify frequencies of these perceptions for each session. A Spearman's rank correlation coefficient was calculated to identify the strength and direction of the relationship between these ordinally scaled perceptions of learning and enjoyment.
PARTICIPATION AND COMPLETION RATES
VAS's were disseminated to all 289 participants prior to each summit session for a total of 884 VAS's distributed across all sessions. Of the 289 middle school students who attended the BCS Youth Summit, a majority completed the brief VAS measure for each session. Evaluation completion rates ranged from 73.01% for the Safe Places session to 93.43% for the Ready for 21 session (see Table 3).
GOAL ACHIEVEMENT RATES: LEARNING AND ENJOYMENT
Most participants reported having learned "some" or "a lot" during the Job Skills, Healthy Start, Ready for 21, and Safe Places sessions (82.3%, 86.9%, 95.9%, and 96.2%, respectively). Further, a majority of students reported having enjoyed the sessions "some" or "a lot" (75.5%, 84.8%, 90.3%, and 94.3%, respectively). Table 4 provides a detailed description of the session-specific learning and enjoyment as reported by participants.
The VAS only included 2 items, measuring two independent concepts (i.e., learning and enjoyment). Although it is inappropriate to assess the internal reliability consistency of this 2-item instrument, a correlation coefficient was calculated to determine if perceived learning was related to enjoyment. A strong positive association was identified between perceptions of learning and enjoyment ([rho] = 0.69, p < 0.001).
The internal validity of the VAS (i.e., determining if the VAS accurately measured what the evaluators intended it to measure) was confirmed through the use of behavioral observations (McKyer, Outley, & Smith, 2009). Evaluators used a systemic approach to observe youth during sessions, and resulted in consensus that participants generally enjoyed the sessions and were generally engaged for the duration of the sessions. Behavioral observations were more favorable for reported engagement and enjoyment by participants in the Ready for 21 and Safe Places sessions than the Job Skills session (which was consistent with the self-report data collected with the VAS).
DISCUSSION AND CONCLUSION
Perceptions persist regarding the disparate concerns of researchers and practitioners, particularly as they relate to balancing the needs of program evaluation. Researchers seek measurement validity, and although practitioners appreciate this need for validity, they must also deal with the realities of potential chaos commensurate with large-scale events involving youth. The authors conclude the VAS meets these oft competing needs, demonstrating that feasibility and measurement quality are not mutually exclusive conditions. The VAS protocol resulted in an extremely high completion rate, yielded no data collection errors, enabled investigators to calculate participation and response rates, and afforded investigators and event coordinators the ability to gauge program effectiveness on youth learning and enjoyment.
There are limitations to this tool. The current VAS included two items intended to globally capture two concepts: youths' perceived learning and enjoyment. Two items, particularly in the case where one item is used to measure each concept, is usually inadequate to establish measurement validity as determined with measures of internal reliability consistency. While using solely two items was appropriate for the BCS Youth Summit, the authors recommend that additional items be added in future administrations of this VAS (i.e., at least two items per concept) in order to improve the quality of the measures. This recommendation is accompanied, however, with a warning to refrain from adding too many items, and thereby defeating the purpose of a brief and quickly completable evaluation tool.
The formative application of a Brief VAS Measure in a large youth-focused setting has allowed the authors to illustrate and highlight both the benefits and limitations to its use (see Table 5).
A major strength of the VAS is its visual analog format. Research has shown that pictorial representations (i.e., visual analog scales) provides children and adolescents an easy opportunity to match their internal feelings concerning a specific context or state (Chambers and Craig, 1998), albeit most are used in clinical and medical settings. From the lessons learned during the BCS Youth Summit evaluation process, the authors conclude the visual analog format is appropriate for large-scale events as well.
A strength of the BCS Youth Summit evaluation design and protocol was its ability to triangulate other measures with the VAS. In this case, the use of behavioral observations enabled the Youth Summit Evaluation Team to effectively triangulate the accuracy of the VAS to measure outcomes associated with the youth summit's effectiveness (e.g., enjoyment). The high completion rates obtained from this exercise serves as additional support to the VAS's utility, as well as emphasizing the importance of careful advanced planning by the evaluation team. The completion rates were easily computed because the summit evaluation protocol afforded evaluators to efficiently track and compare the number of distributed VAS to the number of completed/collected VAS.
The utilization of the VAS as an evaluation tool demonstrates critical lessons. First, an extremely simple data collection method (such as the VAS) is a feasible, easy, and efficient means of collecting evaluation data. Second, the VAS's visual format, brevity of measure, and the data collection plan maximized the participants' completion of evaluation materials, ensured participants' comprehension of evaluation items regardless of literacy level, and enabled evaluators to accurately track participants' compliance without disrupting transition between sessions. Finally, color coding the VAS and strategically positioning members of the evaluation team during data collection was instrumental in eliminating potential summit fidelity issues and ensuring the accurate calculation of evaluation completion rates.
This VAS tool shows great utility and potential for use in an array of social contexts (studies or fieldwork) involving diverse participants of varying verbal or cognitive abilities. The ultimate usefulness of the brief VAS measure is dependent upon its application and protocol fidelity. The inexpensive nature and simplicity of the measure makes this, and similar, VAS a desirable research tool in most settings and contexts. The authors encourage other interested researchers to use the scale in their particular fields of interest.
America's Promise Alliance (1997). Available online at: http://www.americaspromise.org.
Breitbart, M. M. & Kepes, I. (2007). The Youth Power story: How adults can better support young people's sustained participation in community-based planning. Children, Youth and Environments, 17(2), 226253.
Cahill, C. (2007). Doing research with young people: Participatory research and the rituals of collective work. Children's Geographies, 5(3), 297-312.
Cahill, C. & Hart, R. A. (2007). Re-thinking the boundaries of civic participation by children and youth in North America. Children, Youth and Environments 17(2): 213-225.
Chambers CT, & Craig KD. (1998). An intrusive impact of anchors in children's faces pain scales. Pain, 78, 27-37.
Checkoway, B., Dobbie, D., & Richards-Schuster, K. (2003). Involving young people in community evaluation research. Community Youth Development Journal, 4, 7-11.
Checkoway, B. & Richards-Schuster, K. (2004). Youth participation in evaluation and research as a way of lifting new voices. Children, Youth and Environments, 14(2), 84-98
Checkoway, B. & Richards-Schuster, K. (2003). Youth participation in community evaluation research, American Journal of Evaluation, 24, 21-33.
Dworkin, J. B., Larson, R., & Hansen, D. (2003). Adolescents' accounts of growth experiences in youth activities. Journal of Youth and Adolescence, 32, 17-26.
Ginwright, S., Noguera, P., & Cammarota, J. (eds.) (2006). Beyond resistance! youth activism and community change: New democratic possibilities for practice and policy for America's youth. New York: Rout ledge.
Goodyear, L. & Checkoway, B. (2003). Establishing the importance of youth participation in community evaluation research. Community Youth Development Journal, 4, 5.
McKyer, E. L. J., Outley, C., & Smith, M. L. (2009). The use of behavioral observations for large youth focused events. Unpublished manuscript.
Smith, T., Genry, L., & Ketring, S. (2005). Evaluating a youth leadership life skills development program. Journalof Extension, 43(2). Available at: http://www.joe.org/joe/2005april/rb3.shtml.
Proescholdbell, S. K., Scott, S. A., & Placona, M. L. (2005). Above and beyond "it was great": Evaluation of youth summits using innovative methods. National Conference on Tobacco or Health, Chicago, IL.
E. Lisako J. McKyer, PhD, MPH, is an Assistant Professor and Director of Child & Adolescent Health Research Lab, Department of Health & Kinesiology, Texas A&M University. Corliss Outley, PhD, is an Assistant Professor and Faculty Affiliate of Youth Development Program, Department of Recreation, Park & Tourism Sciences, Texas A&M University. Matthew Lee Smith, PhD, MPH, CHES,is a Research Associate and Faculty Affiliate of Child & Adolescent Health Research Lab, School of Rural Public Health, Texas A&M Health Science Center. Please send all correspondence to E. Lisako J. McKyer, PhD, MPH, Assistant Professor and Director, Child & Adolescent Health Research Lab, Department of Health & Kinesiology, Texas A&M University, 4243 TAMU, Dulie Bell Bldg, Rm 217, College Station, TX 77843-4243. Phone: 979-8459280. Fax: 979-845-4941.
Table 1. BCS Youth Summit Sessions and Descriptions Youth Summit Description & Purpose Session Ready for 21 Introduce participants to caring adults working in professions of interest to youth. Adults were invited to attend and discuss their job responsibilities and day-to-day work Healthy Start Teach participants the importance of nutrition quick, healthy, and low-cost snacks. Participants were provided a snack while learning the benefits of low fat diets, consuming recommended portion sizes, and Job Skills Introduce effective education to participants by focusing on the concept of personal finances and techniques to manage their money. Lecture, discussion, and a board game were used to allow participants to apply knowledge and skills learned. Safe Places Teach participants the importance of internet safety and skills to identify internet threats and avoid internet predators. Participants were led through discussions by facilitators to identify inappropriate content in Facebook profiles and why the inclusion The Call Introduce the concept of service learning to the participants and provides an avenue for opportunities to help others. This session encouraged participants to volunteer Table 2. Youth Summit Session Logistics Ready Safe Healthy Job for 21 Places Start Skills Number of Rotations 1 3 3 3 Number of Classrooms 1 6 1 2 Total Sessions 1 18 3 6 Number of Room Exits Per Room 3 1 1 1 Total Number of Bags Needed * 3 18 3 6 Paper & Bag Color White Blue Green Yellow * For Dual-wide doors, may want to have a bag on each side of the door. Thereby double the number of bags per door. Table 3. Youth Summit Session Evaluation Completion Rates Evaluation Completion Rates (n = 289) Completion Rates (%) Job Skills 73.34 Healthy Start 84.43 Ready for 21 93.43 Safe Places 73.01 Table 4. Reported Learning and Enjoyment Levels by Youth Summit Participants Session Frequencies: I learned during this session. A Lot Some Not A Lot Missing Job Skills 89 (42.0%) 85 (40.1%) 30 (14.2%) 8 (3.8%) Healthy Start 154 (63.1%) 58 (23.8%) 19 (7.8%) 13 (5.3%) Ready for 21 182 (67.4%) 77 (28.5%) 5 (1.9%) 6 (2.2%) Safe Places 170 (80.6%) 33 (15.6%) 2 (0.9%) 6 (2.8%) Session Frequencies: I enjoyed this session. A Lot Some Not A Lot Missing Job Skills 68 (32.1%) 92 (43.4%) 46 (21.7%) 6 (2.8%) Healthy Start 146 (59.8%) 61 (25.0%) 22 (9.0%) 15 (6.1%) Ready for 21 161 (59.6%) 83 (30.7%) 10 (3.7%) 16 (5.9%) Safe Places 151 (71.6%) 48 (22.7%) 3 (1.4%) 9 (4.3%) Table 5. Strengths and Limitations of the Brief VAS Measure Strengths Easy to administer The protocol utilized is very easy to oversee, quick to complete, required very little instruction and did not disrupting summit sessions By providing simple instructions and materials High completion to volunteer data collectors and the youth rate participants resulted in very low response rates. The addition of the colored coded sheets and bags for collection increased the VAS measure visibility/acknowledgment for completion as well Less recall bias By conducting the evaluation immediately after each session participants did not have to recall their perceptions of the sessions The only expenses included paper, copying Inexpensive and collection bags and can be viewed as being less expensive than other measurements. This is especially true given that a typical 8.5 X 11 inch sheet could contain 4-5 VAS measures Can calculate The ability to determine the exact number response rates of VAS measures disseminated and collected is vital in program evaluation. The protocol used for this study enabled the researchers to increase the response rate Ability to assess The use of "behavioral observations" internal validity enabled evaluators to triangulate data to confirm the internal validity of data collected with the VAS Used for varying The scale is beneficial for use in low literacy levels literacy or non-English speaking participants Limitations 3-point Likert- The use of the 3-point scale does not type scales provide as much variance in responses as a 5 or 7-point scale. Such that some levels of psychological states or perceptions are not easily captured Session likes vs. The VAS measure focused only on whether Program or not participants enjoyed the sessions, objectives whether or not the summit met its programmatic goals. The addition of two or more questions on the form could/should be expanded to specifically measure program objectives No comparison Due to the one-time event the ability to compare group (internal with other participants was not included. and external The level of validity could not be measured validity) The intent of the Youth summit is to No pre/post-test introduce America's Promise 5 promises to of behavioral the participants. Given the event logistics change the incorporation of a pre-test/post-test to measure knowledge and behavioral intent change was not possible. In such, that there is limited knowledge of how the event changed behavior of the participants Lack of The VAS measured was designed to be very demographic data brief and did not include information about school and/or sex for more comparison in analysis
|Gale Copyright:||Copyright 2009 Gale, Cengage Learning. All rights reserved.|