The implications of policy pre-post test scores for street-level bureaucratic discretion.
Abstract: Substantial reductions in audit error rates observed over the past few years suggest eligibility workers have moved toward an eligibility compliance culture described by Bane and Ellwood. However, the results of this study indicate that social service caseworkers responded correctly to 49% of the targeted policy items at the pre-test stage and 68% at the post-test stage. Such findings provide preliminary support for the hypothesis that, in instances when caseworkers lack policy knowledge, they use their own discretion. Such a finding not only supports Lipsky's theory but also supports the notion that administrators should be encouraged to utilize 'mastery learning' procedures whereby caseworkers are retained in new-hire and follow-up training classes until they have mastered 100% of targeted policy information. Retention of caseworkers may also reduce federal and local audit errors and errors in crediting the reduction of caseloads to social service policies when in fact significant components of them have not been implemented (learned or utilized). And, most importantly, retention in training classes may increase the appropriate provision of services to the needy.
Subject: Social service (Management)
Social service (Analysis)
Services industry workers (Compensation and benefits)
Author: Dorch, Edwina L.
Pub Date: 09/22/2009
Publication: Name: Journal of Health and Human Services Administration Publisher: Southern Public Administration Education Foundation, Inc. Audience: Academic Format: Magazine/Journal Subject: Government; Health Copyright: COPYRIGHT 2009 Southern Public Administration Education Foundation, Inc. ISSN: 1079-3739
Issue: Date: Fall, 2009 Source Volume: 32 Source Issue: 2
Topic: Event Code: 200 Management dynamics; 280 Personnel administration Computer Subject: Company business management
Product: Product Code: 9105130 Social Service Support Programs NAICS Code: 92313 Administration of Human Resource Programs (except Education, Public Health, and Veterans' Affairs Programs)
Geographic: Geographic Scope: United States Geographic Code: 1USA United States
Accession Number: 250032042
Full Text: Friendly visitors and settlement house workers (both early versions of our current social services caseworkers) enjoyed virtually unfettered discretion about the types of services they provided to clients as well as the methods they used to determine who was deserving of those services and who was not. And, in 1980 Lipsky introduced the term "street level bureaucrats" to describe workers who interact directly with citizens and who have substantial discretion in the execution of their work. These were social workers, teachers, law enforcement officers and officials, and similar public employees who grant access to government programs and provide services within them. Such workers generally have a great deal of latitude in determining precisely what their jobs entail. Rather than simply implementing policies formulated by federal legislators, Lipsky argued that street level bureaucrats, in effect, create policies through the exercise of their discretion during service delivery.

However, contrary to Lipsky, Bane and Ellwood (1994) argued that the administration of Aid to Families with Dependent Children (AFDC) was fundamentally transformed between 1970 and 1980 from one "characterized by discretion and a highly personalized relationship between caseworker and client into an impersonal system for verifying eligibility and writing welfare checks" (p. 15). They state that concerned and involved caseworkers were replaced by eligibility specialists" or "eligibility technicians" and that these new job titles reveal altered and narrower job definitions i.e., in more recent history, eligibility technicians simply accept applications, verify information and monitor clients for their continuing welfare eligibility.

Maynard-Moody and Musheno (2003), taking a position somewhere between Lipsky and Bane and Ellwood, observed that the discretion exercised by street-level bureaucrats does not imply absolute autonomy: Specifically, they state, "[P]olicies, rules, and administrative oversight pervade their work and are ever present in their calculations about what to do" (p. 4). At times, they note, "[L]aw, public policy and agency procedures provide a good match with the street level worker's views of the people they encounter. At other times tension intrudes between workers' inclinations and the dictates of the law. When policy is ill-matched with caseworkers' views of fairness or appropriate action, caseworkers experience conflict over what is the right decision and what is the right thing to do (p. 9) but this tension is resolved by the caseworkers' sense of a clients deserving character.

A review of street level bureaucrat literature reveals that Lipsky, Bane and Ellwood and Maynard-Moody et. al. all assume that workers have been trained and have learned social service policy but (in an ambivalent or even obstructionist fashion) decide to ignore it and to use their own discretion. However, it is possible that caseworkers have been trained but have not learned social service policy and therefore: utilize their own discretion; do poorly on federal and local audits and incur financial penalties; welfare reform policy is not a causal agent in the reduction of social service caseloads in the last few years;. For theoretical, administrative, political and financial reasons it is important to determine whether caseworkers do indeed learn significant amounts of targeted policy information.

As a point of clarification, this study does not attempt to examine caseworker discretion itself, but rather, attempts to determine whether caseworkers learn significant amounts of targeted social service policy while in policy training. Establishing amount of caseworker policy knowledge post policy training is considered a preliminary step in demonstrating that, caseworkers utilize their discretion in lack-of-policy information circumstances.

The next section more formally describes the relationship between social service caseworker training, knowledge and associated federal and local policy audits and penalties.

According to the U.S. General Accountability Office, GAO, (2005), there were an estimated $1.4 billion of errors in fiscal year 2003--76% of which were due to overpayments and 24% due to underpayments. By fiscal year 2005, the monetary value of errors had risen to an estimated $1.7 billion (U.S. General Accountability Office 2007)

Some version of the current Food Stamp Program (the nation's primary defense against hunger) has been in place since 1963 (U.S. Department of Agriculture, Food and Nutrition Service, 2008). The present program (renamed the Supplemental Nutritional Assistance Program, effective October 2008) was re-authorized by the Food Security and Rural Investment Act of 2002. Households that have income below 130% of the poverty line and possess few assets qualify for food stamps. For FY 2006, the individual participation rate of eligible individuals was estimated to be 67.3%, with a household participation rate of 62.7%, down from the highest participation rates in 1994 of 74.8% for individuals and 69.6% for households (United States Department of Agriculture, Food and Nutrition Services, 2008).

Since the Food Stamp Act of 1977, the federal government has exacted some types of financial penalties for states with high error rates. Because food stamp benefits are fully funded by the federal government but administered by the states, the federal Food Stamp Program Quality Control system (United States Department of Agriculture, Food and Nutrition Services, 2008) oversees the accuracy of state agencies' food stamp payments. State and federal auditors check a sample of a state's food stamp caseload to determine whether these households have received too much or too little in food stamps. From the sum of the errors found, compared to the total food stamp benefits issued, the auditors produce an overall state error rate. Under provisions of the 2002 Food Stamp Authorization Act, each state's error rate is compared to the national average error rate. States with error rates above the national average are subject to fiscal sanctions. This means that half the states every year are subject to penalties based on a sliding scale that penalizes most sharply those with the highest error rates. According to the GAO (2007) the error rate declined by approximately 40% during the period from 1999, when it was 9.86%, to 2005, when it reached an all-time low of 5.84%.

This case study focused on the efforts of a large metropolitan Northeastern city to train its food stamp workers in order to reduce the error rates in that city. Errors in food stamp administration can include erroneous denials, reduced benefits to eligible families, and awarding benefits to ineligible applicants (U.S. General Accountability Office, May 2005). Additional errors result from computer and technological glitches-inputting wrong computer codes, failure to enter coded information, or to enter it in a timely fashion, and so forth. The error rate for the study state was less than the national average in FY 2003 (U.S. General Accountability Office, 2005). The error rate for the study city was slightly lower than that for the state as a whole.

At the time of the study, there were more than a half dozen specialized Food Stamp Centers throughout the City and hundreds of eligibility workers serving thousands of food stamp recipients. The City intake process began when a person seeking financial assistance approached a receptionist at a Job Center. After turning in the paperwork, an eligibility worker determined whether the applicant's request could be met by providing a 'one-time' assistance grant. Regardless of the decision, all applicants were finger printed by the Automated Fingerprint Imaging System to confirm their identity and prevent fraud and the subsequent Eligibility Verification and Review process included a review of income, assets, residency, and family composition, wages and other income reports from banks and credit reporting agencies.

The training focused on the eligibility verification and review process. The objective was to ensure that food stamp eligibility workers had adequate knowledge in certain error-prone policy areas in order to comply with federal food stamp policy, to avoid potential financial penalties imposed on those states with higher than average error rates and to reduce eligibility worker's reliance on discretion in policy areas wherein their lacked knowledge .

State Audit

The bureau of audit and control at the State agency responsible for administration of the Food State program annually conducts an audit of program performance, which is submitted to the federal government for verification. A summary of the analysis for the city for fiscal year 2004 appears in Table 1.

The City error rate for that period was estimated to be 3.92% (down substantially from a high of 13.9% in fiscal year 1998.) The analysis suggests that approximately three quarters of the identified errors are due to the agency, whereas about a quarter of the errors are due to the client not reporting information. Errors relating to three factors -wages and salaries; household composition; and SSI benefits--accounted for roughly two-thirds of the errors, whether measured in terms of either the gross number of errors or the dollar costs associated with them.

The Relation between Pre-Post Test Items and Sources of Error Identified by Audit

The type of error that each item in the pre-post test addresses is identified in the leftmost column of Table 2, below. (Table 4 displays the correspondence between those sources of error addressed in the training and those identified by the audit.) According to the audit, errors related to wages and salaries accounted for 34% of the total monetary costs; this error was addressed by 50% of the items on the pre-post test. Likewise, household composition accounted for 23.1% of the total monetary costs and was addressed by 30.4% of the pre-post items. Although errors associated with SSI benefits accounted for an estimated 9.6% of the monetary errors identified by the audit, this topic was not addressed in the pre-post test.

METHODOLOGY

The Research Hypotheses

Lipsky indicated that determining whether there is a relationship between what superiors seek and what subordinates do involves the development of four capabilities. Specifically, he indicated:

* Agencies must know what they want workers to do.

* Where the objectives are multiple and conflicting, agencies must be able to rank their preferences.

* Agencies must know how to measure workers' performance.

* Agencies must be able to compare workers to one another to establish a standard for judgment (Lipsky 1980, 161).

In Lipsky's terms, the amount of the audit errors associated with each policy area provided a "rank of [training curriculum] preferences", the pre-post tests items were used to "measures worker's performance" (both individually and as a group and both before and after the policy training). Additionally, the test scores allowed workers to be compared to one another.

Research Design: The gold standard of social science research is true experimental designs in which members of the study population are randomly assigned to experimental programs or a control group. A variety of political, institutional, and organizational obstacles typically make it impossible to achieve the experimental ideal in social welfare research. Such was the case in the present study. Thus a pre-post test pre-experimental design with no control group (Campbell and Stanley 1963) was used. This type of design lacks randomized selection and assignment of trainees hence a variety of alternative explanations for pre-post test results cannot be ruled out.

The Training Program

As part of an effort to reduce the number of food stamp errors and to avoid potential federal financial penalties, a two-day food stamp training was implemented to provide training to food stamp eligibility workers throughout the city.

The Training Curriculum

The State Food Stamp Policy Manual (created by the state contractor's curriculum development writers) was used in the morning sessions. The City Food Stamp Policy Manual was used in the afternoon. Morning trainers focused on material associated with items 1-to-8 on the pretest and the afternoon trainer focused on material associated with the other items. The training manual created by the state training contractor was divided into several sections. Section topics include client civil rights, client responsibilities, expedited services, authorized representatives, the client interview, the verification process, the clearance report, definitions of able bodied adults, work requirements, work exemptions, self employment, six month and 10-day re-certifications, fraud, resource determination, resource limits, transferable resources, income limits, earned and unearned income, shelter and utility cost and vehicle value, disqualifications and penalties of active and closed cases. The trainers themselves (not the curriculum writer or the state officials) decided the information they would cover and they focused on only targeted topics contained in their training manual. The targeted items matched the high audit error items.

The Trainers

A team of three food stamp trainers were used in the morning. These trainers were employees of the training agency contracted by the state. Two of these trainers had been training for several years but one was newly hired. There were dozens of other trainers at the agency but these were the ones assigned to training food stamp workers in the City. The one City trainer (who had been training for several years) provided training in the afternoon. She was one of dozens of trainers in the City's training department.

The Trainees

Three hundred ninety two Food Stamp Eligibility Workers from 23 Food Stamp centers throughout the city participated in training sessions during the period December 2005 through March 2006. At that time there were approximately 1,200 eligibility workers with various titles and various specialty areas (food stamp, Medicaid etc). Trainees were notified of possible training days and they self-selected the days they attended.

PROCEDURE

Twelve training sessions with approximately 20 trainees per session were provided. Within the first hour of training, employees were greeted, introductions were made, an overview was presented, and trainees took a pre-test consisting of 23 items. The questions and the correct answers to them are presented in Table 2. Trainees utilized their memory and were not allowed to look at the handbook. Trainers presented information through a lecture format supported by the use of Power Point presentations. Trainees were provided with notebooks containing hard copy of the presentations. Trainees took the post-test after the curriculum was presented but before the training evaluation form was distributed so they had no opportunity to learn their test scores. The post test consisted of the same 23 items as appeared on the pre-test. Trainees could ask questions at any time during the training sessions but could not ask questions or use the manual during the pre-post tests.

The Training Evaluation

The first two levels of Kirkpatrick's 4-level training evaluation model were used. These two levels were operationalized as a participant reaction survey and a pre-post test, respectively.

RESULTS

Kirkpatrick Level 1: Trainee-Reaction Survey

Trainees responded on a five-point scale (5 = Strongly Agree; 4 = Agree; 3 = Neither Agree nor Disagree; 2 = Disagree; 1 = Strongly Disagree) to nine items assessing their reaction to the training. As can be seen in Table 3, trainees' (n=395) ratings on the nine items were uniformly high, ranging from 4.43 to 4.51 on the 5point scale.

Kirkpatrick's Level 2: Pre-Post Test

Trainees responded to a 23-item test intended to assess their knowledge of and ability to apply relevant policies both before and after training.

The three right most columns of Table 2 give the percent answering correctly on the pre-test and on the post test, as well as the change from in percentage from pre to post test. The pre test group mean was 49.4 and the post test group mean was 68.9. Random responses would have lead to an expected score of 35.6%. The trainees did well on the pre-test for some items; more than 80% of trainees gave the correct answers to five test items (i.e., test items 8, 12, 17, 20 and 23). The trainees performed poorly on some items on the post-test; after training, fewer than 60% of trainees gave the right answer on six items (i.e., test items 2, 13, 15, 18, 19, and 22). The percentage of positive change from pre-test to post-test was significant at the .01 level for 14 of the 23 items. For one item (#19) the percentage answering correctly was significantly lower on the post-test than on the pre-test. Finally, there was a substantial range in trainees' level of performance even after training; individual post-test scores ranged from 21.7% (5 of 23 correct) to 95.7% (22 of 23 correct).

DISCUSSION

Implications for the Street Level Bureaucrat Literature

Judging from the substantial reductions in audited error rates observed over the past few years, researchers have assumed a 'compliance' caseworker culture as described by Bane and Ellwood resulting from a combination of factors including automated information processing systems, administrative incentives, state and local audit feedback and policy training. However, the results of this study indicate that caseworkers leave training lacking knowledge of targeted policy information identified by federal audits results. Such a finding provides one reason for why caseworkers utilize their own discretion at some times versus others. Furthermore, to the extent that noncompliance is provoked by a simple lack of knowledge or ignorance of program policy, retaining new-hires in training academies until they have mastered targeted policy information may reduce the probability of their use of discretion.

Implications for Policy Implementation: Caseworker Knowledge

In terms of Kirkpatrick's Level 1, trainees indicated that the course objectives were very clear; the knowledge and skills gained from the course would help them perform their job more effectively; the materials and handouts were helpful. However, the percentages of incorrect responses on the post-test did not corroborate the trainees' rosy perceptions of the quality and effectiveness of the training. Such contradictory results suggest that it is probably unwise to use reaction surveys alone to determine the amount of knowledge learned.

In terms of Kirkpatrick's Level 2, trainees responded correctly to 49% of the policy items at the pre test stage and 68% at the post test stage. If the pre-post test items corresponded perfectly with the audit items, the city would have been able to determine the percent of audit errors that remained unknown to their trainees. But the city did not seek to obtain any correspondence between the two.

More than 80% of trainees responded correctly to 5 of the 23 items at the pre test stage. Generally there is little reason to include items that the vast majority of workers already know in an assessment of training effectiveness. Some of these items, however, may tap knowledge that is so essential to worker competence that they necessarily will remain a key element of the curriculum. Whether such items ought to be included in pre-post-test depends on the purpose of the instrument. For example, it may make sense to include such items if the purpose of including them is primarily to assess how much policy content trainees know. Including such items may be less useful, however, if the purpose is to assess the degree to which training increases the amount of trainee knowledge of policy information. When most trainees respond correctly to a specific pre-post test item but the audit reveals errors in that domain, this may be because the item does not fully capture federal policy or because the error is caused by something other than a lack of knowledge e.g., a lack of time, or a large caseload.

Implications for Policy Implementation: Training Curriculum and Testing

Fewer than 60% of trainees gave the correct response for six items on the post-test and there was less than a 10% improvement in correct response rate on 10 of the 23 items from pre to post test. It is unclear why so few trainees would respond correctly post training. Thus, it is important to determine whether written curriculum was omitted in the oral presentation, not communicated effectively, communicated incorrectly and whether State regulations conflicted with city regulations and was thus out-dated or trainees are poor test-takers.

Trainees' post-test scores ranged from 21.74% to 95.65% correct. Such a wide range is probably reflective of the wide range of trainee work experience i.e., some workers have been on the job six months and some for over twenty years. The premise of 'mastery learning' (Bloom, 1984) is that all trainees can master required knowledge or skill if given sufficient time and appropriate training methods. If so, training sections could engage in mastery learning training to reduce the use of caseworker discretion.

Implications for Policy Implementation: Federal, State and Local Audits

After the food stamp policy training, neither trainers nor supervisors attempted to determine whether training improved trainees' processing of cases on the job. Thus, there was no attempt to assess Kirkpatrick's Level 3--transfer of learning to application. After the training exercise was completed, the city received another annual round of State and Federal Audit Reports. Thus, it received some feedback at Kirkpatrick's Level 4 of evaluation i.e., agency outcome (and in this case the audit error continued to decline). However, the city could not determine whether or to what degree the improvement was associated with the training it provided i.e., there was no attempt to determine whether the audit cases had been worked on by employees who attended the training. More specifically, the City did not use a logic model which connected their training curriculum to their trainee pre-post tests, their supervisory case review instruments or their high audit error items. Thus, the city was severely constrained in its ability to assess the degree to which training reduced its audit errors. This is unfortunate because the financial penalties associated with excessive audit errors include the training costs of addressing them, as well as the Federal fines.

CONCLUSION

Previous theorists including, Lipsky and Maynard-Moody and Musheno et. al. have proposed theories assuming caseworkers have learned targeted policy information and policy implementation researchers (including welfare reform researchers) have assessed policy outcomes as if the causal variable (utilization of policy) was operative. This study indicates that (in this large northeastern city with numerous social service caseworkers) the average caseworker does not know substantial amounts of targeted policy information. Hence, this study provides support for the hypothesis that in some instances, caseworkers could use personal discretion because of they lack policy knowledge.

Specifically, it is possible that caseworkers seek information regarding approved rules and regulations from their supervisors or other informed sources. But, it is also possible they don't and that (at least in some percent of instances) they use their own discretion, because it's expedient. Given the expediency of using one's own discretion, one way to lessen caseworker discretion in lack-of-knowledge situations retain new-hires in training academies until they have master 100% of targeted policy information.

This study also provides support for the idea that the reduction of welfare caseloads (in this city at least) may not be solely a result of the new welfare reform policy since caseworkers in this study lacked substantial knowledge of targeted policy information at the beginning and the end of their training. Moreover, this study provides support for the idea that there should be a correspondence between audit errors, training curriculum, pre-post tests items and supervisor review items and that training administrators use a logic model to obtain feedback loop a connection between the four.

Street level bureaucrat theorist, welfare reform researchers, policy implementation researchers and federal policy auditors all start with the assumption that caseworkers have been trained and have learned substantial portions of targeted social service policy information. The results of this study cause us to question such an assumption and ask calls for action.

REFERENCES

Bane, M. J., & Ellwood, D. T. (1994). Welfare Realities: From Rhetoric to Reform. Cambridge, Massachusetts: Harvard University Press.

Bane, M., & Ellwood, D. (1986). Slipping Into and Out of

Poverty: The Dynamic of Spells. The Journal of Human Resources, 21, 1-23.

Bloom, B. (1984). Taxonomy of Educational Objectives. Allyn and Bacon, Boston, MA.

Campbell, D., & Stanley, J. (1963). Experimental and Quasi-Experimental Designs for Research. Chicago: Rand-McNally.

Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler Publishers.

Lipsky, M. (1980). Street Level Bureaucracy: Dilemmas of the Individual in Public Services. New York: Russell Sage Foundation.

Maynard-Moody, S., & Musheno, M. (2003). Cops, Teachers, Counselors: Stories from the Front Lines of Public Service. Ann Arbor: University of Michigan Press.

Meyers, M. K., Glaser, B., & Mac Donald, K. (1998). On the Front Lines of Welfare Delivery: Are Workers. Journal of Policy Analysis and Management, 17, 1-22.

U.S. Department of Agriculture, Food and Nutrition Service. (2008, September 28). Supplemental Nutrition Assistance Program--A Short History of the Food Stamp Program. Retrieved October 26, 2008, from http://www.fns.usda.gov/FSP/rules/Legislation/about_f sp.htm

U.S. General Accountability Office. (2007). Food Stamp Program: Payment Errors and Trafficking Have Declined despite Increased Program Participation. Washington, D.C.: U.S. General Accountability Office.

U.S. General Accountability Office. (May 2005). Food Stamp Program: States Have Made Progress Reducing Payment Errors, and Further Challenges Remain. Washington, DC: U.S. General Accountability Office.

United States Department of Agriculture, Food and Nutrition Services. (2008, September 30). Quality Control. Retrieved October 27, 2008, from Supplemental Nutrition Assistance Program: http://www.fns.usda.gov/fsp/qc/default.htm

United States Department of Agriculture, Food and Nutrition Services. (2008, June). Trends in Food Stamp Program Participation Rates: 2000 to 2006. Retrieved October 26, 2008, from Current Perspectives on Food Stamp Program Participation:

http://www.fns.usda.gov/oane/MENU/Published/snap/F ILES/Participation/Trends2000-2006.pdf.

This work was performed while Professor Dorch was a Research Scientist at Rockefeller College of Public Affairs and Policy, University at Albany, State University of New York (SUNY), 135 Western Avenue, Albany, New York 12222

EDWINA L. DORCH

Texas A&M University
Table 1 Results from State Quality Control Error Audit

Estimated City Food Stamp Quality Control Error Rate for Fiscal Year
2004: 3.92%; n = 639 cases

Most Frequent Causal Factors for Errors :                     % Total
  Agency incorrectly applied policy                             32%
  Agency disregarded or did not apply known information         29%
  Client did not report information                             18%
  Other causes                                                  21%

Type of Error:      Attributable    Attributable    Total    % Dollar
                      to Agency      to Clients                Value

1. Wages &                6               6           12       34.0%
Salaries

2. Household              6               3           9        23.1%
Composition

3. SSI Benefits           5               0           5        9.6%

4. Shelter                3               1           4        6.1%
Deduction

5. Citizenship/           3               0           3        6.1%
Non-Citizenship
Status

6. Standard               4               0           4        5.8%
Utility
Allowance

7. Employment &           0               1                    3.9%
Training
Programs

8. Time Limited                           0                    3.1%
Participation

9. Child Support                          0                    2.0%
Received

10. Self-                                 0                    2.0%
Employment

11. TANF, PA or                           0                    1.8%
GA

12. RSDI                                  0                    1.4%
Benefits

13. Unemployment                          0                    1.2%
Compensation

TOTAL:                   34              11           45

Percentage               76%             24%         100%     100.0%

Source: Adapted from Bureau of Audit and Quality Control, Agency
Responsible for Food Stamp Program in Study State, 2005

Table 2 Percentage of Food Stamp Trainees' Correct Responses on
Pre-Post Test

                                                   Percent Answering
                                                      Correctly

               Test items: (correct answers in
               bold)                              Pre    Post   Chang.

Error type
addressed      Multiple-choice (4-part):

Household      1. Melody Harris and her younger
composition    daughter, Donna, buy and prepare   49.4   73.    24.0 *
               most of the meals for the Harris          4
               family. As a result, Melody
               indicates on the application
               that her husband, Charles, and
               her son, Robert, do not purchase
               and/or prepare food. Are Charles
               and Robert part of the Food
               Unit?

               a) No. The information on that
               part of the application is
               correct.

               b) Yes. The information on that
               part of the application is
               correct.

               c) No. The information on that
               part of the application is
               incorrect.

               d) Yes. The information on
               that part of the application is
               incorrect.#

Household      2. Genoa Venice is part-time       23.1   48.7   25.6 *
composition    housekeeper and babysitter for
               the Harris family. In exchange
               for her services, she receives
               room and board. Genoa also
               works part-time at a local
               beauty salon. What is Genoa's
               status in the household
               according to Food Stamps?

               a) Boarder

               b) Roomer

               c) Live-in Attendant#

               d) Mandated Household
               Member

Household      3.  Sharon Carson, Melody's        30.9   68.8   37.9 *
composition    daughter, lives with her
               husband, Harry, and daughter,
               Mary, in her parent's house. The
               Carson's maintain that they eat
               separately from the Harris
               family. The Carson family:

               a) Can have a separate FS case.

               b) Can choose to be in the
               Harris FS case.

               c) Must be in the Harris FS
               case.#

               d) Must have a separate FS
               case.

Household      4.  Sharon begins attending        48.8   62.7   13.9 *
composition    college part-time and brings her
               daughter, Mary, to the on-
               campus child care center while
               she attends class. What is
               Sharon's status?

               a) A mandated household
               member.

               b) An eligible FS student.#

               c) An ineligible FS student.

               d) None of above

Household      5. What if Sharon and Harry        53.4   87.1   33.7 *
composition    divorce and Harry moves out of
               the house. Harry gets his own FS
               case. They have joint custody of
               their daughter, Mary, who eats
               with each of them. In which case
               can Mary be a part?

               a) Her mother's

               b) Her father's

               c) Her mother's or her
               father's#

               d) Her mother's and her father's

Citizenshi     6. For the purpose of this one     53.6   67.3   13.7 *
p/Non-         question, assume Genoa Venice is
citizenship    an immigrant. Genoa's green
status         card shows she has been in the
               United States for 10 years, and
               was granted LPR status in 2003.
               What is Genoa's status?

               a) She is an ineligible alien.#

               b) She is an eligible alien.

               c) She is an undocumented
               alien.

               d) None of the above

Household      7. A child living with their       68.1   88.2   20.1 *
composition    parents is a mandated FS
               household member until the age
               of:

               a) 18   b) 19
               c) 21   d) 22#

Household      8. An LPR under 18:                86.8   91.5   4.7

composition    a) is eligible if they meet the
               five year rule

               b) is ineligible until their
               parents are eligible

               c) is eligible because they are
               under 18#

               d) is ineligible until their
               parents accumulate 40 qualifying
               quarters.

               True-False Questions:

Wages and      9. Additional funds received for   40.8   68.0   27.2 *
salaries/      training in an approved program
Employmen      are considered earned income if
t & training   they exceed the actual amount of
programs       the training program. True#

Wages and      10. Additional funds for work      58.2   71.5   13.3 *
salaries       related expenses not included as
               reimbursements that exceed
               actual expenses are considered
               as earned income. True#

Wages and      11. Assistance payments from       71.9   76.7   4.8
salaries       Federal or federally funded need
               based programs are considered
               unearned income. True#

Wages and      12. Support or alimony payments    82.9   91.4   8.5 *
salaries       made directly to households by
               non-household members are
               considered unearned income.
               True#

Wages and      13. Payments from dividends are    53.3   56.0   2.7
salaries       considered as earned income for
               Food Stamp budgeting purposes.
               False#

Wages and      14. If the alien has already       64.2   67.9   3.7
salaries /     reported  gross income
Citizenship    information on his/her sponsor
/ non-         under PA sponsorship alien
citizenship    rules, that income may be used
status         for FS purposes. True#

Wages and      15. Portions of the gross income
salaries /     of a sponsor and the sponsor's
Citizenship    spouse (if living with the
/ non-         sponsor) shall be deemed to be
citizenship    unearned income of a sponsored
status         alien for three years following
               the alien's admission for
               permanent residence to the
               United States. False#              41.5   46.7   5.2

               Fill in the correct income
               (earned, unearned, or exclusion)
               for the following:

Wages and      16. Annuities (unearned)#          65.7   75.3   9.6 *
salaries

Wages and      17. Alimony Payments               87.2   91.1   3.9
salaries       (unearned)#

Wages and      18. Training Allowance             31.8   47.9   16.1 *
salaries/      (earned)#
Employment
& training
programs

Wages and      19. Legally Obligated Child        76.9   54.4   -22.5 *
salaries/      Support (exclusion)#
Child
support

Wages and      20. Child Support (unearned)#      84.3   87.9   3.6
salaries/
Child
support

Wages and      21. Foster Care Payments           50.0   63.2   13.2 *
salaries       (exclusion)#

Wages and      22. Adoption Subsidy               38.4   58.0   19.6 *
salaries       (exclusion)#

Self-          23. Self-employment (earned)#      95.6   96.0   0.4
employment

* = p < .01, double-sided

Note: Test items: (correct answers in
bold) is indicated with #.

Table 3
Trainee Responses to Items on the Trainee Reaction Survey

Survey Items:                                            Mean    Mode

1. The course objectives were clear                      4.45      5
2. The course content supported the objectives           4.43      4
3. The knowledge and skills will help me                 4.46      5
4. The instructor was well prepared                      4.48      5
5. The instructor was knowledgeable                      4.48      5
6. The instructor has excellent communication skills     4.49      5
7. The instructor responded appropriately;               4.49      5
8. The materials and handouts were helpful               4.51      5
9. The course was of high quality                        4.44      5

Evaluation Key:

5 = Strongly Agree; 4 = Agree; 3 = Neither Agree or Disagree;
2 = Disagree; 1 = Strongly Disagree

Table 4 Percentage of Errors Attributable to Agency in Audit Compared
to Percentage of Pre-Post Test Items.

                                                       Percentage of
                                    Percentage of          Items
                                     Dollar Value        Addressing
                                   Attributable to     Error on Pre-
Type of Error                           Error            Post Test

1. Wages & Salaries                     34.0%              50.0%

2. Household Composition                23.1%              30.4%

3. SSI Benefits                          9.6%                0%

4. Shelter Deduction                     6.1%                0%

5. Citizenship/Non-Citizenship
Status                                   6.1%               8.7%

6. Standard Utility Allowance            5.8%                0%

7. Employment & Training
Programs                                 3.9%               4.3%

8. Time Limited Participation            3.1%                0%

9. Child Support Received                2.0%               4.3%

10. Self-Employment                      2.0%               2.2%

11. TANF, PA or GA                       1.8%                0%

12. RSDI Benefits                        1.4%                0%

13. Unemployment Compensation            1.2%                0%
Gale Copyright: Copyright 2009 Gale, Cengage Learning. All rights reserved.