Whereas group designs, typically the basis for summative
   evaluation, are readily accepted as providing scientific evidence
   for program effectiveness, single subject designs typically form
   the basis for formative evaluation. While both group and single
   subject designs are descended from highly successful scientific
   traditions, and both may provide equally rigorous and informative
   results, single subject design is relatively less understood. Both
   do, however, differ in the questions asked; one asks about the
   behavior of groups, the other asks about the behavior of
   individuals.

Table. 3X3 Matrix. The level of rigor for each type of evaluation
is indicated by the letters A-C for summative evaluation, with
column C representing the most rigorous; the numbers 1-3 indicate
the level of rigor for each type of formative evaluation, with
row 3 representing the most rigorous. Cell 3-C represents the
most rigorous intersection of formative & summative evaluation.

                             Approaches to Summative Evaluation:
Approaches                      Basis for Outcomes Assessment
to Formative
Evaluation: Basis      A. Experiential--        B. Evidence Based--
for Program Revision   Assessment               Assessment

1. Experiential--      Cannot predict group     Provides some
Program Development    or individual            indication that the
                       performance,             program may be
                                                effective with a
                                                group; but

                       Works or not with        Cannot confidently
                       groups or individuals    predict group or
                       purely subjective; a     individual
                       matter of opinion        performance.
                       argued on point of
                       view--a matter of
                       social agreement

2. Evidence Based--    If limited tryouts,      Provides some
Program Development    may indicate that the    indication that the
                       program might work       program may be
                       with those tested; but   effective with a
                                                group; but

                       Cannot confidently       Cannot confidently
                       predict group or         predict group or
                       individual               individual
                       performance.             performance.

                       Still primarily a        If works, not really
                       matter of social         clear why, if it does
                       agreement, but has       not work, can lead to
                       some validity by         re-evaluation of
                       relation to past         principles or the way
                       research and perhaps     they were applied Not
                       limited tryouts.         clear where the
                                                problem is.

3. Scientific--        Able to predict group    Able to predict group
Program Development    performance based on     performance based on
                       individual               individual
                       performance; and         performance; and

                       Can confidently          Can confidently
                       predict individual's     predict individual's
                       performance.             performance.

                       Since program able to    If doesn't work,
                       predict individual's     issues are in
                       performance, some        transfer,---able to
                       prediction of group      identify and isolate
                       performance implied;     variables to change
                       may have some validity   and revise for retest,
                       by relation to past      Individual data not
                       research.                lost and can be
                                                analyzed in relation
                                                to outcome

                       Approaches to Summative Evaluation:
Approaches                Basis for Outcomes Assessment
to Formative
Evaluation: Basis      C. Scientific--Controlled Group
for Program Revision   Research & Assessment

1. Experiential--      Can confidently predict group
Program Development    performance, but

                       Cannot predict individual's
                       performance (Sidman, 1960).

                       If works or not, not clear what
                       program elements, alone or
                       together are responsible.

2. Evidence Based--    Can confidently predict group
Program Development    performance; but

                       Cannot confidently predict
                       individual's performance.

                       If works or not, not clear what
                       program elements, alone or
                       together are responsible, but
                       can lead to reconsideration of
                       principles or the way they were
                       applied.

3. Scientific--        Can confidently predict group
Program Development    performance; and

                       Can confidently predict
                       individual's performance.

                       If doesn't work, issues are in
                       differences in formative
                       criteria & summative
                       measurement instruments--able
                       to identify and isolate
                       variables to change and revise
                       criteria & program for retest,
                       or to revise summative
                       measurement instruments,
                       Individual data not lost and
                       can be analyzed in relation to
                       outcome.