Summative Evaluation is defined as the design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners. According to Dick and Carey (1996), the expert judgement phase of summative evaluation is used to find out if either current or candidate instruction can meet an organization's identified instructional needs. The following activities are part of the expert judgement phase of summative evaluation when reviewing candidate instruction:
- evaluating the congruence between the organization's instructional needs and candidate instruction
- evaluating the completeness and accuracy of candidate instruction
- evaluating the instructional strategy contained in the candidate instruction
- evaluating the utility of the instruction
- determining current users' satisfaction with the instruction (p. 323)
The Purpose of Summative Evaluation
- Make “go-no-go” decisions
- Keep current materials?
Field Trial
The purpose of the field trial phase of summative evaluation is to determine the effectiveness of instruction with the target group in the intended setting (Dick & Carey, 1996). There are two parts to the field trial phase: outcomes analysis and management analysis. The outcomes analysis reviews the impact of the instruction on the learner, the job and the organization. Management analysis assesses "instructor and supervisor attitudes related to learner performance, implementation feasibility, and costs" (p. 323).
Reference
Dick, W. and Carey, L. (1996). The Systematic design of instruction, 4th ed. New York: Harper Collins Publishing.
REFLECTION
Why is evaluation important? Possible reasons:
- I think without evaluation it cannot be said whether the designers have properly understood the users. Therefore, one would not know whether the whole design is intolerant of minor errors.
- It'll be difficult to know whether the systems cause disruption, frustration, unacceptable changes or conflict in organisations.
- Also, one would not know how to improve the systems in order to fit the users needs better.
- Without evaluation, alternative designs could not be compared
- Or whether systems cause a cognitive overload in users
- It would also be hard to assess whether computer systems force users to perform tasks in undesirable ways. Thus, it would be hard to check conformance to a standard.
- Engineering towards a target (often expressed as some form of metric) would be hard to achieve.
I think it is very important for us to constantly evaluate the instructional strategy ensure that is staying the course with what the organization has in mind while being usable and feasible for the organization. I think the summative evaluation does just that. Summative evaluation's existence provides monitoring and help to develop the training for future events.
As future ID, we need to design and conduct summative evaluation by looking at all four levels of Kirkpatrick. Kirkpatrick stated that Levels 3 and 4 (Behavior and Results) are important indicators of the training's value to the organization. Thus, we should not limit ourselves to only the first two levels (Reaction and Learning).
EXTENDED KNOWLEDGE
Smith and Ragan (1999) suggest to determine goals of evaluation as the first step in a goal-based summative evaluation. The most important part of this stage is determining questions that should be answered as a result of the evaluation. The client organization and/or funding agencies and other stakeholders should identify the questions. These questions will guide the remainder of the summative evaluation. Questions might include:
Does implementation of the instruction solve the problem identified in the needs assessment?
Do the learners achieve the goals of the instruction?
What are the costs of the instruction? What is the "return on investment" of the instruction? (p. 355)
Both the client and evaluator should agree on the questions before moving on to subsequent steps of summative evaluation.
Reference
Smith, P. and Ragan, T. (1999). Instructional design (2nd ed.). New York: John Wiley & Sons, Inc.
Contrasting Summative and Formative Evaluations
Summative Evaluation
• Provides teachers and students with information concerning achievement
• Has a high-point value
• End-goal is to compare student achievement at the end of an instructional unit by comparing it to a set benchmark
Formative Evaluation
• Is used to check student progress as an instructional unit is occurring
• Guides the next steps in instruction by identifying other opportunities to aid in success
• Must be designed in a way to respond to students’ needs
Summative Evaluation
• Provides teachers and students with information concerning achievement
• Has a high-point value
• End-goal is to compare student achievement at the end of an instructional unit by comparing it to a set benchmark
Formative Evaluation
• Is used to check student progress as an instructional unit is occurring
• Guides the next steps in instruction by identifying other opportunities to aid in success
• Must be designed in a way to respond to students’ needs