The central element of systematically designed instruction is called the learner-centered assessment or criterion-referenced testing. This "type of testing is important for evaluating both learners' progress and instruction quality. Criterion-reference tests enable learners to reflect on their own performances by applying established criteria to judge their own work." Developing assessments should be parallel to learners’ ability to perform as described in the objectives. The emphasis should be on the relationship between the kind of skills described in the objectives to the assessment requirements. Below is Dick & Carey model in Developing Assessment Instruments.
Norm-Referenced vs Criterion -Referenced Tests
According to this chapter criterion-referenced test is focused on specific goals and objectives within a given content area. It's main purpose is to examine a person's or group's achievement in a carefully defined area. In contrast, norm-referenced tests are used to compare the relative performance of learners in larger areas of content. Instructional designers used the data from both type of tests. One test is not superior than the other. Norm-references tests describe learners' general achievement and ability levels using terms such above average, average, or below average in a given subject area. The data from this test are not useful for instructional design and development but they are useful for field trial groups in developing instructional materials. In contrast, criterion-referenced tests are the backbone of the assessment used for decision making in the development and evaluation of instruction.
Four Types of Criterion-Referenced Tests and their Uses
According to this chapter criterion-referenced test is focused on specific goals and objectives within a given content area. It's main purpose is to examine a person's or group's achievement in a carefully defined area. In contrast, norm-referenced tests are used to compare the relative performance of learners in larger areas of content. Instructional designers used the data from both type of tests. One test is not superior than the other. Norm-references tests describe learners' general achievement and ability levels using terms such above average, average, or below average in a given subject area. The data from this test are not useful for instructional design and development but they are useful for field trial groups in developing instructional materials. In contrast, criterion-referenced tests are the backbone of the assessment used for decision making in the development and evaluation of instruction.
Four Types of Criterion-Referenced Tests and their Uses
Using Portfolio Assessments:
Portfolios are collections of work that together represent learners’ achievements over an extended period of time. This could include tests, products, performances, essays, or anything else related to the goals of the portfolio. They allow you to assess learners’ work as well as their growth during the process. As with all other forms of assessment, whatever is included in the portfolio must be related to specific goals and objectives. The choice of what to include can be decide on entirely by the teacher, or in cooperation with students. Assessment of each portfolio component is done as it is completed, and the overall assessment of the portfolio is carried out at the end of the process using rubrics. In addition, learners are given the opportunity to assess their own work by reflecting on the strengths and weaknesses of various components. Portfolios can also be used as part of the evaluation process to determine what students did and did not learn, and then that information can be used to strengthen the instruction.
Evaluating Congruence in the Design Process:
One of the most crucial aspects of the assessment phase of the design process is to be able to evaluate the congruence of the assessment against the objectives and analyses that have been performed. Remember that this is a systematic approach to instructional design, which means that every step in the process influences subsequent steps. As such, all of your skills, objectives, and assessment items should be parallel. One way to clearly represent this relationship is to create a three-column table that lists each of the skills from your instructional analysis, the accompanying objective, and the resulting assessment item. At the bottom of the table you would finish up with your main instructional goal, the terminal objective, and the test item for the terminal objective.
Reference
Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of instruction. (7th ed., pp. 130-163). Upper Saddle River, NJ: Pearson.
Test Item Criteria
There are four categories of test item qualities to consider during the creation of test items and assessment tasks:
Portfolios are collections of work that together represent learners’ achievements over an extended period of time. This could include tests, products, performances, essays, or anything else related to the goals of the portfolio. They allow you to assess learners’ work as well as their growth during the process. As with all other forms of assessment, whatever is included in the portfolio must be related to specific goals and objectives. The choice of what to include can be decide on entirely by the teacher, or in cooperation with students. Assessment of each portfolio component is done as it is completed, and the overall assessment of the portfolio is carried out at the end of the process using rubrics. In addition, learners are given the opportunity to assess their own work by reflecting on the strengths and weaknesses of various components. Portfolios can also be used as part of the evaluation process to determine what students did and did not learn, and then that information can be used to strengthen the instruction.
Evaluating Congruence in the Design Process:
One of the most crucial aspects of the assessment phase of the design process is to be able to evaluate the congruence of the assessment against the objectives and analyses that have been performed. Remember that this is a systematic approach to instructional design, which means that every step in the process influences subsequent steps. As such, all of your skills, objectives, and assessment items should be parallel. One way to clearly represent this relationship is to create a three-column table that lists each of the skills from your instructional analysis, the accompanying objective, and the resulting assessment item. At the bottom of the table you would finish up with your main instructional goal, the terminal objective, and the test item for the terminal objective.
Reference
Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of instruction. (7th ed., pp. 130-163). Upper Saddle River, NJ: Pearson.
Test Item Criteria
There are four categories of test item qualities to consider during the creation of test items and assessment tasks:
Goal-Centered Criteria:
Test items should be congruent with the terminal and performance objectives by matching the behavior involved. What this means is that each test item should measure the exact behavior and response stated in the objective.
Learner-Centered Criteria
Test items should take into consideration the characteristics and needs of the learners. This includes issues such as learners vocabulary and language levels, motivational and interest levels, experiences and backgrounds, and special needs.
Context-Centered Criteria:
Remember the context analysis you wrote. Well, when writing test items you should consider both the performance context and the learning context . It is important to make your test items as realistic and close to the performance setting as possible.
Assessment-Centered Criteria
Test items should be well written and free of spelling, grammar, and punctuation errors. Directions should be clearly written to avoid any confusion on the part of the learner.
Dick and Carey Five Steps for Creating an Assessment Instrument:
1. Identify the elements to be evaluated. These elements should be taken directly from the behaviors and criteria included in your objectives.
2. Paraphrase each element. Also, make sure that a "Yes" response on the instrument always corresponds with a positive performance, and a "No" response with a negative performance.
3. Sequence the elements on the instrument. The order in which the elements are listed should match the natural order of the performance.
4. Select the type of judgment to be made by the evaluator. When evaluating a performance, product, or attitude, judgments can be made using checklists, rating scales, or frequency counts.
5. Determine how the instrument will be scored. With checklists you can simply add up the "yes" answers to obtain a score for each objective and for the entire process or product. With rating scales you can add up the numbers assigned for each element.
REFLECTION
I can't help but think about how I have used assessments in the past to either gauge my personal knowledge and skill levels. How succesful is the assessment? Am I able to analyze the aptitudes of my learners? These are just some of the questions I've been thinking about as I read this chapter.
I read this comment about assessment I though I'd share: "No area of educational practice is as controversial and problematic as student assessment. Throughout the world, a move is on whose stated goals are to make schools more accountable for their outcomes. A key piece of this movement is the use of standardized tests as the primary measure of effectiveness of student learning and evaluation of schools. The fact is, however, if you are interested in pursuing personal excellence and citizenship for your students, standardized tests don’t contribute very much. First, their focus is much too narrow. Most emphasize basic facts in core subjects such as reading, math, and science. Presently, such assessment is limited to these three areas under the requirements of NCLB in the United States. Even in these subjects, however, most tests do not engage students in problem solving and using skills and information in performing authentic tasks. Few involve students in meaningful thinking beyond factual recall. Of course, there simply are no tests for other important areas in children’s development – eg. honesty, relationships, critical thinking, and more. The second problem is related to the first. Many schools are following the narrow focus of these tests to further narrow their curriculum, reducing and restricting the quality of learning" (Kohn, 1999; Neill, Guisbond, & Schaeffer, 2004).
Overall, I think that the assessments helped to gain a better understanding of my knowledge and that of my learners.
EXTENDED KNOWLEDGE
Purposes of assessment
Assessment for improved student learning and deep understanding requires a range of assessment practices to be used with three overarching purposes:
Assessment FOR learning - occurs when teachers use inferences about student progress to inform their teaching
Assessment AS learning - occurs when students reflect on and monitor their progress to inform their future learning goals
Assessment OF learning - occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards.
Principles for assessment
Substantial research exists on the characteristics of good practice for assessing student learning. This research is summarised in the following set of principles.
The primary purpose of assessment is to improve student performance
Good assessment is based on a vision of the kinds of learning we most value for students and how they might best achieve these. It sets out to measure what matters most.
Assessment should be based on an understanding of how students learn
Assessment is most effective when it reflects the fact that learning is a complex process that is multi-dimensional, integrated and revealed in student performance over time.
Assessment should be an integral component of course design and not something to add afterwards
The teaching and learning elements of each program should be designed in full knowledge of the sorts of assessment students will undertake, and vice versa, so that students can demonstrate what they have learned and see the results of their efforts.
Good assessment provides useful information to report credibly to parents on student achievement
A variety of assessment methods provide teachers with evidence of what students know and can do, and their particular strengths and weaknesses. Teachers can then report to parents on how far their child has progressed during the year, where they are compared to the relevant standards, and what the student, the parent and the teacher need do to improve the student's performance.
Good assessment requires clarity of purpose, goals, standards and criteria
Assessment works best when it is based on clear statements of purpose and goals for the course, the standards which students are expected to achieve, and the criteria against which we measure success. Assessment criteria need to be understandable and explicit so students know what is expected of them from each assessment they encounter. Staff, students, parents and the community should all be able to see why assessment is being used, and the reasons for choosing each individual form of assessment in its particular context.
Good assessment requires a variety of measures
In general, a single assessment instrument will not tell us all we need to know about student achievement and how it can be improved. Therefore, we need to be familiar with a variety of assessment tools so we can match them closely to the type of information we seek.
Assessment methods used should be valid, reliable and consistent
Assessment instruments and processes should be chosen which directly measure what they are intended to measure. They should include the possibility of moderation between teachers where practical and appropriate to enhance objectivity and contribute to a shared understanding of the judgments that are made.
Assessment requires attention to outcomes and processes
Information about the outcomes students have achieved is very important to know where each student ends up, but so too is knowing about their experiences along the way and, in particular, the kind of effort that led to these outcomes.
Assessment works best when it is ongoing rather than episodic
Student learning is best fostered when assessment involves a linked series of activities undertaken over time, so that progress is monitored towards the intended course goals and the achievement of relevant standards.
Assessment for improved performance involves feedback and reflection
All assessment methods should allow students to receive feedback on their learning and performance so assessment serves as a developmental activity aimed at improving student learning. Assessment should also provide students and staff with opportunities to reflect on both their practice and their learning overall.
I think the model below enables us to understand the concept well.
Reference:
http://www.education.vic.gov.au/school/teachers/support/pages/advice.aspx
Test items should be congruent with the terminal and performance objectives by matching the behavior involved. What this means is that each test item should measure the exact behavior and response stated in the objective.
Learner-Centered Criteria
Test items should take into consideration the characteristics and needs of the learners. This includes issues such as learners vocabulary and language levels, motivational and interest levels, experiences and backgrounds, and special needs.
Context-Centered Criteria:
Remember the context analysis you wrote. Well, when writing test items you should consider both the performance context and the learning context . It is important to make your test items as realistic and close to the performance setting as possible.
Assessment-Centered Criteria
Test items should be well written and free of spelling, grammar, and punctuation errors. Directions should be clearly written to avoid any confusion on the part of the learner.
Dick and Carey Five Steps for Creating an Assessment Instrument:
1. Identify the elements to be evaluated. These elements should be taken directly from the behaviors and criteria included in your objectives.
2. Paraphrase each element. Also, make sure that a "Yes" response on the instrument always corresponds with a positive performance, and a "No" response with a negative performance.
3. Sequence the elements on the instrument. The order in which the elements are listed should match the natural order of the performance.
4. Select the type of judgment to be made by the evaluator. When evaluating a performance, product, or attitude, judgments can be made using checklists, rating scales, or frequency counts.
5. Determine how the instrument will be scored. With checklists you can simply add up the "yes" answers to obtain a score for each objective and for the entire process or product. With rating scales you can add up the numbers assigned for each element.
REFLECTION
I can't help but think about how I have used assessments in the past to either gauge my personal knowledge and skill levels. How succesful is the assessment? Am I able to analyze the aptitudes of my learners? These are just some of the questions I've been thinking about as I read this chapter.
I read this comment about assessment I though I'd share: "No area of educational practice is as controversial and problematic as student assessment. Throughout the world, a move is on whose stated goals are to make schools more accountable for their outcomes. A key piece of this movement is the use of standardized tests as the primary measure of effectiveness of student learning and evaluation of schools. The fact is, however, if you are interested in pursuing personal excellence and citizenship for your students, standardized tests don’t contribute very much. First, their focus is much too narrow. Most emphasize basic facts in core subjects such as reading, math, and science. Presently, such assessment is limited to these three areas under the requirements of NCLB in the United States. Even in these subjects, however, most tests do not engage students in problem solving and using skills and information in performing authentic tasks. Few involve students in meaningful thinking beyond factual recall. Of course, there simply are no tests for other important areas in children’s development – eg. honesty, relationships, critical thinking, and more. The second problem is related to the first. Many schools are following the narrow focus of these tests to further narrow their curriculum, reducing and restricting the quality of learning" (Kohn, 1999; Neill, Guisbond, & Schaeffer, 2004).
Overall, I think that the assessments helped to gain a better understanding of my knowledge and that of my learners.
EXTENDED KNOWLEDGE
Purposes of assessment
Assessment for improved student learning and deep understanding requires a range of assessment practices to be used with three overarching purposes:
Assessment FOR learning - occurs when teachers use inferences about student progress to inform their teaching
Assessment AS learning - occurs when students reflect on and monitor their progress to inform their future learning goals
Assessment OF learning - occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards.
Principles for assessment
Substantial research exists on the characteristics of good practice for assessing student learning. This research is summarised in the following set of principles.
The primary purpose of assessment is to improve student performance
Good assessment is based on a vision of the kinds of learning we most value for students and how they might best achieve these. It sets out to measure what matters most.
Assessment should be based on an understanding of how students learn
Assessment is most effective when it reflects the fact that learning is a complex process that is multi-dimensional, integrated and revealed in student performance over time.
Assessment should be an integral component of course design and not something to add afterwards
The teaching and learning elements of each program should be designed in full knowledge of the sorts of assessment students will undertake, and vice versa, so that students can demonstrate what they have learned and see the results of their efforts.
Good assessment provides useful information to report credibly to parents on student achievement
A variety of assessment methods provide teachers with evidence of what students know and can do, and their particular strengths and weaknesses. Teachers can then report to parents on how far their child has progressed during the year, where they are compared to the relevant standards, and what the student, the parent and the teacher need do to improve the student's performance.
Good assessment requires clarity of purpose, goals, standards and criteria
Assessment works best when it is based on clear statements of purpose and goals for the course, the standards which students are expected to achieve, and the criteria against which we measure success. Assessment criteria need to be understandable and explicit so students know what is expected of them from each assessment they encounter. Staff, students, parents and the community should all be able to see why assessment is being used, and the reasons for choosing each individual form of assessment in its particular context.
Good assessment requires a variety of measures
In general, a single assessment instrument will not tell us all we need to know about student achievement and how it can be improved. Therefore, we need to be familiar with a variety of assessment tools so we can match them closely to the type of information we seek.
Assessment methods used should be valid, reliable and consistent
Assessment instruments and processes should be chosen which directly measure what they are intended to measure. They should include the possibility of moderation between teachers where practical and appropriate to enhance objectivity and contribute to a shared understanding of the judgments that are made.
Assessment requires attention to outcomes and processes
Information about the outcomes students have achieved is very important to know where each student ends up, but so too is knowing about their experiences along the way and, in particular, the kind of effort that led to these outcomes.
Assessment works best when it is ongoing rather than episodic
Student learning is best fostered when assessment involves a linked series of activities undertaken over time, so that progress is monitored towards the intended course goals and the achievement of relevant standards.
Assessment for improved performance involves feedback and reflection
All assessment methods should allow students to receive feedback on their learning and performance so assessment serves as a developmental activity aimed at improving student learning. Assessment should also provide students and staff with opportunities to reflect on both their practice and their learning overall.
I think the model below enables us to understand the concept well.
Reference:
http://www.education.vic.gov.au/school/teachers/support/pages/advice.aspx