SCORING RUBRIC / GRADING

                                     

A.    Definition

1.      Scoring

‘A score is a number derived from a measure...’ (Anderson in Sulistyo, 2007). In testing context, this follows then that scoring is a process of utilizing a number to represent the responses made by the test taker.

2.      Grading

Grading rubrics are an effective tool for providing students with your expectations on a given task as well as providing you with a standardized grading mechanism. Rubrics identify the important traits (those to which you assign a score or deduct points) of a task and articulate performance levels. Rubrics attempt to make explicit the implicit criteria by which you grade or assess student performance.

Typically, grading rubrics are created to correspond to an instructor’s student learning objectives. In addition to providing students with clear expectations, a rubric is a valuable assessment tool for faculty wishing to demonstrate direct evidence of students learning.

A grading rubric should be created with the following items:

ð  Specific qualities (or traits) to be assessed

ð  Standards or levels of performance

ð  Specific indicators describing what the various levels of performance look like for each of the qualities being assessed

Be cautious not to make the indicators so broad that they are non-descriptive; they should be as focused as a student learning objective.

Barbara Walvoord and Virginia Anderson’s book, Effective Grading: A Tool for Learning and Assessment has a wealth of information on using rubrics to facilitate the grading process and as a formative assessment tool for both the faculty member and the student.

3.      Rubric

A rubric is a scoring guide that’s intended to help those who must score students’ responses to constructed-response items. The most important component in any rubric is the rubric’s evaluative criteria, which are the factors a scorer considers when determining the quality of a student’s response. For example, one evaluative criterion in an essay-scoring rubric might be the organization of the essay. Evaluative criteria are truly the guts of any rubric, because they lay out what it is that distinguishes students’ winning responses from their weak ones. A second key ingredient in a rubric is a set of quality definitions that accompany each evaluative criterion. These quality definitions spell out what is needed, with respect to each evaluative criterion, for a student’s response to receive a high rating versus a low rating on that criterion. A quality definition for an evaluative criterion like “essay organization” might identify the types of organizational structures that are acceptable and those that aren’t. The mission of a rubric’s quality definitions is to reduce the likelihood that the rubric’s evaluative criteria will be misunderstood.

Finally, a rubric should set forth a scoring strategy that indicates how a scorer should use the evaluative criteria and their quality definitions. There are two main contenders, as far as scoring strategy is concerned. A holistic scoring strategy signifies that the scorer must attend to how well a student’s response satisfies all the evaluative criteria in the interest of forming a general, overall evaluation of the response based on all criteria considered in concert. In contrast, an analytic approach to scoring requires a scorer to make a criterion-by criterion judgment for each of the evaluative criteria, and then amalgamate those per-criterion ratings into a final score (this is often done via a set of predetermined, possibly numerical, rules).

B.     Types

1.      Dichotomous

Dichotomous scoring entails viewing and treating the response as either one of two distinct, exclusive categories. The number utilized in this kind of scoring is 0 (zero) and 1 (one).

2.      Continuous

Continuous scoring views and treat the test taker’s response as being grade in nature. The test taker’s response is considered as having a gradation or degree in it.

C.   Strategy in Scoring

1.         Holistic

Holistic scoring considers the test takers’ response as a whole totality rather than as consisting of fragmented parts. In holistic scoring, the focus is laid on the global quality of the test taker’s response. The benefits of this scoring are quick marking, relatively high inter-rater consistency, easy interpretation due to use of standards, showing the strength of the test takers, and a wide spectrum of use among disciplines. The challenges are absence of diagnostic power, concealing differences across sub-skills within each score, genre-limited application, extensive training on the part of raters.

2.         Primary Trait

Primary trait scoring focuses on one specific type of features or traits that test takers need to demonstrate. The key of primary trait is specificity of discourse to be exhibited by the test takers.

3.         Analytic

The analytic scoring emphasizes on individual points or components of the test taker’s response. In an analytic scoring plan, linguistic and non linguistic features considered important in a discourse become the building blocks of the scheme, and these features are commonly evaluated separately as individual components.

D.      Component of Rubric

A scale of points to be assigned in scoring a piece of work on a continuum of quality. High numbers are typically assigned to the best work.

1.      Descriptors for each level of performance that contain criteria and standards by which the performance will be judged

2.      Indicators are often used in descriptors to provide examples or signs of performance in each level

3.      Criteria that describe the conditions that any performance must meet to be successful

Five categories to consider:

ð  Impact – the success of performance, given the purposes, goals and desired results

ð  Work quality/Craftsmanship – the overall polish, organization, and rigor of the work

ð  Methods – the quality of the procedures and manner of presentation, prior to and during performance

ð  Content – the correctness of the ideas, skills or materials used

ð  Sophistication of the performance – the relative complexity or maturity of the knowledge used. Should describe both strengths and errors (errors should be described particularly in lower levels of performance)

4.      Standards that specify how well criteria must be met

E.     Steps in Designing Rubric

A step-by-step process for designing scoring rubrics for classroom use is presented below.

Information for these procedures was compiled from various sources (Airasian, 2000 & 2001;

Mertler, 2001; Montgomery, 2001; Nitko, 2001; Tombari &Borich, 1999). The steps will be

summarized and discussed, followed by presentations of two sample scoring rubrics.

Step 1: Re-examine the learning objectives to be addressed by the task. This allows you tomatch your scoring guide with your objectives and actual instruction.

Step 2: Identify specific observable attributes that you want to see (as well as those youdon’t want to see) your students demonstrate in their product, process, orperformance. Specify the characteristics, skills, or behaviors that you will belooking for, as well as common mistakes you do not want to see.

Step 3: Brainstorm characteristics that describe each attribute. Identify ways todescribe above average, average, and below average performance for eachobservable attribute identified in Step 2.

Step 4a: For holistic rubrics, write thorough narrative descriptions for excellent workand poor work incorporating each attribute into the description. Describe thehighest and lowest levels of performance combining the descriptors for allattributes.

Step 4b: For analytic rubrics, write thorough narrative descriptions for excellent workand poor work for each individual attribute. Describe the highest and lowestlevels of performance using the descriptors for each attribute separately.

Step 5a: For holistic rubrics, complete the rubric by describing other levels on thecontinuum that ranges from excellent to poor work for the collectiveattributes. Write descriptions for all intermediate levels of performance.

Step 5b: For analytic rubrics, complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for each attribute. Write descriptions for all intermediate levels of performance for each attribute separately.

Step 6: Collect samples of student work that exemplify each level. These will help you in designing scoring rubrics for your classroom.

Step 7: Revise the rubric, as necessary. Be prepared to reflect on the effectiveness of the rubric and revise it prior to its next implementation.

F.     Criteria of A Good Rubric

1.      Are sufficiently generic to relate to general goals beyond an individual performance task but specific enough to enable useful and sound inferences on the task. The word “rubric” derives from the Latin word for “red.” It was once used to signify the highlights of a legal decision as well as the directions for conducting religious services, found in the margins of liturgical books—both written in red.

2.      Discriminate among performances validly, not arbitrarily—by the central features of performance, not by the easiest to see, count, or score.

3.      Do not combine independent criteria in one rubric.

4.      Are based on analysis of many work samples, and based on the widest possible range of work samples—including valid exemplars.

5.      Rely on descriptive language—what quality, or its absence, looks like—as opposed to relying heavily on mere comparatives or value language (e.g. “not as thorough as,” or “excellent product”) to make the discrimination.

6.      Provide useful and apt discrimination to enable sufficiently fine judgments—but not using so many points on the scale as to threaten reliability (typically involving, therefore, 6-12 points on a scale).

7.      Use descriptors that are sufficiently rich to enable student performers to verify their score, accurately self-assess, and self-correct.

8.      Highlight the judging of the “impact” of performance—the effect, given the purpose—as opposed to over-rewarding merely the processes, the formats, or the content used; and/or the good faith effort made.

References:

Airasian, P. W. (2000). Assessment in the classroom: A concise approach (2nd ed.). Boston: McGraw-Hill.

Airasian, P. W. (2001). Classroom assessment: Concepts and applications (4th ed.). Boston: McGraw-Hill.

Mertler, C. A. (2001). Using performance assessment in your classroom. Unpublished manuscript, Bowling Green State University.

Montgomery, K. (2001). Authentic assessment: A guide for elementary teachers. New York: Longman.

Nitko, A. J. (2001). Educational assessment of students (3rd ed.). Upper Saddle River, NJ: Merrill.

Tombari, M. & Borich, G. (1999).Authentic assessment in the classroom: Applications and practice. Upper Saddle River, NJ: Merrill.