|
|  | Equating and Scaling
Even though different forms or editions of the same exam are designed to be as similar as possible, slight variations in difficulty are unavoidable. Statistical procedures, known as equating methods, have been designed to deal with the problem of adjusting scores so as to compensate for differences in difficulty between different forms of the same exam. Equating of AP Exams is carried out using a set of multiple-choice items (a "mini-test") which appear on both the current form of the exam and a previous form.
These repeated, or "common," items provide a direct link between the current form of the exam and previous forms. They make it possible to compare the performance of this year's students with the performance of a previous group on exactly the same set of items. In selecting common items, the content expert and the Development Committee are careful to avoid any items expected to become easier or harder over the course of time. For example, a question in the U.S. Government and Politics exam concerning the role of the Electorial College in deciding Presidential elections would be a great deal easier to answer now than it would have been five years ago. Similarly, the prominent positioning of a piece of artwork in a new edition of an art history textbook would likely affect students' answers to a question about that piece. Attention is also paid to other factors which might influence the difficulty of an item, such as its relative position on the exam.
The Base Scale
Every score is reported on some scale in particular units. For each AP Exam, there is a "base scale," which is the score on the multiple-choice section of one form (the "base form") of that exam. Using the equating function, multiple-choice scores on any other form of the exam can be expressed on the base scale.
For example, without going into the details of the calculations involved, the equating function might map a score of 28.25 on the current year's multiple-choice section to a score of 29.50 on the base scale. The score of 28.25 on the scale of the current year's multiple-choice section is exchangeable with a score of 29.50 on the base form. Given any score on the current year's multiple-choice section, the equating function can be used to find out the "equated score" on the base scale.
The equating function is symmetric in nature. This means that if we are given a score on the base scale, we can apply the equating function in the reverse direction to find out the "equated score" on the current year's multiple-choice section. In the above example, if we were given a score of 29.50 on the base scale, applying the equating function in the reverse direction would tell us that the equated score on the current year's multiple-choice section is 28.25, which is the number we started with.
The specific equating function used can vary depending upon factors such as:
- The performance of the two groups of students on the common items; and
- The relative difficulties of the two multiple-choice sections being equated.
One type of equating function used on AP Exams is based on a method first proposed by Ledyard R. Tucker (Angoff, 1984). Another is based on a method known as equipercentile equating (Angoff, 1984).
Comparing Student Groups Across Years
The base scale can be used to compare groups of students taking the exam in several different years. A box-and-whiskers plot can be produced (see Figure 3.1) that shows the performance of students at the 10th, 25th, 50th, 75th, and 90th percentiles over five years of the exam. Note that the scores at these percentiles are expressed on the base scale, thus making it possible to compare them across years.
To understand how to interpret the box-and-whiskers plot, suppose that the score at the 50th percentile for the current year is 60 percent of the maximum possible score on the base scale, and the score at the same percentile is 58 percent for the previous year. Then, in terms of the base scale, half of the students in the current year score below 60 percent, while half of the students in the previous year score below 58 percent. Thus, in terms of the base scale, students at the 50th percentile in the previous year performed at a lower level than did students at the 50th percentile in the current year.
Equipercentile Scaling
The method of equipercentile scaling allows us to establish a correspondence between the multiple-choice and composite score scales on the same form of an exam. The procedure involves the following steps:
- Find the percentile rank for a given multiple-choice score.
- Find the composite score at the same percentile rank.
This method allows us to take any multiple-choice score on a given exam and place it on the scale of the composite score for the same exam. However, we cannot treat the two numbers as if they were exchangeable, because the multiple-choice and composite scores are likely to be measures of different constructs or abilities. Therefore, this process is called scaling and not equating.
Scaling functions, like equating functions, are symmetric.
Adjusting Cut-off Scores That Determine Grades
Table 3.4 illustrates the process of adjusting the cut-off scores that determine grades. Every year, such a table is produced for each AP Exam. Here is a detailed explanation of each of the columns in the table:
- The first column shows the four grade boundaries. The first row, labeled '5', is the 5/4 grade boundary, the second is the 4/3 grade boundary, and so on.
- The second column shows the cut-off scores on the base scale. That is, these are the minimum scores required to obtain each grade, expressed on the base scale for the exam. These are obtained in the following manner:
- For each of the previous eight years, the cut-off scores, which are on the composite score scale, are expressed on the multiple-choice score scale for that year using equipercentile scaling.
- Each of the eight sets of cut-off scores is then expressed on the base scale using the equating function for that year.
- Now that they are all on a comparable scale, the eight cut-off scores are averaged. These averages are shown in the second column of the table.
Note: If the standards for the exam were revised in the one of the last eight years (for example, as a result of a college comparability study), the average only includes the years following the revision of standards.
- The third column shows the cut-off scores on the multiple-choice score scale of the current year's exam. These are obtained by applying the equating function for the current year's multiple-choice section in the reverse direction.
- The fourth column shows estimates of error associated with the equating for each of the four cut-off scores. The degree of error is influenced by the size of the student groups. For most AP Exams, the student groups are so large that the error is less than 0.005, and can be neglected.
- The error shown in the fourth column is used to generate a range of cut-off scores by adding and subtracting one and a half times the error from the cut-off score. Note that the scores in the fifth column are still on the multiple-choice score scale of the current year's exam.
- The sixth column shows the percentile ranks corresponding to the scores in the fifth column.
- The last column shows the composite scores corresponding to the percentile ranks in the sixth column. Columns five through seven, as you may have noticed, are simply an equipercentile scaling which express the multiple choice cut-off scores on the composite score scale of the current year's exam. These adjusted cut-off scores are presented to the Chief Reader along with other materials, including the box-and-whiskers plot described above.
|
Course Home Pages