Judgmental Standard Setting

Judgmental Standard Setting Methodology

For the judgmental standard setting study, panels of subject matter experts from across the country met to review the NAEP test and determine the minimal level of academic performance on NAEP that demonstrates preparedness for placement in a college course without remediation or preparedness for entry into a job training program for five occupations: automotive master technician, computer support specialist, heating ventilation and air conditioning technician, licensed practical nurse, and pharmacy technician. The panels of subject matter experts were college instructors and job training program instructors for the five occupations in courses with math and reading content.

In college and for the five occupations, the panels used a modified bookmark standard setting method to recommend a cut score for 12th grade NAEP reading and NAEP mathematics that represents the minimal level of academic preparedness. To evaluate the reliability of the results, a replicate panel design was used. In this design, two independent panels for each of the five different occupational job training programs and for college course work recommended a cut score. Results from each pair of panels were then compared. The process involved 24 panels, across the six study areas for reading and mathematics, with each study having two panels for the replicate design.

The modified bookmark standard setting method was based on the process developed originally for NAEP achievement level setting by ACT, Inc. Panelists reviewed assessment items that were ordered by difficulty starting with the easiest item and progressing to the most difficult. They evaluated each item against a description of borderline performance for "just academically prepared" in each of the six areas studied (five occupations and college)--until they determined where a minimally prepared student could no longer respond correctly with a 0.67 probability. A bookmark was placed immediately preceding that item to locate the cut score for each respective area. Within each panel, individual panelists’ cut scores were used to compute the median, which served as the panel’s cut score for "just academically prepared."

For the course content analysis studies, course materials (artifacts) were reviewed from a representative sample of entry-level college courses and job training programs. The college course content analysis targeted college mathematics course titles (pre-calculus/calculus, college algebra, finite mathematics and statistics), and college courses with substantial reading demands (English literature, psychology, U.S. government, and U.S. history). The course analysis for job training programs targeted the same five occupations noted above.

The primary objectives were to identify the knowledge, skills, and abilities (KSAs) that are prerequisite to these entry-level courses and then to compare these prerequisite KSAs with NAEP content identified in the judgmental standard setting study for "just academically prepared." Course artifacts included syllabi, assignments, examinations, reading/textbook passages, and textbook tables of contents. The college study analyzed course artifacts from 160 courses at 151 institutions. The job training study analyzed artifacts from 165 courses at 122 institutions.

Teams of mathematics and reading content-area experts collaborated in a group process to analyze the course artifacts and identify the prerequisite reading and mathematics KSAs, i.e., the minimal knowledge and skills required of students entering the courses. Reviewers identify which of the NAEP KSAs (NAEP reading and mathematics framework objectives) were prerequisite KSAs based on evidence in the course artifacts. If only a part of a NAEP KSA statement was applicable, reviewers marked the inapplicable section. Reviewers were also asked to indicate any additional prerequisite KSA statements evident in course artifacts that were not among the NAEP KSAs. Reviewers first completed these artifact analyses independently, and then convened to discuss their determinations, resolve differences, and decide on whether consensus could be reached.

Judgmental Standard Setting Key Conclusions

Judgmental Standard Setting Study – College and Job Training

• Overall, the results across the six postsecondary areas were not confirmed by the replicate panels or other preparedness research results.
• The data collected from the panelists in the judgmental standard setting studies for the NAEP 12th grade assessments in reading and mathematics do not support the conclusion that minimal academic preparedness for college is the same as minimal academic preparedness for training programs for the five occupational areas that were examined.
• Panelists in the job training studies identified many NAEP 12th grade items they deemed as not required for determining academic preparedness for their job training programs.

Course Content Analysis Studies

College Mathematics and Reading

The results from the study indicate that most of the prerequisite KSAs for both mathematics courses and courses that require extensive college-level reading are reflected in the NAEP frameworks. The prerequisite KSAs either mapped to the NAEP frameworks or they were included as additional KSAs beyond the NAEP frameworks.

For College Mathematics

• The mathematics KSAs were mostly specific to a course title, i.e., there was not a single set of prerequisite KSAs that covered all entry-level, credit-bearing courses. However, the majority of prerequisite KSAs were represented in objectives within the NAEP framework’s Numbers and Operations subject area, and many of the sampled courses did not have measurement or geometry objectives as prerequisite KSAs.
• The prerequisite KSAs focused more heavily on application of mathematics (e.g., simulations, addressing real-world problems) than is evident in the NAEP assessment items. The prerequisite content focused on application problems, which do match the NAEP framework, but the items do not reflect the same level of focus.
• After accounting for the pieces of NAEP objectives that were not considered relevant, some prerequisite KSAs are found in the 8th grade NAEP framework.

For College Reading

• The NAEP reading assessment aligned well with the prerequisite KSAs identified in this study, especially in the cognitive domains of Locate/Recall and Integrate/Interpret. The prerequisite KSAs focused less on the ability to critique or evaluate, which may be what is covered within a college course and therefore not a prerequisite to entry into the course.
• Some of the reading KSAs that are prerequisite to entering entry-level, credit-bearing college courses have substantial informational text reading demands, and some of the reading KSAs demand engagement with literary texts. KSAs common to both included the ability to locate or recall textually explicit information within and across texts and the ability to take different perspectives in relation to a text.

Job Training Mathematics and Reading

The course prerequisite knowledge, skills, and abilities identified are largely included in the grade 12 NAEP frameworks, but the full content of NAEP frameworks is much larger and broader. Course artifacts provided evidence that most course prerequisites were measured by NAEP. However, many grade 12 NAEP assessment items were deemed not required for determining academic preparedness for job training programs.

For Mathematics in Job Training Programs:

• The job training programs studied have few prerequisite expectations represented in the grade 12 NAEP mathematics framework.
• The largest number of prerequisites across all occupational training programs are found in the Number Properties and Operations domain, specifically: the "Systems of measurement," "Variables, expressions, and operations," and "Equations and inequalities" standards. No programs had prerequisites in the Data Analysis, Statistics, and Probability domain, and few programs had prerequisites in the Geometry domain.
• Generally, the artifacts included no evidence of irrational numbers, exponents and logarithms, or absolute value as prerequisites—these topics appear within Number Properties and Operations, Measurement, and Algebra domains.
• Between 64% and 78% of the 130 mathematics objectives were not evident as prerequisite in any course within the five occupations.

For Reading in Job Training Programs

• 18 on ACT Reading on a scale of 1 to 36
• Across all job training programs, the only grade 12 NAEP reading objectives identified as prerequisites for entry-level courses in all five occupational areas were those related to reading informational texts. NAEP objectives in the areas of literary text and literary devices were not found to be present in any programs.
• Specific reading skills that are prerequisite to all five job training programs include "Locate or recall causal relations" and "Locate or recall organizing structures of texts, such as comparison/contrast, problem/solution, enumeration, etc."
• The portions of the NAEP mathematics KSA statements that were identified as inapplicable or excluded from the training course content prerequisites, eliminated much of the complex mathematics knowledge and skills that differentiate the grades 8 and 12 frameworks. As a result, some prerequisite KSAs appear to be better described by the grade 8 objectives.
• The number of reading objectives not evident as prerequisite in any course within the five occupations ranged between 16% and 68% of the 37.