Reporting Scale Scores and Achievement Levels

Reporting Scale Scores and Achievement Levels

Results of the NAEP Technology and Engineering Literacy Assessment will be reported in terms of percentages of students who attain each of the three achievement levels, Basic, Proficient, and Advanced, discussed below. The NAEP Technology and Engineering Literacy Assessment is an assessment of overall achievement, not a tool for diagnosing the needs of individuals or groups of students. Reported scores are always at the aggregate level. By law and by design, scores are not produced for individual schools or students. Results will be reported for the nation as a whole as well as for regions of the nation. The NAEP Technology and Engineering Literacy Assessment will not provide results for individual states, since the student samples will be drawn to report at the national level only.

The project committees have recommended that the results of the assessment be reported in terms of three subscores, each of them reflecting performance in one of the three main areas of technology and engineering literacy: Technology and Society, Design and Systems, and Information and Communication Technology. An overall composite score will also be reported. At this time technology instruction in most K-12 schools focuses on one or another of the three areas rather than a fusion of the three, so a composite score can be expected to have less relevance than the scores from the three areas.

Reporting on achievement levels is the primary way in which NAEP results reach the general public and policymakers. Achievement level results indicate the degree to which student performance meets the standards set for what students should know and be able to do at the Basic, Proficient, and Advanced levels. Definitions of achievement levels articulate expectations of performance at each grade level. They are reported as percentages of students within each achievement level range, as well as the percentage of students at or above Basic and at or above Proficient ranges. Results for students not reaching the Basic achievement level are reported as below Basic. Results are also reported for subgroups of students using demographic data and background variables specific to the NAEP Technology and Engineering Literacy Assessment. An individual student's performance cannot be reported based on NAEP results.

Table 5.1 displays the Governing Board's generic policy definitions for Basic, Proficient, and Advanced achievement that pertain to all NAEP subjects and grades.

Table 5.1 Generic achievement level policy definitions for NAEP

There are three components to the NAEP achievement levels:

• Achievement level definitions;
• Cut scores; and
• Examples of students' responses.

Achievement Level Definitions

Since 1990, the Governing Board has used student achievement levels for reporting results on NAEP assessments. The achievement levels represent an informed judgment of "how good is good enough" in the various subjects that are assessed. Generic policy definitions for achievement at the and Advanced levels describe in very general terms what students at each grade level should know and be able to do on the assessment. Technology and Engineering Literacy achievement levels specific to the 2014 NAEP Technology and Engineering Literacy Framework will be developed to elaborate the generic policy definitions of Basic, Proficient, and Advanced achievement for NAEP assessments. Preliminary achievement level definitions have been developed for each of the three areas to be reported separately in the assessment and they will be used to guide item development and initial stages of standard setting for the 2014 NAEP Technology and Engineering Literacy Assessment. (See appendix G for these preliminary definitions.)

The preliminary achievement level definitions will be revised when actual student responses have been collected and analyzed. The Governing Board will convene panels of experts to examine the preliminary achievement level definitions and to recommend final achievement level definitions for each grade level. A broadly representative panel of exceptional teachers, educators, and professionals will then be convened to engage in a standard setting process to determine the cut scores that correspond to these achievement level definitions. The panelists will be trained and will engage in a series of discussions designed to ensure informed judgments about mapping cut scores to the assessment.

Cut Scores

Cut scores, the second component of reporting on achievement levels, represent the minimum score required for performance at each NAEP achievement level. Cut scores are reported along with the percentage of students who scored at or above the cut score. As described in chapter four, the assessment design for the 2014 NAEP Technology and Engineering Literacy Assessment incorporates scores from selected responses and written responses, as well as measures of the patterns of action a student takes in problem solving. Selected responses in which there is a single best answer will be scored as correct or incorrect and written responses will be scored using a rubric that rewards answers according to their match to descriptions in the rubric. The pattern-tracking will be evaluated by comparing the pattern of action against a set of possible patterns, and students will get more credit for a course of action that is optimal than for alternative patterns of action. Scores can then be combined to produce overall scores so that cut score decisions can be made.

Examples of Students' Responses

The third component of achievement level reporting includes examples of student responses on released tasks from the NAEP Technology and Engineering Literacy Assessment. These examples provide illustrations of student skills within each level of achievement for each of the three areas and will be developed after the first administration of the assessment. Example responses will be annotated to explain the score for the response.