Chapter One: Overview of Recommendations

Introduction

Since 1973 the National Assessment of Educational Progress (NAEP) has gathered information about student achievement in mathematics. The results of these periodic assessments, produced in print and Web-based formats, inform citizens about the nature of students’ comprehension of the subject, inform curriculum specialists about the level and nature of student understanding, and inform policymakers about factors related to schooling and its effect on student proficiency in mathematics.

Based on these surveys of students at the elementary, middle school, and high school levels, NAEP has provided comprehensive information about what students in the United States know and can do in the area of mathematics, as well as in a number of other subject areas. These reports present information on strengths and weaknesses in students’ understanding and their ability to apply that understanding in problemsolving situations; provide comparative student data according to race/ethnicity, type of community, and geographic region; describe trends in student performance over time; and report on relationships between student proficiency and certain background variables.

This document contains the framework and a set of recommendations for the 2005 NAEP mathematics assessment. It includes descriptions of the mathematical content of the test, the types of test questions, and recommendations for administration of the test. In broad terms, this framework attempts to answer the question: What mathematics should be assessed in 2005 on NAEP? The answer to this question must necessarily take into account the constraints of a large-scale assessment such as NAEP, with its limitations on time and resources. Of critical importance is the fact that this document does not attempt to answer the question: What mathematics should be taught (or how)? This is an assessment framework, not a curriculum framework. It was developed with the under- standing that some concepts, skills, and activities in school mathematics are not suitable to be assessed on NAEP, even though they may be important components of a school curriculum. (An example would be an extended project that involves gathering data, or a group project.)

The Assessment and Item Specifications for the 2005 NAEP Mathematics Assessment, a companion document to this framework, gives more detail about the items and conditions for the 2005 NAEP mathematics assessment.

Note: To comply with the provisions of the No Child Left Behind Act of 2001, the National Assessment Governing Board (NAGB) adopted a revised NAEP assessment schedule in March 2002. This schedule moved the 2004 mathematics assessment to 2005 because NAEP will test mathematics and reading in odd-numbered years.

Context for Planning the 2005 Assessment

The National Assessment Governing Board, created by Congress in 1988, is responsible for formulating policy for NAEP. NAGB is specifically charged with developing assessment objectives and test specifications, identifying appropriate achievement goals for each age and grade, and carrying out other NAEP policy responsibilities.

Congress also authorized the trial state assessment program (more commonly known as the state NAEP program) in 1988. The voluntary trial state NAEP began with the 1990 assessment of mathematics at grade 8; the state NAEP program continued with the 1992 mathematics assessment at grades 4 and 8. The regular national mathematics assessment of students at grades 4, 8, and 12 also occurred in 1990 and 1992.

In preparation for the 1990 mathematics assessment, a contract was awarded to the Council of Chief State School Officers (CCSSO) to design a framework for the assessment. The CCSSO project gave special attention to the formal state objectives and frameworks for mathematics instruction. As a part of the work, state-, district-, and school-level objectives were considered, as well as the frameworks on which previous NAEP mathematics assessments had been based and a draft version of the National Council of Teachers of Mathematics’ (NCTM’s) Curriculum and Evaluation Standards for School Mathematics. The result was a “content by mathematical ability” matrix design that was used to guide both the 1990 and 1992 mathematics assessments conducted by NAEP at the national and state levels. The design was reported in Mathematics Objectives: 1990 Assessment.

To prepare for the next NAEP mathematics assessment, NAGB awarded a contract in fall 1991 to the College Board to develop the assessment and item specifications for the 1994 mathematics assessment. The College Board work was completed between September 1991 and March 1992. Due to a budget shortfall, the 1994 mathematics assessment and the science assessment were rescheduled for 1996.

The NAEP mathematics project conducted by the College Board had two primary purposes: (1) to recommend a framework for the overall design of the mathematics assessment—a structure for describing what students know and can do in mathematics, and (2) to develop specifications for the assessment items, paying particular attention to the mix of item formats, the item distribution for content areas within mathematics, and the conditions under which items are presented to students (for example, use of manipulatives, use of calculators, and time to complete tasks). The work of the project was also to be considered in light of the three NAEP achievement levels adopted by NAGB—Basic, Proficient, and Advanced.

The College Board effort resulted in the “content areas by mathematical abilities by mathematical power” grid that was used to guide the development of the 1996 and 2000 national and state mathematics assessments conducted by NAEP. The design was reported in Mathematics Framework for the 1996 and 2000 National Assessment of Educational Progress. The 1996/2000 Framework was used in the design of the assessments for those years. Those assessments maintained the trend lines that began in 1990.

Other changes have taken place in NAEP in recent years. A number of policy, legislative, civil rights, and technical considerations caused NAEP to look more closely at its administration and assessment procedures and to consider implementing changes that can increase participation among students with disabilities and limited-English-proficient students. In the early 1990s, participation of these students in NAEP was limited, with schools excluding them from the assessment in accordance with criteria provided by the program at that time. This exclusion raised concerns that some special-needs students who could be meaningfully assessed were being excluded from NAEP.

In the mid-1990s, based on requirements from the reauthorization of the Individuals with Disabilities Education Act (IDEA), the general trend of state assessments toward more inclusion of special-needs students, and recommendations from various offices of the U.S. Department of Education, NAEP modified its procedures with the aim of increasing the participation of special-needs students. For the 1996 assessment, modifications were made in two areas: inclusion criteria were revised to make them clearer, more inclusive, and more likely to be applied consistently across jurisdictions in the United States; and a variety of assessment accommodations was offered for the first time. This shift toward greater inclusiveness allowed NAEP to be more similar to state and district testing policies. NAEP used a split-sample approach to study the changes made and their possible impact on trends, and issued several reports from their findings (see Olson and Goldstein, 1997; Reese et al., 1997; Mazzeo et al., 1999).

In the late 1990s, states had implemented assessments that included most, if not all, of their students and were beginning to report assessment results that included the performance of special-needs students. An array of assessment accommodations was being offered by states routinely, and by NAEP as well. In 1998, NAEP began new trend lines for some subject areas that presented data reported for the first time on samples that included accommodated special-needs students. In 2000, NAEP reported the mathematics assessment results for the first time on both a nonaccommodated sample of students as well as a sample of regular and special-needs students who received accommodations in The Nation’s Report Card: Mathematics 2000 (Braswell et al., 2001).

This trend toward more inclusive assessments has also led to a closer look at how test items are developed and assessments designed. Over the years, NAEP has implemented procedures that are designed to make items more accessible for a variety of special-needs students. The 2005 mathematics framework enables the continuation of this best practices approach, and the assessment and item specifications document (Assessment and Item Specifications for the 2005 NAEP Mathematics Assessment) provides clear and detailed guidance on how to develop a more inclusive assessment.

The long-range plan established by NAGB called for the development of a new mathematics framework to be used as the basis for the 2005 national and state assessments of mathematics. Prior to initiating work on a new framework, the Governing Board conducted a series of public meetings to gather input to serve as the basic purpose(s) for the new framework. At the public meetings, a strong concern was raised by states about the need to continue the short-term trend lines for the nation (at grades 4, 8, and 12) and the states (at grade 8 and later grade 4) that began with the 1990 mathematics assessment. The short-term trend lines were being used by many states as an independent monitor of their standards-based reform efforts.

Several other concerns were also raised. Had computational skills been overlooked by the reform efforts because of the emphasis placed on higher order skills by most state mathematics standards and the work by NCTM? Were U.S. students capable of handling a more rigorous curriculum, especially algebra and geometry, at grade 8? What should our mathematical expectations be for grade 12 students?

Based on the input received, the Governing Board awarded a contract to CCSSO in September 2000 to update the mathematics assessment framework used for the 1996 and 2000 assessments.

The revisions were to address the concerns raised at the public meetings while maintaining the short-term trend lines in grades 4 and 8 that began with the 1990 mathematics assessment.

It is within this context that the recommendations contained in this document were developed.

The Consensus Approach

The CCSSO established a steering committee, representative of national policy organizations, mathematics associations, research mathematicians, business and industry, and educators, to develop policy recommendations for the mathematics assessment and to guide the direction and scope of the project. A planning committee of mathematics educators, mathematicians, curriculum supervisors, and teachers was established to draft the content of the framework, working within the policy recommendations established by the steering committee. Care was taken with both committees to ensure that the diversity of opinion regarding mathematics issues was represented and reflected. To reach meaningful consensus, all points of view were heard and considered. Such consensus has been the goal of this project.

A technical advisory panel, composed of university professors, state testing specialists, and measurement experts from private research organizations, was established to consider the policy recommendations and to draft content from the perspective of whether it would threaten NAEP’s ability to continue the short-term mathematics trend lines that began in 1990. The members of all three committees and project staff are presented in appendix C.

The steering committee and planning committee both began their work with a review of the framework used for the 1996 and 2000 assessments. A discussion of the current debates in mathematics education was also part of their meetings. In their deliberations, the committees considered state mathematics content standards and frameworks, new standards prepared by NCTM, reports from the Third International Mathematics and Science Study, reports from the Achieve Project, and the recent report issued by the National Research Council of the National Academy of Sciences, Adding It Up (Findell et al., 2001). In addition, input was provided by mathematics teachers and supervisors as well as others through CCSSO’s partners, the Council for Basic Education and the Association of State Supervisors of Mathematics.

The suggested revisions contained in the framework for the 2005 mathematics assessments are intended (1) to reflect recent curricular emphases and objectives; (2) to include what various policymakers, scholars, practitioners, and interested citizens believe should be in the assessment; (3) to maintain the short-term trend lines in grades 4 and 8 that began with the 1990 mathematics assessment to permit the reporting of changes in student achievement over time; and (4) to include clearer and more specific objectives for each grade level.

Achievement Levels

The updated NAEP Mathematics Framework was considered in light of the three NAEP achievement levels: Basic, Proficient, and Advanced. Basic denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. Proficient represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter. Advanced represents superior performance. These levels are intended to provide descriptions of what students should know and be able to do in mathematics. Established for the 1992 mathematics scale through a broadly inclusive process and adopted by NAGB, the three levels per grade are a major means of reporting NAEP data. The updated mathematics framework was developed with these levels in mind to ensure congruence between the levels and the test content. See appendix A for the NAEP Mathematics Achievement Level Descriptions. Note: Since the grade 12 NAEP mathematics assessment will start a new trend line, the achievement level descriptions will change for this grade. New achievement level descriptions will be finalized in late 2004.

Recommendations for the 2005 NAEP Mathematics Assessment

As a result of analysis and review, the steering committee and planning committee endorsed the following recommendations for the 2005 NAEP mathematics assessment:

1. Content Areas

The 2005 NAEP mathematics assessment should be based on essentially the same five content areas as the 1996 and 2000 assessments: (1) Number Properties and Operations, (2) Measurement, (3) Geometry, (4) Data Analysis and Probability, and (5) Algebra. Details about each of these content areas can be found in chapter 3.

2. Mathematical Complexity of Items

The second dimension of the framework, formerly known as “mathematical abilities,” should be refined to describe the level of mathematical complexity that an item demands of a student. Each level describes the degree of procedural knowledge, conceptual understanding, problemsolving, reasoning, or communicating required to respond to an item at that level. Further description of how this revision relates to the former framework is presented in chapter 2, and the levels are described at length in chapter 4.

3. Distribution of Items

The percentage of items allotted to each of the five content areas should remain the same at grade 4 as called for in the framework for the 1996 and 2000 assessments. The percentages for grade 8 should be altered somewhat in the areas of Number Properties and Operations and Algebra to reflect the increasing importance of algebraic concepts. At grade 12, the percentages should be changed to more closely correspond to the mathematics that high school students experience in a typical 3-year sequence of courses, with algebra and geometry/measurement forming the anchors, along with an increasing emphasis on data analysis and probability. The specific recommended percentages are discussed in chapter 2.

4. Item Formats

Given that NAEP is a paper-and-pencil test administered in a timed setting (for students not receiving accommodations), the assessment should continue to use three types of items: multiple choice, short constructed response, and extended constructed response. As in the previous framework, approximately half of a student’s testing time should be allotted to multiple-choice items, with the remaining half devoted to constructed-response items of both types. Further description of the item formats can be found in chapter 5. For items with multiple-score points, students receive full credit only if they fulfill all the requirements of the item and provide the correct solution.

5. Special Study

A special study should be designed to probe grade 8 students’ understanding of mathematical topics that are critical for further study of algebra and more advanced mathematics. The study would answer questions about the depth of students’ knowledge of such algebraic ideas as rate of change and variable, and their understanding of proportionality. Further discussion of this special study is provided in chapter 2.

6. Manipulatives

The assessment should continue to utilize reasonable manipulative materials, where possible, in measuring students’ ability to represent their understandings and to use tools to solve problems. Such manipulative materials and accompanying tasks should be carefully chosen to cause minimal disruption of the test administration process.

7. Calculators

It is appropriate for some portions (two-thirds) of the NAEP at all grade levels to assess students’ mathematical knowledge and skills without access to a calculator, but for other portions (one-third) of the test to allow the use of a calculator. At grade 4, a four-function calculator should be supplied by NAEP with appropriate training at the time of administration. Eighth- and twelfth- grade students should be allowed to bring whatever calculator, graphing or otherwise, they are accustomed to using in the classroom. More details and discussion about the calculator recommendations can be found in chapter 2. Note: NAGB determined that eighth-grade students will be provided with scientific calculators for the 2005 assessment based on a survey of state testing programs.