Assessment Design and Student Diversity

Students should have the opportunity to demonstrate their knowledge of the concepts and ideas that the NAEP Technology and Engineering Literacy Assessment is intended to measure. The assessment needs to be responsive to the challenges that stem from an increasingly diverse student population in the nation, the inclusion of all types of students in the general curriculum, and an increased emphasis and commitment to serve and be accountable for all students. NAEP should strive to develop assessments that allow for the participation of the widest possible range of students so that interpretation of scores of all who participate leads to valid inferences about the levels of their performance.

As the NAEP Technology and Engineering Literacy Assessment will be computer-delivered and will contain a large proportion of scenario-based assessment items, great attention needs to be paid to ensuring that the assessments are valid and accessible to a wide range of students. Two types of populations need to be considered in order to properly address student diversity in the design of the assessment, English language learners—students who are developing English as their second language—and students with disabilities. Both populations are tremendously heterogeneous; the former, because of the wide range of students' first languages, cultural influences, and stages in English development; the latter, because of the wide range of disabilities—from physical to sensory to cognitive. Whereas both populations may have common sets of needs, some design issues may be particularly relevant to one or the other. Both populations should be considered carefully in the design of the NAEP Technology and Engineering Literacy Assessment.

To effectively include English language learners and students with disabilities in the technology and engineering literacy assessments, two areas that need to be addressed include the participation of students in the pilot stages of test development and the use of testing accommodations. These aspects are discussed in more detail in the Assessment and Item Specifications for the 2014 NAEP Technology and Engineering Literacy Assessment.

Pilot Student Samples

The term "inclusion" does not refer solely to the participation of students with special needs in large-scale testing. It also refers to the notion that, to attain equity in testing, appropriate actions should be taken to ensure that these students participate in the process of assessment development.

Issues related to students with special needs should not be addressed only at the end of the process of assessment development, guided by the erroneous belief that testing accommodations provided during test administration will ensure equitable testing. Rather, English language learners and students with disabilities should be included in all of the assessment tryout stages in which information is collected from pilot students about the ways in which they interpret stimulus materials and test items and the difficulties they experience in providing their responses. Even when assessments are to be administered only in English, English language learners can provide test developers with valuable information for improving the assessment (for example, the wording of its items). Also, the information provided by students with disabilities can help to identify and address usability issues that are relevant to administering the assessment to all students.

A potential challenge in the testing of students with special needs is the fact that the critical characteristics of these students are not always carefully identified (Solano-Flores, 2009). As a result, they are not properly represented in the samples of students who participate in the pilot stages of assessment development. The use of very small samples of English language learners and students with disabilities makes it difficult for test developers to properly address the heterogeneity of these groups. Thus, preparation for the operational assessment should address the diversity within the population of English language learners (for example, diverse first language backgrounds and different levels of English development) and within the population of students with disabilities (for example, diverse types of disabilities and various levels of severity of disability).

Universal Design

The items in the NAEP Technology and Engineering Literacy Assessment that make use of the multimedia capabilities of the computer offer a chance to more easily reach a wide range of students than traditional paper-and-pencil items allow. Universal design is a concept that started in architecture and has been applied to assessment by identifying the relevant essential elements for assessment (Thompson, Thurlow, & Malouf, 2004). Since its early explication, the concept has been applied to instruction as universal design for learning (Orkwis & McLane, 1998; Rose & Meyer, 2002). At the time of the writing of this Framework, NAEP is considering how the principles of universal design might be applied in its assessments, but no policy has yet been adopted.

The Universal Design for Computer-Based Testing (UD-CBT) framework and a detailed set of UD-CBT guidelines (Dolan et al., 2007) specifically address the design of novel computer-delivered assessments such as the one that will be developed for the NAEP Technology and Engineering Literacy Framework. The UD-CBT represents, at the time of publication of the NAEP Technology and Engineering Literacy Framework, a useful guideline for developing interactive scenario-based assessments. However, developers of the assessment will need to pay attention to developments in the field as a greater number of innovative assessments are developed and more information is available about how to apply the principles of universal design to them.

Accommodations

NAEP strives to assess all students selected by its sampling process. Rigorous criteria are applied to maximize the number of English language learners and students with disabilities included in NAEP assessments. Participating students with special needs are permitted to use accommodations, as stated in current NAEP policy:

All special-needs students may use the same accommodations in NAEP assessments that they use in their usual classroom testing unless the accommodation would make it impossible to measure the ability, skill, or proficiency being assessed, or the accommodation is not possible for the NAEP program to administer (NCES, 2005, Current Policy section, ¶ 4).

To meet this commitment, it will be necessary for the assessment delivery system to include tool options that allow the students to benefit from the sorts of accommodations that they need. An example of these tool options is screen magnification, which can benefit blind or low vision students. Another example is text-to-speech, which may also benefit blind or low vision students as well as English language learners whose proficiency in English is better in the listening mode than in the reading mode.

To make accommodations more likely to successfully serve students with special needs, an important fact should be taken into account in assessment design. Each English language learner is unique as to his or her level of English proficiency in the listening, speaking, reading, and writing modes and each student with disabilities is unique as to the kind and severity of his or her disabilities. As a consequence, what works for one student does not necessarily work for another student within the same group of students with special needs. Designing an assessment delivery system that is capable of providing all NAEP-authorized accommodations and that allows selection of the set of accommodations that best meets the needs of each student is critical to properly meeting the goals of inclusion in the NAEP Technology and Engineering Literacy Assessment.