As this framework is written, the use of computer-based scenarios for assessment purposes is an emerging, but growing area. The 2009 NAEP Science Framework called for the use of interactive computer tasks as part of its assessment, and in 2009 three long and six short interactive tasks were administered to national samples of students in grades 4, 8 and 12. Another set of interactive assessment tasks has been used in state tests of science achievement in an Enhanced Assessment Grant project funded by the U.S. Department of Education to determine how simulation-based scenarios in science might form part of district and state accountability systems. As assessment developers gain more experience in this emerging field, they will develop a better sense of how to create the tasks efficiently and how to ensure that the tests produce valid and reliable measures.
Here a descriptive outline is provided of the main features of the scenarios that will be developed for the NAEP Technology and Engineering Literacy Assessment. At the beginning of the scenario, it is important to set the context for the activities in which the student will be involved. This introduction provides a setting for the assessment tasks that, as far as possible, should reflect tasks that might be performed in society, either within an academic setting or outside of school. In addition, near the start of the scenario, a motivating question or goal will be introduced. This goal provides the driving rationale for the tasks that the student will perform, and it offers a storyline that helps define the relevance and coherence of the tasks and motivates the student to undertake them. The motivating goal might be to solve a particular problem or to achieve a certain goal within the scenario.
An advantage of computer delivery of the assessment is that the introductions to the scenarios can use appropriate multimedia to present the settings for the assessment tasks. As a result, there is less need for text and therefore less of a reading demand. The multimedia can include video segments or animations that the student observes, and it will also generally use text, numbers, and graphics to convey information necessary for the tasks to be accomplished. All of the representations, such as graphics, video, or simulations, must be carefully chosen to serve a purpose in the assessment tasks, and none should be present simply for visual interest.
One type of assessment scenario will include a representation of a system. Depending on the context for the particular scenario, this might be an engineering system such as an irrigation system or a dam. Whatever the system, it will have components that are dynamically related, so that a student can observe the role of a particular component (e.g., watch what happens when a valve is opened in an irrigation system) or interact with a component (e.g., by setting a value for a parameter or moving a part of the system) and see a resulting change of state in the system (e.g., a rise in water levels or flooding of fields).
A second type of scenario will lay out an overarching goal or problem which students will reach or solve by conducting various interrelated tasks. Such a goal might be, for instance, to develop a play on a historical period. Component tasks could involve searching for information about a famous incident in the period, analyzing the necessary ticket sales, and creating a playbill advertising the production.
Within a scenario, students may be asked to select tools from a toolkit and use them within the system. Students might be asked, for example, to select a graphing or spreadsheet tool or to use a simulation. Various tools may be made available, depending on the scenario. Word-processing, texting, or presentation tools might be available for communication tasks, for example, and Web design or page layout tools might be used for the presentation of large amounts of information. For some scenarios it might be appropriate to provide more specialized tools, such as computer-aided design, geographical information systems, or video editing tools.
By interacting with the components of the system or task that are key elements for achieving the goal, students are able to respond to tasks that ask them to explore alternative outcomes, control certain variables, and observe the resulting changes in the system. The students can observe and describe the patterns or characteristics of the outcomes and can interpret the feedback from the system. They can then evaluate the outcomes of the choices they made in manipulating the components of the system or in using particular tools, and, finally, they can form conclusions.
In some cases it might be necessary to simulate virtual features of real-world equipment that can be used within a scenario. For example, a temperature gauge might give feedback from a heating system, an anemometer might be used to measure wind speed in a scenario about wind turbines, or a table might be used in an ICT scenario requiring the collection of data about the types of symbolism in Shakespeare's plays. Alternatively, graphics or images might be constructed or selected to communicate a design or idea.
In providing tools in a scenario, it is necessary to determine which elements of a tool are necessary for the activities in the scenario and which features of the tool will be used by students. It is not necessary to provide or simulate a fully featured version of a tool. For example, only certain functions of a spreadsheet tool might be provided in order that students could take a table of data resulting from actions in the system and transform it into a graphical representation of their choice (a line graph, say, or a bar graph or pie chart). It would not be necessary to provide all the other features of the spreadsheet tool, and, in fact, it would be distracting to students and produce measurement "noise."
Throughout their interaction with a system, students may be asked to use tools to find relevant resources; to communicate to others about their actions, decisions, or results (e.g., texting a virtual team member); or, at the end, to convey their conclusions (e.g., creating a slideshow presentation).
The discrete item sets will comprise approximately 10-15 stand-alone items in either selected or constructed response format to be completed within a 25-minute block. These items would not be part of a complex scenario or related to one another. Each discrete item would provide a stimulus that presents enough information to answer the particular question posed in the stem of the item. Items in discrete sets will be selected response items (e.g., multiple choice) or short constructed response items in which a student writes a text-based response.
