Technology and Engineering Literacy Framework for the 2014 NAEP

Chapter One: Overview

The Framework Development Process

The Framework Development Process

In October 2008, the Governing Board awarded a contract to WestEd to develop a framework and specifications for assessing technological literacy. In carrying out its work, WestEd collaborated with the Council of Chief State School Officers (CCSSO), ISTE, ITEEA, the Partnership for 21st Century Skills, and the State Educational Technology Directors Association (SETDA). In working with these groups, WestEd used a process for developing the framework and related products that was inclusive, deliberate, and designed to achieve as much broad-based input as possible.

A two-tiered committee structure with a steering committee and a planning committee provided the expertise to develop the framework as specified by the Governing Board. (See NAEP Technology and Engineering Literacy Project Committees and Staff section for lists of committee members.) The two committees were composed of members who were diverse in terms of role, gender, race or ethnicity, region of the country, perspective, and expertise regarding the content of the assessment to be developed.

The Steering Committee members included leaders from a variety of fields and subject areas, including schools, engineering, education, 21st century skills, the Internet, business, science education, general education, and assessment. The co-chairs were balanced, with one representing technology in schools and the other STEM and assessment. Functioning as a policy and oversight body, this group outlined the Planning Committee's responsibilities in developing the framework. The committee reviewed and provided feedback on drafts of the framework and related materials. The interaction between the two committees was iterative over the course of the project.

The Planning Committee, as supported by the project staff, was the development and production group responsible for drafting the framework, specifications, recommendations for background variables, and preliminary technology and engineering literacy achievement level definitions. This committee was composed of business leaders, researchers, state and district technology coordinators, teachers, and representatives from educational organizations as well as experts in research, assessment, and evaluation. As with the Steering Committee, the Planning Committee co-chairs were balanced, with one being an expert in ICT-based learning and assessment and the other being an expert in K-12 science and engineering education.

Summary of the Steering Committee Guidelines

The following definition of technology and engineering literacy should be used: Technology and engineering literacy is the capacity to use, understand, and evaluate technology as well as to understand technological principles and strategies needed to develop solutions and achieve goals. The Steering Committee also suggested that the Planning Committee examine other definitions of technology and engineering literacy, especially those used by the states.
The assessment should consist of technological content areas, to be reported as scale scores, and technological practices that characterize the field. The content areas must include Design and Systems, Information and Communications Technology, and Technology and Society.
Three practices should be addressed in the assessment: Understanding Technological Principles, Developing Solutions and Achieving Goals, and Communicating and Collaborating.
The content and context for the assessment should be informed by existing state standards and assessments, national and international standards, and research.
Assessment tasks and items should minimize the need for prior knowledge of specific technologies other than ICT; tasks should relate to real-world problems and contexts.
As far as possible, life situations familiar to students and local and contemporaneous conditions should be used to design tasks as a way to confer relevance to each grade level being assessed.
The assessment should focus on a broad base of knowledge and skills, not on specific technological processes that may change.
The assessment should use innovative computer-based strategies that are informed by research on learning and are related to the assessment targets.
The computer-based assessment strategies should be informed by what is known about all learners, including English language learners and students with disabilities.
All the items and tasks included as exemplars in the framework and specifications documents should have feasible responses that can be awarded score points.

The Planning Committee's work was guided by policies, goals, and principles identified by the Steering Committee (see summary box above—for full text, see Appendix B ). In addition to the sources cited previously, the Planning Committee relied on guidance provided by NAEP Technological Literacy Framework and Specifications Development: Issues and Recommendations, a paper prepared by Sharif M. Shakrani and Greg Pearson for the National Assessment Governing Board.

The work was carried out in a series of meetings, with numerous telephone calls and e-mail exchanges between meetings. From December 2008 through September 2009, the Steering Committee met four times and the Planning Committee met six times. Three of the Steering Committee and Planning Committee meetings overlapped so that the two committees could share understandings and discuss critical issues. Governing Board staff supported and participated in all of the meetings. In addition, between formal work sessions, Governing Board members and staff provided ongoing feedback and guidance on project documents and processes.

After the development of initial drafts of the framework, WestEd led a series of outreach efforts to solicit feedback. Formal activities included, but were not limited to, presentations and sessions with industry representatives (e.g., IBM, Cisco), ITEEA, the Organisation for Economic Co-Operation and Development (OECD) International ICT Research Workshop, CCSSO, SETDA, ISTE, and the Partnership for 21st Century Skills Webinar. (See NAEP Technology and Engineering Literacy Project Committees and Staff section for more complete lists of individuals and organizations that contributed to the development of this framework.) The Planning Committee reviewed feedback from these groups as well as from the Steering Committee and made revisions, as it deemed appropriate. Governing Board members and staff closely monitored the project and provided comments on draft documents. After final approval from the Steering Committee, the framework was submitted to the National Assessment Governing Board for action.

[NEXT]

[PREVIOUS]

[HOME]