Statistical Relationships

Statistical Relationships Methodology

In a statistical linking study the same sample of students take both NAEP and another relevant assessment. Using these data the relationship between the two tests can be evaluated and, if supported, a statistical link between the two can be established. This link allows a score on one test to predict a score on the other test. The degree to which the linked scores have the same meaning depends both on the strength of the statistical relationship, expressed as a correlation, and the content overlap of the two exams.

A statistical linking study was conducted between NAEP and the SAT, using the scores of more than 15,000 public school students in grade 12 in 2009 that took both tests. The matched sample accounted for approximately a third of the 12th graders in the NAEP sample, about the same proportion that take the SAT nationwide. This work was conducted via a data sharing agreement with the College Board.

The researchers used two methods to link the exams: concordance and projection. The concordance established an equipercentile link in which scores were matched along the full distribution of achievement. For example, the score at the 99th percentile on one test was matched with the score at the 99th percentile on the other, and so on to the 1st percentile. Each score on one test was given a corresponding score on the other. A strong correlation between the two assessments is needed to rely on concordance for establishing links between the two tests.

In projection, a less stringent method for linking two tests, scores on one test are related to another using the mathematical procedure of linear regression. This produced a series of probabilities of achieving a particular NAEP score, given a particular SAT score. The researchers gave projected NAEP scores for probabilities of 50, 67, and 80 percent.

An additional set of statistical linking studies was conducted using projection to link 2013 NAEP 8th grade reading and mathematics scores of students in three states—Tennessee, Kentucky and North Carolina—with their scores on the ACT EXPLORE® reading and mathematics assessments. This work was conducted via data sharing agreements with these states.

A longitudinal study collects data at different points in time on the same sample of students. For NAEP preparedness research, data are collected on samples of students who have taken 12th grade NAEP via data sharing agreements with states. These data provide a follow up to show how performance on NAEP relates to later outcomes, such as the courses taken and grades received in college, and potentially other activities.

The longitudinal analyses used data on Florida students that participated in the 2009 NAEP 12th grade assessments, approximately 3,200 in mathematics and 3,400 in reading. Information was obtained through the state's longitudinal data system on the students’ enrollment status in colleges and universities, first-year course placements, and the grade-point average of those attending public colleges in Florida for the 2009-2010 academic year--about 54 percent of the NAEP sample. Students in Florida community colleges comprised 36 percent of the original sample, and approximately 17 percent attended Florida four-year colleges and universities.

The Florida database was also used to obtain information on student performance on the SAT and ACT college entrance examinations, and on ACCUPLACER, a college course placement exam used to determine students needing remedial coursework. NAEP results were compared to first-year performance in college and to the other tests available.

Additional longitudinal studies are currently underway using 2013 NAEP reading and math data at grades 8 and 12, via data sharing agreements in a few states.

Statistical Relationships Key Conclusions

Key Conclusions for the 2009 NAEP-SAT National Linking Study

• The correlation is very strong between NAEP and SAT in mathematics (0.91). It supports the use of a concordance in relating the two exams.
• The NAEP-SAT correlation in reading is lower (0.74). That is not sufficiently strong to support a concordance, though projections may be used for linking group results on the two exams.
• For the SAT reading benchmark of 500, the NAEP-SAT concordance score and the projected NAEP score using a 50 percent probability are both very close to the NAEP cut score for Proficient.
• In mathematics, the SAT benchmark of 500 is below the NAEP Proficient cut score when using either the concordance or a 50 percent probability projection linking method. At the 80 percent probability projection, the SAT benchmark of 500 is very close to the NAEP cut-score for Proficient.

Key Conclusions for 2013 NAEP-EXPLORE Linking Studies in Three States

• The correlations between NAEP and ACT EXPLORE in mathematics ranged from 0.72-0.74 in reading and from 0.81-0.82 in mathematics. These correlations are not sufficiently strong to support a concordance, though projections may be used for linking group results on the two exams.
• In all three states, for the EXPLORE reading benchmark of 16, the projected NAEP score using a 50 percent probability is close to the NAEP cut score for Proficient.
• In all three states, for the EXPLORE mathematics benchmark of 17, the projected NAEP score using a 50 percent probability is close to the NAEP cut score for Proficient.

Key Conclusions for Longitudinal Study of Florida Students in 2009 NAEP Sample

• Overall, the analyses of Florida data confirm the national linking study of NAEP and the SAT. Further, these analyses provide additional evidence about potential points on the NAEP scale relevant to academic preparedness for college.
• The average NAEP score of students reaching college-readiness benchmarks on either the ACT or SAT was slightly above the Proficient cut score on NAEP in both reading and mathematics.
• The average NAEP score of students qualifying for credit-bearing courses through ACCUPLACER, based on the cut scores used in Florida, was between Basic and Proficient in both subjects. In Florida, the ACCUPLACER is given primarily in community colleges.
• Students in remedial courses had much lower NAEP scores than students who were not required to take such courses. The average for college freshmen not placed in remedial courses was 3 points below Proficient in reading and 11 points below Proficient in mathematics. Students taking remedial courses averaged about 30 to 40 points lower than non-remedial students—at the Basic level for reading and below Basic in mathematics.
• Students achieving a first-year college grade point average of B- or better had much higher NAEP scores than those with lower grades. The NAEP average of the B- or better students was 4 points below NAEP Proficient in reading and 14 points below NAEP Proficient in mathematics. The data for these analyses include both credit-bearing and remedial courses in both four-year and community colleges.
• The Florida results generally indicate that the region around Proficient would be a reasonable benchmark for academic preparedness for college work.