Explainer: Student Achievement in Madison

Here is a red-hot take.  I am going to say something positive about David Blaska’s run for school board.  I appreciate that he seems to be the only School Board candidate bringing up and asking questions about measurable student achievement in MMSD.

On his website, Blaska includes among his criticisms of the school district that over the last five years, MMSD’s score on the report card prepared by the Department of Public Instruction has gone down, as has the “achievement” component of DPI’s report card formula.  He also points out that “the racial achievement gap persists,” again as measured by DPI-administered tests.

As George W. Bush famously said, “Rarely is the question asked, is our children learning?”  To his credit, Blaska has asked the question.  Let’s try to answer it.  To do so, we have to look at the results of standardized tests.  But we should address some preliminaries before we dig in.

First, standardized tests have limited utility.  It would be a mistake to assign too much significance to their results.  There is much more that goes into the educational effort than can be measured on multiple-choice answer sheets or computer screens.

We tend to overvalue standardized test results because they are quantifiable and provide a basis for comparison.  But judging and ranking schools by the results of standardized test results is like judging and ranking basketball teams on the basis of the average height of their players.  Sure, it’s both quantifiable and relevant, but there is so much more that goes into success on the basketball court, as in the classroom.

So, why am I bothering to write this?  Because a school district cannot simply ignore the standardized tests and report cards that the state has identified as means of accountability for school districts. The results are out there; school districts have to deal with them.  Families moving to the Madison area and researching schools will certainly check them out.  If you go to the Waunakee school district web site, the first thing you see is something about the school district report cards.  (Understandably, since Waunakee, with the highest percentage of white students in the Madison area, perhaps not coincidentally also scored the highest on the report cards as well.  But read all the way to the end of this post for some interesting context.)  

Second, yes, test results for students of color are far lower than anyone would like them to be. It would be a worthy endeavor to try to unpack why that is, but that’s way beyond the scope of this post.  In general, I am taking the scores as they are and trying to explore their trajectory – is it up, down, or flat?  We can certainly agree with the statement that standardized test scores for students of color should be much higher, but if that is where an argument criticizing the school district both begins and ends, it simply does not tell us very much.

Third, we always need to bear in mind that average test scores achieved by groups of students are not attributable to each student in that group.  Individual performances can vary widely.  For example, any particular Black student may outperform any particular white student.  Individual students are not captives of their demographics.  But aggregate performances of groups of students still have much to tell us.

Now, let’s move on to DPI’s report cards. It is difficult and invariably misleading to draw comparisons about the performance of school districts and schools.  Different schools have different demographic mixes, and, on average, groups of students differ in predictable ways in their performance on standardized tests.

One school with a relatively high number of low-income students for example, might provide a better educational experience than a second school whose students are predominantly upper-income.  Even so, the students in the second school are quite likely to have higher scores on standardized tests, and so that school is likely to be misleadingly labeled as a more successful school than the first.

DPI’s report card scores are not solely based on a straightforward ranking of test score results and do attempt to take demographic differences into account. The effort is not entirely successful. Since DPI employs a very complicated formula to come up with its report card scores, a formula that it seems to continuously tweak, the process is a black box to most.  As a result, most observers don’t really know what to make of the scores that result.

As it happens, the Madison school district’s report card score has bounced around in a narrow band over the years as the DPI formula has changed.  The scores have been: 68.5, 69.8, 67.5, 66.5 and 68.0.  There is no clear trend that jumps out of those figures.

The Forward Exam: Wisconsin’s Standardized Test du Jour

This brings us to the Forward Exam, which is the standardized test that Wisconsin students from third through eighth grade have taken since the 2015-16 school year. Results on the Forward Exam, massaged in different ways, are the biggest input into the school district report cards that DPI issues and that Blaska has cited.

Rather than attempt to tease out the nuances of DPI’s school district report cards, it seems more instructive simply to look at how MMSD students have done on the English and math components of the Forward Exam.   How have Madison’s scores been?

The short answer: not so great.

There are many ways to categorize Forward Exam results.  The most common is to look at the percentage of students who are deemed “proficient” because they score in the higher two (advanced and proficient) of the four categories of results.

Here are the percentages of MMSD students labeled proficient on the English and Math portion of the Forward Exam during the last three years:

The percentages increase in each category each year, so that’s a good thing.  But for 2017-18, as for previous years, MMSD’s percentages of proficient students also fell below the state average for each category, which isn’t so good (42.4% to 36.6% in English and 43.8% to 38.2% in math).

We need to dig deeper. Most of the School Board candidates have acknowledged that the achievement gap is the most pressing challenge facing the school district. Because it is, we should take a look at the Forward Exam data disaggregated by race.

Here they are:

On the English part of the test, the percentages for Asian students declined, the percentages for white students increased, and the results bounced around for Black and Hispanic students.  Math results look better.  Again, the percentages of Asian students declined and the percentages of white students increased.  But this time the percentages for Black and Hispanic students also increased.

The results for students of color are no cause for celebration.   As has been the case for many years, the percentages of MMSD students of color who score proficient or advanced are lower than the statewide averages for their peers, while MMSD white students outperform their statewide peers.  This is shown on the following tables of 2017-18 data:

These tables illustrate the scope of the achievement gap in Madison.  On a statewide basis, Wisconsin has one of the most pronounced achievement gaps in the country.  And yet the results for Madison show our students of color underperforming students throughout the state while our white students easily outpace their statewide peers.  The result is an even wider achievement gap for Madison than for the state.

This isn’t the end of the story.  There are legitimate reasons to question the reliability of the Forward Exam results for Madison students.  The Forward Exam results are not available until after the end of the school year, so they serve no pedagogical purpose for teachers.  Partly as a result, the school district simply doesn’t pay much attention to the test.  There is no organized effort to prepare students for their test-taking.  In addition, a significant number of families opt their children out of taking the test, which reduces the reliability of the results.

MAP: Madison’s Preferred Standardized Test

MMSD prefers the Measures of Academic Progress (MAP) test for assessing how our students are learning. Most Madison students take the MAP in reading three times a year and the math MAP twice.   Results are almost immediately available, so they are timely enough to inform teachers’ approaches to their students’ learning.

The MAP measures students’ proficiency in English and math, as well as students’ growth.  The growth figure is calculated as a percentage of students who do better on the test then would have been predicted in light of their past performance, so there is a longitudinal aspect to MAP that the Forward Exam lacks.  The school district’s Strategic Framework identifies key MAP results as the primary metrics by which the district will measure its progress on its first Strategic Framework goal of enhanced student achievement.

The MAP results look better than the Forward Exam results:

Overall, these figures show that there has been positive and generally steady gains in the measures that the Strategic Framework has identified as particularly important.  This is encouraging.  We see more progress at the elementary school level than at the middle school level.

But what about the disaggregated data?  Do the overall figures mask significant variability in the levels of progress made by different groups of students?

Let’s take a look. First, here are the same figures for Asian students:

Black or African American students:

Hispanic/Latino students:

And white students:

It is good to see that all student groups are making progress, at least as measured by MAP results. This indicates that the school district’s approach to instruction seems to be working.  (If you are curious about other ways of looking at the MAP data, you can find much more here.)

So, if we focus on MAP results and the metrics that have been identified as important in the district’s Strategic Framework, we can feel encouraged by the school district’s progress. If we look at DPI data and the results on the Forward Exam, any positive feelings would be considerably more muted.

Is there any way to make sense of the differences?  How much confidence can we feel in the claim that the Forward Exam results can be discounted because the test is so de-emphasized in MMSD?

ACT: The Standardized Test Where the Results Matter to Students

There is one more set of test results we can look at – the ACT taken by high school students.  This is a test administered to nearly all Wisconsin high schoolers and so there is a basis for comparison of the results for MMSD students with those for students throughout the state.  In addition, the ACT is a meaningful test.  Students care how well they do, given the role it plays in college admissions.  It avoids the shortcoming of the Forward Exam, where the results for individual students have no consequences.  Of course, the ACT is still a standardized test and subject to the limitations and shortcomings of all such tests.

Here are average composite 2017-18 ACT scores for MMSD and statewide groups of students:

With the exception of MMSD Black students who score just a tenth of a point lower than Black students statewide, MMSD students did better on the ACT than their statewide peers.  Given MMSD’s dismal history of educating students of color, at least as measured by standardized test results, this actually represents progress.   In addition, these results look strikingly different from the Forward Exam results shown above, where Madison students of color did consistently worse than students of color in the rest of the state.

This is evidence that the unimpressive performance of MMSD students of color on the Forward Exam can be attributed at least in part to the extent to which MMSD deemphasizes the test. This in turn lessens the extent to which the encouragement we can feel about the MAP results should be tempered by concern about the Forward Exam results.

The ACT results also highlight a hidden fact about MMSD: white students do quite well here, at least as measured by standardized test results.  It is not politically feasible for the school district to tout its successes educating white students while students of color continue to struggle in such a pronounced way.  Nevertheless, the data are clear.

Here are the 2017-18 average composite ACT scores for white students at the public high schools in Dane County, ranked from highest to lowest:

On his campaign website, David Blaska highlights the claim that  “Parents [are] fleeing Madison schools.”  We can safely assume that, compared to the school district’s demographics, white families are over-represented in this group.  As the ACT results show, odds are that these families are “fleeing” to schools that are lower-performing for their kids.

Here are some quick takeaways from all these figures:

• To state the obvious, the achievement gap is clearly the most challenging and urgent problem facing the school district. The emphasis on Black Excellence in MMSD’s most recent Strategic Framework is appropriate and necessary.
• As I have written before, it takes a long time for large school districts to show sustained, system-wide improvement.  Relatedly, incremental progress is good progress.  That said, it would have been reassuring to have seen greater progress in the performance of MMSD students of color on standardized tests over the last couple of years.
• In light of this, it will be very interesting to see how successful One City proves to be with its encompassing approach to teaching young Madison students of color. From this perspective, One City actually seems kind of close to Albert Shanker’s original vision for charter schools as genuine “educational laboratories” for the ultimate benefit of traditional public schools.
• Significant gaps in students’ knowledge and skills show up from the first time they are measured.This underscores the importance of early education, a point raised by School Board candidates Kaleem Caire and Albert Bryan.  However, I think there is an unresolved question about which of our public institutions are primarily responsible for helping with the development of our children before they reach school age.
• The school district should continue to pay particular attention to LaFollette to help ensure a productive learning environment for all it students. It would also be good to take a look at why the scores of Asian students have been decreasing on the Forward Exam and why they have seen less robust progress on the MAP results than other groups of students.
• On the other hand, raise your hand if you thought white students at East would outperform white students at Middleton and Waunakee on the ACT. I don’t see Waunakee touting on their website that their high-schoolers score almost as high on the ACT as white kids at East.
• David Blaska should visit East. I bet someone he knows could give him directions.  It’s the handsome old building on East Wash.