The "Education Business:" Ask Hard Questions

Education software and related products are an $8 billion global industry Links to an external site..   The USA  Every Student Succeeds Act of 2016 Links to an external site. (ESSA) strongly backs "evidence-based" materials.  But no federal agency conducts independent evaluations of commercial educational software.   School districts usually rely on "insider" statistics provided by the for-profit companies.

1.  As a statistics student, what possible sources of bias might you be alert to  when reading reports of "insider" studies?  How might such biases show up in the reports?

2.  The Hechinger Report Links to an external site. promises to provide "in-depth, fact-based, unbiased reporting on education that is free to all readers."  What more might you want to know about this organization as you read its reports?   

A Hechinger Report review Links to an external site. states that dozens of companies promote the effectiveness of their products, including some that sell very widely-used education software.  Many overstate their effectiveness based on very limited evidence or on effects not seen by independent evaluators.  Some "tout student growth that their own studies admit is not statistically significant."   

3.  What is meant by "statistically significant"?

The following table is reproduced from materials produced by a company called Edgenuity Links to an external site. in the article quoted above Links to an external site..

4.  Study this side-by-side bar graph.  State at least two things that it appears to show.  Then write at least one or two questions that are not answered by the graph but could give more context to the information it presents.

The Hechinger study Links to an external site. noted that Edgenuity used this chart to claim that use of its math course supposedly led to an eightfold increase in pass rates on state math tests at Altamont High School in Utah.   But the school only introduced Edgenuity in the 2015-2016 school year.  

5.  Where do you think Edgenuity got its "eightfold increase" estimate?  Are you convinced?    Knowing that the school started with Edgenuity in 2015-2016, what would be your estimate of the increase in test scores?  

6.  Most students who were in 9th grade in 2015 were probably in 10th grade in 2016 and so forth.  How do you think that might affect your interpretation of the graphs, if at all?

7.  What is the measure of success being used in this table?  Comment on the pros and the cons of using this measure.

Now post an observation, based on this exercise and your own K-12 math experiences, in the Discussion:  The Education Business.