There May Be Evidence for This, But...

About: Hugues Draelants & Sonia Revaz, L’évidence des faits. La politique des preuves en éducation, Puf

In 2018, the French government announced the creation of a Scientific Council of National Education. Some twenty researchers in psychology, cognitive science, and sociology were brought together to develop recommendations and pedagogical tools for teachers and education policy actors based on the findings of scientific studies. This initiative was representative of a form of public policy known as “evidence-based policy,” which consists in calling on science to guide and justify reforms in all areas of public life: education, medicine, but also justice, sustainable development, road safety, etc. The assumption is that because science provides an objective understanding of reality, it can render political decision-making both more impartial and more effective. From this perspective, one would expect the actors tasked with implementing these reforms (teachers, doctors, etc.) to see them as a means of rationally improving their professional practices. However, the reality on the ground does not match this expectation: Evidence-based policy prescriptions often provoke resistance from professionals, who sometimes go so far as to develop work-around strategies—for instance, as we will see later, Belgian teachers more frequently redirected students when these were not allowed to repeat a grade. How can we understand this tension between scientific knowledge and professional know-how?

Combining an epistemological critique of the scientific evidence mobilized in evidence-based policy with a sociological analysis of the know-how of professionals, Hugues Draelants and Sonia Revaz argue in L’évidence des faits. La politique des preuves en éducation (The evidence of facts: Evidence-based policy in education) that teachers resist because the type of generalization on which scientific evidence is built ignores aspects of reality that they see as essential to acting in situation. The case of education serves as a starting point for a more general reflection on the limits of the use of science in policy-making. However, the authors do not contend that we should abandon the project of using science to improve professional practices or to design public reforms. Rather, they show that knowledge founded on scientific evidence is often of little use to professionals on the ground.

From Statistical Reason to Evidence-Based Policy

The authors begin by reviewing the history of statistical reasoning as part of a broader reflection on the origins of evidence-based policy. Drawing on the work of Alain Desrosières, [1] they recall that statistics served two distinct purposes at the beginning of the nineteenth century: an administrative and descriptive purpose on the one hand (statistics were used to describe states and their populations), and a mathematical and heuristic purpose on the other (they were mobilized to simplify vast and diverse ensembles and to turn these into objects of reflection). These two uses were gradually merged over the course of the nineteenth century, with statistics becoming a mathematical instrument for political decision-making. This merging first manifested itself in the way French hygienists dealt with the cholera epidemic of 1832, which they attributed to unsanitary housing. Using statistical tools to compare cholera mortality in different arrondissements of Paris, they were able to demonstrate that people living in precarious economic and sanitary conditions were more likely than wealthier inhabitants to catch and die from the disease. Here, statistical analysis was used not only to test a hypothesis, but also to justify reforms for preventing unsanitary conditions (sanitation of housing, fight against alcoholism, distribution of food and clothing, etc.).

Over time, the instruments of proof were refined. While in the nineteenth century data were essentially based on averages, in the twentieth century they were generated using more precise mathematical tools (correlation, regression, variance, etc.). Starting in the 1990s, the use of randomized controlled trials became widespread in state agencies, leading to unprecedented rigor of statistical data. One then began to speak of “evidence-based practices” and “evidence-based policy.” However, this historical development was repeatedly challenged by professionals (particularly doctors), who criticized both these tools and the evidence they produced. Thus, doctors who embraced Claude Bernard’s experimental method rejected the numerical tools, arguing that they entailed a simplification of reality since they were only concerned with “the average.” While these tools helped to justify preventive measures for populations considered to be distant from this average, they were often deemed of little use when it came to determining how to manage specific cases.

The new mathematical tools for understanding the social world were also widely used in a variety of ways in the field of education. From the 1950s to the 1980s, all educational policies in the United States rested on the premise that teachers with a solid scientific culture could develop a better understanding of what they were doing and could therefore improve their practice. This approach, referred to by sociologist Martyn Hammersley [2] as the “enlightenment model,” saw scientific knowledge as a resource enabling teachers to reflect more critically on their own work, rather than as an expertise for determining which educational practices should be followed. At the time, scientific studies frequently employed qualitative methodologies that helped to reflect on the variability of the contexts in which educational practices took place. With the emergence of New Public Management in the 1980s, public authorities began to approach education from a more economics-oriented perspective. Accordingly, they turned the school institution into a system of knowledge and skill production aimed at making students efficient, competitive adults, whether in their professional, social, or private lives. The “enlightenment model” was rejected on the grounds that it was unable to meet these new demands and was replaced with a model that relied much more heavily on quantitative methodologies: the “evidence-based education” model, which transformed science into a tool of evaluation (assessing the effectiveness of practices), prescription (justifying the valorization of certain practices), and comparison (defining standards of comparison on a national or international scale).

Automatic Grade Promotion and its Discontents: The Belgian case

However, there remains a gap between the promises of evidence-based policy and the results it yields. To elucidate this gap, the authors explore in detail what happens on the ground when reforms justified by statistical evidence are implemented. They specifically examine the case of the elimination of grade repetition in Belgium. In 1993, education specialist Marcel Crahay published an article entitled “Échec des élèves ou échecs de l’école?” [3] (Failure of students or failure of schools?), in which he statistically demonstrated that grade repetition does not make up for the difference in level between students. In fact, he argued, it is an aggravating factor, since students who repeat a grade tend to make less progress than those who are promoted to the next grade. He also pointed out that, in addition to being inefficient, grade repetition is costly, and argued for reinforcing the logic of efficiency with that of profitability. The critique of grade repetition was institutionally validated in 1994 through a reform introducing automatic grade promotion. The measure, however, had unexpected side-effects. In the eyes of professionals, grade repetition made it possible to group students of the same level, thereby allowing for the creation of homogeneous classes. Following its elimination, teachers were confronted with the challenge of dealing with very heterogeneous groups of students, and so they had to devise ways of conducting within the classroom the kind of differentiation that was previously performed at the school level. Many of them developed work-around strategies so as to group students of the same level without resorting to grade repetition—for instance, by more frequently redirecting struggling students. Given the widespread rejection of the reform, the Belgian government decided to abolish automatic grade promotion in 2001. This example illustrates the kind of problems that can arise when governments adopt reforms based on statistical evidence without taking any real interest in how actors work on the ground. It also shows that the simplification of reality required to produce statistical data is more conducive to the emergence of binary debates (Should one be for or against grade repetition?) than to the critique of specific practices (What constitutes good—or bad—grade repetition?).

Evidence-Based Policy and the Problem of Willful Ignorance

In the final chapter of the book, the authors’ analysis of the resistance of teachers to evidence-based education prescriptions serves as the starting point for a more global reflection. At the heart of this reflection is the notion of “willful ignorance,” derived from the work of the political scientist Herbert Weisberg. [4] According to this notion, statistics temporarily ignore certain dimensions of reality, thereby allowing to uncover aspects of the social world that cannot be perceived on an individual level. Without statistical studies, it would be impossible to establish, for instance, the existence of social inequalities. However, there is a significant risk that this ignorance will cease to be temporary and that the success of statistical data will cause policy-makers to overlook the factors that researchers have failed to consider. For in the test of reality, what scientists have chosen to ignore becomes blindingly obvious. One clearly sees that what was neglected should be taken into account (for instance, the specific social characteristics of a school, the number of students per class, variations in students’ attention during the day, etc.), and that it is the experience of this discrepancy that causes professionals to resist evidence-based policy. The authors describe three forms of willful ignorance in the book. The first is the ignorance of intuition, whereby evidence-based policy is authoritatively imposed on actors in the name of rigorous statistical justification (“There is evidence for this!”) and actors’ professional intuitions are treated as unfounded beliefs. The second form, the ignorance of causes, refers to the fact that statistical studies privilege prediction over understanding, with the notion of “risk factor” replacing that of “cause.” If this approach is met with resistance on the ground, it is because actors are reluctant to act when they do not understand the reasons for acting in this way. The third and last form, the ignorance of singularities, concerns the discrepancy that actors may feel between what experts say about reality and their own experience of it—a discrepancy linked in particular to the fact that experts and actors do not share the same evaluation criteria. Thus, in the Belgian case, experts rejected grade repetition on the grounds that it had no positive long-term effect, whereas for teachers this observation in no way invalidated the practice since they used it precisely to “give students a boost.”

It is regrettable that this global reflection led the authors to take a rather cursory look at the specificities of the field of education. For instance, it would have been interesting to learn more about the resistance of actors in the field. What arguments do these actors put forward, and under what circumstances do they express them? What is the profile of teachers who resist evidence-based policy prescriptions? A contrario, what is the profile of those who least resist such prescriptions and perhaps even welcome them with open arms? Despite this caveat, Draelants and Revaz propose some very interesting analytical tools for reflecting on the limits of the use of science in policy-making, without abandoning the project of a constructive articulation between scientific knowledge and professional know-how.