Education Research Matters

What is the Impact of Ed Tech on Student Achievement?

We are living through a period of profound disruption in human history. It is now beyond argument that digital technologies have transformed so many aspects of how we live, work, socialize, and play. The advent of information and communication technologies has radically enhanced nearly every imaginable human endeavor save one–education. Professor John Hattie, a giant in the field of education research, recently conducted a meta-analysis in which he analyzed over 10,200 studies looking at the effect of various aspects of computers on student achievement. His conclusions? Technology has an average Effect Size of ES=.34. If we were to graph that on a scale from 1 to 10, the average impact technology has on achievement registers a measly 3 out of 10. What is worse is that this meager impact hasn’t changed in over 50 years (Hattie in Magana, 2017). If education research matters, Hattie’s (2017) research findings matter enormously.

To put this into perspective, there is education research, and then there is education research. Professor Hattie’s methods represent the latter; in fact his research is illustrative of the newest direction in scientific inquiry: Meta-Analysis. A brief overview of research literacy is warranted to appreciate the gravity of Professor Hattie’s findings. There are actually 5 levels of research. Let’s take a closer look at each:

5 Levels of Education Research

1) Personal Experience. Personal experience is when one person experiences possible relationships between two variables, like the impact of technology on student learning. The trouble is that the sample size of this type of research is 1. That means that personal experience is synonymous with personal opinion. One person’s opinion is just as good, or just as bad, as another person’s opinion. Another problem is that random factors and systematic errors, or biases can influence personal opinions. These biases can be explicit (biases of which we are aware), or implicit (biases of which we are unaware). Personal experience is clearly not a solid foundation on which to base broad claims. That means there is very low likelihood that one’s personal experience can be generalized, or transferred, to other contexts or populations. Therefore the research value of Personal Experience (one’s individual opinion) is 1.

2) Collective Experience. Collective experience is much like personal experience, except now two or more people share a common experience. The sample size is greater than one, but typically less than what would be considered statistically significant. That means that there is a strong possibility that collective reasoning errors or biases, or random factors, will influence that which is collectively perceived. We can say that collective experience is synonymous with collective opinion. Again, there is low likelihood that collective experiences can be generalized to other contexts or populations. Therefore the research value of Collective Experience (group opinion, case studies, anecdotal stories, or description representations of phenomena) is 2.

 

3) Systematic Observation. Systematic observation is the happy stage where we enter the world of science. This is the entry point for what can be considered legitimate education research because there is an explicit attempt to mitigate, or correct for random factors and biases. Systematic observation is the scientific exploration of phenomena leading to the publication of a peer-reviewed study. The peer review process allows other scientists to look for strengths and weaknesses of the study in order to more accurately surface patterns between variables–like the effect of technology on student learning. Therefore the research value of Systematic Observation (a scientifically controlled and peer-reviewed study) is 3.

4) Compounding Evidence. Compounding evidence is the level where multiple studies purport the same relationship between two variables. It is often the case that a single study will be contradicted by another single study, which leads to the problem of which study is to be believed. When numerous scientifically controlled and peer-reviewed studies (think dozens of studies) show the same patterns of relationships between variables, this obviously strengthens the argument of influence, or causation, between two variables. Therefore the research value of Compounding Evidence (dozens of scientifically controlled and peer-reviewed studies) is 4.

5) Preponderance of Evidence. Preponderance of evidence is the level where a very large number of studies are analyzed to determine the effect of influences on some outcome, again, like the effect of technology on student learning. This is the level of meta-analysis. Developed and refined by the legendary researcher Dr. Gene V. Glass at the University of Colorado in Boulder, meta-analysis actually describes the process analyzing thousands or tens of thousands of studies to strengthen the observed relationship between variables. Meta-analysis provides a much broader, in-depth, and accurate approach to determining the impact of influences on student learning and achievement. Therefore the research value of Preponderance of Evidence (the analysis of thousands or tens of thousands of scientifically controlled and peer-reviewed studies) is 5. This is the highest level of research that exists today. John Hattie’s (2017) research is clearly at this level.

It is quite easy to argue against personal or collective opinion using systematic observation, compounding evidence, or the preponderance of evidence. However, it is very difficult to argue against the preponderance of evidence using any of the other levels of research–particularly in the context of education. In the education profession, research matters. My hope for this blog post, and others to follow, is to build a kind of research literacy among education leaders and practitioners for the purpose of understanding and using the highest quality research evidence to improve instructional quality and student learning. Arguably, that matters most of all.

Author

Dr. Sonny Magana is an award-winning teacher, best-selling author, and pioneering educational technology researcher. Sonny is a highly sought-after leadership consultant, keynote speaker, and instructional coach who helps educational systems around the world realize the power of transcendent learning. The author of numerous research studies and articles, Sonny’s newest book, Disruptive Classroom Technologies: A Framework for Innovation in Education, was recently published to wide international acclaim.

A tireless advocate for transcending the status quo, Sonny founded and served as Principal of Washington States first CyberSchool in 1996, a groundbreaking blended learning program that continues to meet the needs of at-risk students in Washington. He is a recipient of the prestigious Milken Family Foundation National Educator Award and the Governors Commendation for Educational Excellence.

An avid musician, yoga practitioner, and beekeeper, Sonny holds a bachelor of science degree from Stockton University, a master of education degree from City University (where he was honored with the Presidential Award for meritorious scholarship), an educational administration endorsement, and a doctorate in educational leadership from Seattle University.