The University of San Francisco
USF Scholarship: a digital repository @ Gleeson Library | Geschke
Center
Doctoral Dissertations All Theses, Dissertations, Capstones and Projects
2023
WE ARE STILL PLAYING: A META-ANALYSIS OF GAME-BASED LEARNING IN MATHEMATICS EDUCATION
Thomas Conmy
Follow this and additional works at: https://repository.usfca.edu/diss
Part of the Educational Technology Commons, and the Mathematics Commons
The University of San Francisco
WE ARE STILL PLAYING: A META-ANALYSIS OF GAME-BASED
LEARNING IN MATHEMATICS EDUCATION
A Dissertation Presented
to
The Faculty of the School of Education
Learning and Instruction Department
by
Thomas Conmy
San Francisco
December 7th, 2023
Abstract
WE ARE STILL PLAYING: A META-ANALYSIS OF GAME-BASED
LEARNING IN MATHEMATICS EDUCATION
The purpose of this meta-analysis was to investigate the effectiveness of the use of games as part of mathematics instruction on academic achievement in grades Kindergarten to 12 in the United States. There were 17 studies selected for investigation published from 2010 to 2023 that focused on game-based learning and mathematics. This meta-analysis fills the gap in the knowledge by examining classes that are using game based learning across three platforms of instruction: nondigital games, digital on computers, and mobile devices. The findings from this meta-analysis suggest that the usage of game-based learning in a classroom has a positive effect on students’ mathematics achievement in addition to the suggestion that the findings can apply to more academic domains beyond mathematics.
The moderator variables that were examined in this inquiry were category of games (if a game used was classified as a serious game, an educational game, or a simulation), game platform (if the digital game was played on a computer or a mobile device), the students’ grade level, gender, and frequency of the game-based learning activities. The overall effect size of using game-based learning in the classroom was 0.30 and there were statistically significant findings regarding gender comparisons and grade level comparisons. Out of the 14 moderator analyses conducted, 12 were found to be of statistical significance.
One of the difficulties identified was classifying studies that used mobile devices given that most studies used the term “apps,” which is not sufficient for classification as one does not know if apps referred to the type of digital device or an app on an iPad, phone, computer, or other technological device.
The designing of lessons plan with game-based learning activities requires several factors. Beyond just the technology selected, the type of game and if the game required any modifications that would need additional investigation. A teacher’s familiarity with the game used in the classroom also would be a benefit.
Additionally, educators seeking to use game-based learning should include considerations for how the instructional time is used for game-based learning. The findings of this study provide suggestions for the length of time students should spend on a game and how often games should be used for learning.
The findings from this meta-analysis provide the implication that game-based learning could be used beyond just mathematic education, as statistical significance was found regarding problem-solving activities. In addition, future research should consider the definitions of in the field of game-based learning and the changing technology platforms.
This dissertation, written under the direction of the candidate’s dissertation committee and approved by the members of the committee, has been presented to and accepted by the Faculty of the School of Education in partial fulfillment of the requirements for the degree of Doctor of Education. The content and research methodologies presented in this work represent the work of the candidate alone.
Thomas Conmy December 7th, 2023
Date
Candidate
Dissertation Committee
Dr. Patricia Busk December 7th, 2023
Date
Chairperson
Dr. Devanshi Unadkat December 7th, 2023
Dr. A.S. Helen Maniates December 7th,
Acknowledgements
I have been thinking about writing this section for at this point literally years. Just how can I convey my gratitude, thanks, and sheer appreciation for all those that came and went in my life during this journey. Each moment, big and small, helped to shape my work, my outlook, my very nature as I completed this project.
To begin, I must first thank Dr. Patricia Busk, the chair of my dissertation committee, who has since been a mentor and an advocate for me since I applied to the University of San Francisco back in November 2014.
I want to extend my appreciate and thanks to my Dissertation Committee members Dr. Devanshi Unadkat, Dr. Helen Maniates, and Dr. Xornam Apedoe, each who has been a part of this journey to the end by supporting my ideas and providing feedback, support, and belief in my work.
To the professors and staff of the University of San Francisco, in addition to those I have worked with before, each of you helped me get to where I am today, and I am truly grateful.
My USF cohort, Polin, Emma, Patricia, Elizabeth, and Kim, thank you for dealing and working with this loud punk of a student. We are forever a family as we move into the next phases of our postprogram lives.
My family and my friends; many of you have seen me as the overworked and chaotic academic for the past decade; thank you for the willingness to look past my busy schedule and the chaos of my life. Each of you has made it a bit brighter.
My MTG Judge Family, while it may be strange, I wanted to give a shout out for my growing found family; as we are all students in the program of self-improvement, thank you for helping when times were tough.
To my loving partner frances, my moon and stars, thank you for being a guide in my final moments of completing this program. The labor of love to ensure I had all the coffee in the world and that I wasn’t distracted by another random YouTube dive; to the unwavering support and belief that I could accomplish this goal. I look forward to our next chapter together.
To the lofi hip hop radio beats to relax/study YouTube channel, thanks for the beats to keep studying during long writing sessions.
Lastly to you dear reader, thank you. Thank you for taking the time and the energy to look over this document. In these pages I hope you see some interesting findings, the outlines of a growing academic, and if we are both lucky, a bit of inspiration.
Cheers.
TABLE OF CONTENTS
ABSTRACT………………………………………………………………………………………………………..
SIGNATURE PAGE…………………………………………………………………………………………….
ACKNOWLEDGEMENTS……………………………………………………………………………………
TABLE OF CONTENTS……………………………………………………………………………………..
LIST OF FIGURE……………………………………………………………………………………………….
LIST OF TABLES……………………………………………………………………………………………….
CHAPTER
I. INTRODUCTION……………………………………………………………………………………..
Statement of the Problem…………………………………………………………………………….
Introduction to game-based learning………………………………………………….
Summary of selected meta-analyses in game-based learning…………………
Purpose of the Study………………………………………………………………………………….
Theoretical Rationale…………………………………………………………………………………
Background and Need ……………………………………………………………………………….
Moderator Variables………………………………………………………………………………….
Research Questions……………………………………………………………………………………
Significance of the Study……………………………………………………………………………
Definition of Terms……………………………………………………………………………………
Summary………………………………………………………………………………………………….
II. REVIEW OF THE LITERATURE……………………………………………………………..
Game-Based Learning……………………………………………………………………………….
Moderator Variables………………………………………………………………………………….
Category of games………………………………………………………………………….
Game platform……………………………………………………………………………….
Grade level…………………………………………………………………………………….
Gender…………………………………………………………………………………………..
Frequency of game-based learning activity………………………………………..
Summary………………………………………………………………………………………………….
III. THE METHODOLOGY…………………………………………………………………………….
Restatement of the Problem………………………………………………………………………..
Research Design………………………………………………………………………………………..
Search Strategies……………………………………………………………………………………….
Data Sources…………………………………………………………………………………………….
Inclusion and Exclusion Criteria………………………………………………………………….
Inclusion Criteria……………………………………………………………………………
TABLE OF CONTENTS CONTINUED
CHAPTER Page
Exclusion Criteria…………………………………………………………………………..
Publication Bias………………………………………………………………………………………..
Funnel Plots……………………………………………………………………………………………..
Fail-safe N……………………………………………………………………………………………….
Instrumentation Reliability…………………………………………………………………………
Coding …………………………………………………………………………………………………….
General information……………………………………………………………………….
Setting and participation characteristics…………………………………………..
Game-based learning characteristics……………………………………………….
Program or course characteristics…………………………………………………..
Instructor characteristics……………………………………………………………….
Research design characteristics………………………………………………………
Moderator variables found……………………………………………………………..
Effect sizes and statistical information reported………………………………..
Study quality………………………………………………………………………………….
Research Questions…………………………………………………………………………………..
Data Analysis…………………………………………………………………………………………..
IV THE RESULTS………………………………………………………………………………………..
Results ……………………………………………………………………………………………………..
Overview of Results…………………………………………………………………………………..
Research Questions……………………………………………………………………………………
Research Question 1……………………………………………………………………….
Moderator Variable Results………………………………………………………………………..
Research Question 2……………………………………………………………………….
Research Question 3……………………………………………………………………….
Research Question 4……………………………………………………………………….
Research Question 5……………………………………………………………………….
Research Question 6……………………………………………………………………….
Research Question 7……………………………………………………………………….
Additional Analysis…………………………………………………………………………………..
Summary………………………………………………………………………………………………….
V. DISCUSSION, IMPLICATIONS, AND RECOMMENDATIONS…………………
Summary of the Study……………………………………………………………………………….
Summary of the Results……………………………………………………………………………..
Research Questions……………………………………………………………………………………
Research Question 1……………………………………………………………………….
Research Question 2……………………………………………………………………….
Research Question 3……………………………………………………………………….
Research Question 4……………………………………………………………………….
Research Question 5……………………………………………………………………….
TABLE OF CONTENTS CONTINUED
CHAPTER Page
Research Question 6……………………………………………………………………….
Research Question 7……………………………………………………………………….
Limitations……………………………………………………………………………………………….
Discussion of the Results……………………………………………………………………………
Implications for Practice…………………………………………………………………………….
Recommendations for Future Research………………………………………………………..
Conclusion……………………………………………………………………………………………….
REFERENCES……………………………………………………………………………………………………
APPENDIXES…………………………………………………………………………………………………….
APPENDIX A: Coding Sheet……………………………………………………………………..
LIST OF FIGURES
FIGURE Page
1. Flow diagram of the search, screening, and selection process………………………
2. Funnel Plot for the standard error on the average effect size………………………..
LIST OF TABLES
TABLE Page
1. Summary of Meta-Analysis in Game-Based Learning……………………………………..
2. Average Effect Sizes for the Four Moderator Variables from the Meta-Analyses Reported in Table 1…………………………………………………………………………………….
3. Descriptive Information of the 17 Studies Included in the Meta-Analysis………….
4. Overall Effect Sizes of the 17 Studies Included in the Meta-Analysis……………….
5. Results of Effect-Size Comparison for Gender………………………………………………
6. Results of Effect-Size Comparison for Grade Level……………………………………….
7. Results of Effect-Size Comparison for Game Platform …………………………………..
8. Results of Effect-Size Comparison for Game Category ………………………………….
9. Results of Effect-Size Comparison for Mathematics Domain ………………………….
10. Results of Effect-Size Comparison for Frequency ………………………………………..
CHAPTER I
INTRODUCTION OF PROBLEM
One intricacy of education is trying to identify where students are struggling and finding creative and repeatable solutions to alleviate those struggles. With regard to subject domains, Chen et al. (2022) stated that “although mathematics is an important part of learning, many researchers report that mathematics is not easy to learn” (p. 466). There is evidence that students in the United States are not meeting the expected standards for mathematical education as reflected in the standardized testing results reported in the National Assessment of Educational Progress (NAEP) survey. The United States is ranked a low-achieving country for the instruction of mathematics (Schwartz, 2023). One of the critiques stated that with regard to mathematics instruction was that “doing the same old, same old” is not going to work. (p. 2). When trying to find a way to engage students and to create curriculum that works for each student is a large challenge, everyone deserves a chance to try something new.
According to the Department of Education’s (2020) data reporting on the field of studies that high-school students are interested in 2015, their finding was that only 16% of students expected to join a Science, Technology, Engineering, and Mathematics (STEM)-related job field. Jobs relating to the STEM field in United States are projected to grow faster annually compared with other fields, and there is a growing focus in education on these subjects in the United States (Department of Education, 2020).
In 2019, the Department of Education released a 5-year plan for the improvement of STEM education in the United States by focusing on strengthening digital literacy, increasing equity, and promoting inclusion. One of the key subject components for STEM education is mathematics. The findings in the 2019 survey from the National Assessment of Educational Progress (NAEP) reported on data collected for public and private schools across the nation to evaluate U.S. students’ academic proficiency in grades 4, 8, and 12 across several content domains reflected the need for improvement. The data analysis separated students by the grade level and provided a scaled score. The scores were aggregated, and the interpretation was to determine whether the students are meeting national education standards. There are three possible categories that students grade levels are marked as per each subject: at or above basic (partial mastery of prerequisite knowledge and skills), at or above proficient (demonstrated competency over subject matter), and at advanced (superior performance on subject matter measured). The 2019 data suggested that the highest grade level that met the basic mathematics standards were fourth-grade students (U.S. Department of Education, 2019). This report also stated that the percentage of students who met basic mathematics standards declined with each subsequent grade level tested.
By using these mathematics standards, stakeholders could conclude the level of students’ readiness to move into the STEM job market. Vu and Feinstein (2017) noted in their research that by the time students reach fourth grade, one-third of students have lost interest in STEM classes and half of the student population lost interest by eighth grade. Although the Department of Education (2020) noted that only 7% of all students who are aiming to join a stem-related career are female students, the researchers here found that there was no difference in gender regarding the loss of STEM course interest for the fourth-grade students.
Researchers have identified some reasons why students are struggling with mathematics. According to Chen et al. (2022), students struggling with learning the content because they
• “are prone to experience difficulty and risk of failure,
• [their] negative performance ratings are high,
• and they often experience intense stress and anxiety” (p. 466).
In addition, Cicchino (2015) suggested that the methodology for instruction coupled with the lack of support for students’ higher order thinking skills, also known as 21st century skills contribute to why students struggle in classrooms. 21st century skills refer to developmental skills that are essential for a student’s overall learning (Cicchino, 2015). These skills are critical thinking, collaboration, creativity, and communication, which are the fundamentals to a flipped classroom (Qian & Clark, 2016). Finally, Yang (2015) noted that as students move into the workforce, there is a need to “develop professional competence as individuals use their creative thinking, critical thinking, and problem solving, as well as communication skills, information, and communication technology (ICT) skills, and collaboration” (p. 281).
The problems with mathematics education can be summarized as the following: researchers suggest that students often are working with difficult topics, have a lot of pressure to get the content right the first time, and the exercises are often presented in non-engaging ways. Additionally, policy makers, researchers, and educators sought to fill the gap regarding the STEM field for students across the nation to meet the Department of Education’s (2019) goals of “engaging students where disciplines converge and build computational literacy” (p. 3). Wang et al. (2022) suggested that the teaching of STEM fields is a difficult task. To meet the proposed standards, researchers such as Wang et al. (2022) suggested ensuring that lesson content could be adapted to focusing on real-world problems by developing cross-disciplinary content, expanding digital literacy, and applying the usage of game workshops.
Game-Based Learning
Ellison et al. (2016) identify how students use technology as a major part of their lives and how games are a large factor for students’ time spent. The usage of game workshops in school settings can be expanded more broadly to the usage of games as part of the curriculum. Researchers such as Bai et al. (2020) noted that mathematics is a difficult topic to learn due to how complicated the material can be and suggested that the content is often presented in a dull and passive manner. Additionally, findings from Hung et al. (2014) suggest that minimizing students’ mathematical anxiety, as well as working to improve their mathematics self-efficacy, learning motivation, and learning achievement has been identified as a difficult problem. Echeverria et al. (2010) noticed this situation and sought to engage students the same way they are engaged at home: by using games as part of the instructional activities. Policy makers suggest that STEM knowledge such as mathematical concepts needs to become more accessible and engaging for students. A possible method for helping students reach higher levels of proficiency with mathematical concepts would be using games in the classroom.
The field of game-based learning has created a large amount of insight into the question “can we use games in the classroom” and is now starting to focus on other applications for developing this field. There has been an increase in studies regarding the usage of games, both digital and nondigital, across subject domains and grade levels such as researched by Byun and Joung (2018), Clark et al. (2016), Lamb et al. (2018), and Wang et al. (2022). Implementing games in the classroom has led to research examining how the usage of games and the design of games could produce effective learning outcomes across different subject domains (Johnson & Mayer, 2010). For this meta-analysis, digital games are defined as an experience in which the participants (a) strive to achieve a set of goals within the constraints of a set of rules that are enforced by the software, (b) receive feedback toward the completion of these goals (e.g., score, progress, advancement, win condition, narrative resolution), and (c) find some recreational value (Clark et al., 2016, p. 87). Nondigital games are defined as the usage of physical, board, and card games for instructional purposes (Naik, 2014). The study of using games in a classroom setting is classified as game-based learning, which is the process and practice of learning using games from the learner’s point of view. Game-based learning is also known as a design approach that uses intrinsic and extrinsic motivation to engage, challenge, and provide support for students (Plass et al., 2015).
Historically, there has been much interest in using games and simulations in education (Vu & Feinstein, 2017). Psychologists such as Piaget and Vygotsky believed that play is a necessary part of a child’s development and that the nature of play grows with a child as they age (Plass et al., 2015). Although the study of “play” and the concept of using games in combination with learning has been discussed since the 1960s (Vu & Feinstein, 2017) with an increasing interest in applying game design to academic content (Denham et al., 2016) with a focus on both digital (Sung & Hwang, 2013) and nondigital games (Colby, 2017), the field of game-based learning is a new research field. Game-based learning expands across multiple dimensions and has studies that have suggested that using game-based learning in the classroom has complications when reviewing results. Researchers such as Gee (2005), Plass et al. (2015), and von Gillern and Alaswad (2016) have all commented that games in the classroom are powerful tools that have the possibility to improve students’ achievement, whereas findings from Young et al. (2012) regarding the usage of game-based learning suggest that more research needs to be done to ensure that game-based learning has the desired effect for mathematics education. In addition, there has been controversy regarding what is considered really to be a game in a game based learning environment rather than just applying game elements to instruction (Plass et al., 2015). Although both gamification and game-based learning rely upon motivating the learner by using games or game elements, there is a distinction between using game-like elements in the classroom, called gamification, and game-based learning that requires a game, either digital or nondigital, to be a main instrument in the learning content.
Although there has been a vast amount of research produced about the usage of computer games in education, the research into proper usage of game design that would produce an effective learning result is still relatively in its infancy (Johnson & Mayer, 2010). Since the field of game-based learning is constantly shifting definitions and research goals, this meta-analysis is a contribution regarding recent findings.
Several researchers have attempted to engage students the same way they are engaged at home by using games as instructional tools (Echeverria et al. 2011). There are a series of divides on how to study game-based learning, complications arise due to the trying to understand and translate the various definitions used regarding game-based learning and the overall lacking framework for designing curriculum with games in mind as a major part of classroom instruction. One of the research questions in Wang et al. (2022) study was “are the learning gains higher when using the digital game to support STEM education as compared to non-digital game-based methods?” Wang et al. (2022) findings were that game-based learning instruction compared to traditional instruction resulted in a 0.56 effect size. This finding supports the idea that game-based learning is in some way more effective than traditional instruction methodologies.
Although the findings are supportive for the usage of game-based learning compared to traditional instruction, Wang et al. (2022) stated that the finding should not overlook the how learning goals and methods of content integration are used. When designing curriculum content with game-based learning in mind, Ciccinho (2015) suggested that there must be a series of principles that would help to ensure that the game is used to properly support the learning process. These principles include inspiring critical thinking, providing enough challenge, providing opportunities to discover, having a social component, and providing a winnable or correct series of answers to the problems provided. Plass et al. (2015) suggested that game-based learning relies upon motivation, player engagement, adaptivity, and repetition. These aspects are important factors when trying to create lesson-plan content from which students will learn. Furthermore, Plass et al. (2015) outlined a three-part cycle that can be used to explain the process of game-based learning. This three-part cycle consists of the challenge, the response, and then the feedback. This three-part cycle can be found in traditional instruction just as it is found in experimental studies that use game-based learning. Traditionally the challenge would be an assignment or a question, the response would be created by the student, and the feedback would be either an instant or delayed response. The feedback process may or may not continue. In the case of game-based learning, up to all three parts can be adapted to one game completely or by using game elements in the classroom as part of either the lesson, the assessment, or classroom environment. Game-based learning relies on the process of experimentation and discovery from both the instructors and the learners, and the usage of the game activity has a near infinite repetition so that the learner can learn at their pace (Echeverria et al. 2011).
An example of using game-based learning as part of a lesson could be the following. The challenge could be a puzzle, quest, or problem in a game that the student must solve. The feedback is either prompting the player to try again or to allow them to progress to the next stage, level, or challenge. This feedback is instantaneous and depending on the game design could be adapted to provide hints or assistance to the student as they try to solve the challenge. To expand on the design of using a game activity in the classroom, the instructor would ideally implement lectures, instructions, or a series of debriefs either before, during, or after game play has occurred. The game activity could be used as a form of informal or formal assessment to identify if the student has acquired the understanding from the game activity.
Game-based learning is the term for using games in the classroom either in parts or as a whole activity, and these games can be digital games played on computers, digital mobile games on phone and tablets, or nondigital games, which would be games that are physical and use a game-board or cards and do not use any digital equipment.
Researchers have encountered problems attempting to implement both types of games, digital and nondigital, as part of instruction (Susaeta et al., 2010). Several factors that affect the success rate of games include the instructional design of the learning environment, realistic expectations of how the results will be measured (Akcaoglu, 2016), and the time needed to use the games correctly (Craft, 2016). Games being adaptable for the lesson content is crucial to an effective learning outcome. Adaptability or customization is a big factor when reviewing the usage of game-based learning. By creating a learning environment where games are integrated into both the instruction and the assessment processes, educators allow for the personalization of content, promote engagement for the task at hand, and foster knowledge development (Nebel et al., 2016). As researchers present various topics for studying game-based learning that include method, instructional structure, game type, platform, and amount of time spent playing games in the classes, the researchers make note of how different game-based learning classrooms can be from one another.
Summary of Selected Meta-Analyses in Game-Based Learning
Prior meta-analyses by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022) each focused on different aspects of game-based learning. The moderator variables studies and the average effect sizes found suggested a moderate statistical significance when using games in the classroom as presented in Table 1.
Table 1
Summary of Meta-Analysis in Game-Based Learning
The following summary regarding how Clark et al. (2016), Lamb et al. (2018), Tokac et
al. (2019), and Wang et al. (2022) focused on comparisons in game-based learning expands upon the purpose of each of these meta-analyses and their involvement in this future study.
Lamb et al. (2018) defined three different categories of games that are used by this meta analysis. First, there are serious educational games (SEG), which are a specific form of video game played within a virtual immersive three-dimensional environment used for educational purposes that includes a directed and a pedagogical approach (Lamb et al., 2018, p. 159). Games in this category will have a direct lesson plan and an instructional model in mind. Second, there are serious games (SG), which are games designed to train a broad series of tasks using real-life examples (Lamb et al., 2018, p 158). Third, educational simulations (ES) are a group of technologies supporting highly engaging, often two-dimensional, interactive virtual environments between limited variables (Lamb et al., 2018, p 160) Lamb et al. (2018) focused on digital games in the classroom and reported on a collection of moderator variables including grade level, types of games, and the main effect size comparison was the game type used. Finally, Lamb et al. (2018) compared each game category with another and with traditional instruction and the findings suggested that all three game categories had positive results when compared to traditional instruction. These findings are as noted in Table 2.
The moderator variables from Wang et al. (2022) are the some of the main variables of this meta-analysis. As Wang et al. sought comparisons regarding the traditional instruction compared with game-based learning as a multimedia comparison, Wang et al. (2022) provided the following moderator variables: subject domains, grade levels, using nondigital and digital games, computers and mobile game platforms, and frequency of game-based learning instruction. Wang et al. (2022) found that when comparing digital games with other instructional methods resulted in an effect size of 0.67. This finding suggested that students performed well with game-based learning activities over other instructional methods.
Additionally, Clark et al. (2016) focused on digital games to nondigital game condition comparison, Lamb et al. (2018) focused on the types of games that could be used and compared the learning outcomes found against each other, Tokac et al. (2019) focused on media comparison between games and nongame conditions and comparing between different types of digital games, and Wang (2022) focused on how digital games were effecting achievement in multiple subjects. The studies by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022) provided a framework for this dissertation. This study combines moderator variables from the four studies detailed in Table 2 to fill the gap in research.
Table 2
Average Effect Sizes for the Four Moderator Variables from the Meta-Analyses
Reported in Table 1
Note: k is the number of studies
Limitations and critiques from Young et al. (2012) and Lamb et al. (2018) have found that one challenge in defining game-based learning is translating and standardizing the various
definitions used in the subject matter that was reflected in the studies found in Table 1. The terminology of “game” and game-based learning also contributes to a limitation as there are a series of often interchangeable terms used in this field to describe the type of games used for instruction and can refer to either digital or nondigital usages. For example, Lamb et al. (2017), who referred to digital games, listed three main terms for using games: serious games, serious educational games, and simulations. According to the researchers, a serious game is a game designed to train a broad series of tasks using real life examples, whereas a serious educational game includes a directed and a pedagogical approach (Lamb et al., 2018, p. 159). Furthermore, simulations are usually activities that allow for near infinite practice of a task. These activities provide the user the opportunity to interact at almost any scale or environment regardless of the feasibility in the real world (Lamb et al., 2018, p. 160). Second, the platform for where the game tasks place, such as nondigital games compared to digital games, and digital games can further break into categories including computers and mobile devices also needs to be considered. A meta-analysis by Gao et al. (2020) found that 30 studies published between 2010 and 2019 on mobile games in STEM education called for more research to apply to mobile games.
The final variable regarding game-based learning is determining if gender has any effect on the game-based learning outcomes as research has suggested conflicting findings. Literature collected and reported by Gao et al. (2020) indicated that male and female students enjoyed games for different reasons and identified that female students outperformed male students on game-achievement measurements. This finding is collaborated by Cinnicho (2015) who noted that female students outperformed male students during the posttest question after a game-based learning activity. Although there are findings from Gao et al. (2020) that there is no statistical difference between gender regarding game-based learning activities and from Cinnicho (2015) who reported no difference in the delayed posttest scores regarding gender.
According to the Department of Education’s report (2019), although both male and female students’ average score in mathematics decreases over time, the number of female students who fall below the average is higher than the male students. In order to expand on this variable, further research is needed to identify which domains of mathematics have a large gender disparity and if game-based learning can be an effective way to increase student’s mathematical understanding and academic process.
Although researchers such as Gee (2005), Plass et al. (2015), and von Gillern and Alaswad (2016) have commented that games in the classroom are powerful tools that have the possibility to improve students’ achievement, the findings from Young et al. (2012) suggested that there is a lack of strong evidence that supports association between game-based learning and students’ academic achievement. Young et al. (2012) also suggested that game-based learning is effected due to the lack of clear consensus on the efficiency of game-based learning.
Purpose Statement
The purpose of this meta-analysis was to investigate the effectiveness of the usage of games as part of mathematics instruction on academic achievement in grades Kindergarten to 12. The main goal of this meta-analysis was to obtain the average effect size and if the effect sizes were found not to be homogeneous, then to compare effect sizes for differences between gender, grade level, and game type, with students’ academic achievement scores. This meta-analysis fills the gap in the knowledge by examining courses that are using game-based learning across three platforms of instruction: nondigital, digital on computers, and mobile devices.
Theoretical Framework
For this meta-analysis there are two main theories to consider: social constructivism and the cognitive theory of multimedia learning. Vygotsky’s (1980) theory of social constructivism states that constructivism that stated that knowledge is socially situated and jointly created through collaborative endeavor and suggested that learning new concepts was just outside the learner’s initial grasp but could be attainable with guidance or lesson content. Akcaoglu (2016) suggested game-based learning is that students are learning by design, which is a constructivist approach that views knowing as being situated in action and codetermined by the individual’s environment. Students have opportunities to construct their learning by playing games in a learning environment with a series of in-game feedback, tasks that scale with their expected learning goals by completing increasingly challenging activities accompanied with instructionally designed scaffolding such as lectures, group discussions, individual check-ins, or feedback sessions. Students constructing their own learning in collaboration with others is a tenet of learning theory. In this case of game-based learning, a game can be considered the social peer for the learning activity.
Games can use the concept of Vygotsky’s Zone of Proximal Development to help students achieve a learning goal. According to the concept of zone of proximal development (ZPD), a learner’s cognitive development was divided into two levels: the actual level at which a learner can complete a task independently and the potential level at which the learner’s completion of a task can be achieved with the guidance and help from knowledgeable adults or peers. The gap between the two levels was called ZPD (Vygotsky, 1980). Additionally, this entire process can continue to be repeated with the possibility of increased difficulty similar to how educators focus on creating classroom content that is just at the edge of a student’s abilities, something that the students can achieve in the gameplay activity.
Game-based learning is supported by two theories of how knowledge is developed. Johnson and Mayer (2008) proposed the cognitive theory of multimedia learning, in which a digital game or digital content can help streamline information transfer. To build on this idea, Sweller (2011) proposed that the cognitive learning theory supports the usage of game-based learning courses because the students can focus on both knowledge development and knowledge transfer. If the game requires too much cognitive load, the idea of too much information to process at one time or requires more time to become the standard for the student to process, it is possible that the game becomes a distraction or a burden for the user. Therefore, there is a necessity of using or creating “good” games. To help identify if a study has included an appropriate, or “good”, game for game-based learning, Gee (2005) suggested that there are three core ideas for designing game-based learning environments. The first idea is to provide learners the space to learn, fail safely, and to collaborate with peers, the second is to plan for problem solving that is within the students’ abilities, and the third core idea to ensure there is a series of dialogue or feedback between the learner and the assessor or assessment tools.
Using game-based learning in the classroom is built upon the idea that by using a tool such as a game and a platform, whether a computer or a cellphone to play the game, and the game designed in line with the instructional goal; students would be more likely to gain and retain information and then access their higher order thinking skills. (Chen et al., 2021).
As mentioned earlier in the 2019 Reporting Data from the NAEP, students find it hard to learn mathematics as they progress through grade levels and need methods to develop their mathematics content. Learning theories that address the needs of students building their own knowledge, being engaged with the content, and finding easier ways to conduct knowledge transfer all are found in game-based learning. In the following section, the background and need for more investigation in game-based learning studies is presented and provides context to studies that have focused on game-based learning and the suggestions for further research, which this meta-analysis fills.
Background and Need
The 2019 NAEP reporting data on average mathematics achievement scores of students in the United States supports that after fourth grade, the average number of students who have the basic standard for understanding mathematics concepts will continue to decrease, as measured by standardized testing scores, as they advance in grade level. To address this problem, the usage of game workshops for this meta-analysis was interpreted as using game-based learning as part of the curriculum. Games and by extension game-based learning has been found to help students develop their digital literacy, understanding and engagement with STEM courses (U.S. Department of Education, 2019). Researchers, such as Gee (2005), Plass et al. (2015), and von Gillern and Alaswad (2016), have commented that games in the classroom are powerful tools that have the possibility to improve students’ achievement. Although several of the studies reported post higher-education applications, the need for find more acceptable data regarding primary- and middle-school students to support the NAEP’s call for developing student’s mathematics scores middle school.
Previous meta-analysis such as Young et al. (2012) and Wang et al. (2022) have called for a need for research finding and interpreting more data from quantitative sources regarding game-based learning in mathematics education. This meta-analysis fills this gap of knowledge. The need for this meta-analysis comes from the suggestions of future research from meta analyses conducted by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022),
The meta-analyses by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022) sought to provide more quantitative data regarding the usage of games in the classroom, with all but Clark et al. investigating grade level comparisons to identify any trends with student achievement scores when using game-based learning as part of the instructional model. based learning. Wang et al. (2022) was also investigating if subject matter and grade level contribute to the effectiveness of game-based learning. Wang et al. (2022) investigated some of the inconsistencies regarding how mathematics is taught with digital game-based learning that also suggests that a possible need for further research would be to find the effect sizes on one set of student-grade levels that participate in game-based learning.
Although Byun and Joung (2018) proposed for future research that more empirical studies using game-based learning for the mathematical domains of number and operations, algebra, geometry, measurement, statistics, and probability be investigated; the recommendations for future research from the meta-analysts reported in Table 1 are the following. Clark et al. (2016) suggested an investigation into studies regarding how assessment is considered and conducted with a game-based learning course. Clark et al. (2016), Lamb et al. (2018), and Tokac et al. (2019) recommend investigating length of duration using game activities in future studies. Wang (2022) suggested examining studies that stated the usage of a mobile platform for game based learning.
This meta-analysis filled the gap in the knowledge by examining courses that are using game-based learning across three platforms of instruction, reviewing any effect sizes reported and identifying which of the mathematical domains are focused upon in game-based learning courses. The above studies and observations provide context for the background and need of this meta-analysis. Although using games in the classroom is not a new idea, prior meta-analyses and systematic reviews have been critical about the creation of game-based learning content, the selection of games to use and the tools used to implement and observe learning content when games are employed has been an ongoing development. Although Nadolny et al. (2017) stated that compared with other methodological designs such as project-based learning or problem based learning, game-based learning is more fluid and open for interpretation and design in addition to being able to maintain student interest, a limitation was with game-based learning has been found to lack “consistent and systematic processes” (p. 815).
Vu and Feinstein (2017) conducted a multiyear, multigrade level case study for 101 students by looking at pre and post-test scores. Due to the inconsistent nature of where the data came from, there was no t-test conducted. Each of these students had participated in a game based learning activity at some point during the case study. Although the researchers sought to answer three main research questions, the main question that connects to this meta-analysis was “does game-based learning activities have any positive impact on students’ STEM learning performances?” (p. 585). Vu and Feinstein (2017) stated that from their research findings that there does seem to be some growth for academic achievement for students who participated in game-based learning activities, the authors are concerned that this is more attributed to “novelty” of playing games rather than learning (587.)
Moderator Variables
The moderator variables for this meta-analysis regarding game-based learning in mathematics courses are grade levels ( including comparing primary-, middle- and high-school students), game type (refers to games that are either nondigital or digital via computer or digital via mobile device), subject domain (refers to reporting the type of mathematics content being taught in the study and these domains include number and operations, algebra, geometry, measurement, statistics, and probability), frequency of game-based activity (refers to the amount of time playing games in the classroom), and gender.
Research Questions
This meta-analysis addressed the following research questions:
1. What is the average effect size of using game-based learning on academic achievement in
4 th- to -12 th- grade mathematics classes?
2. Does gender differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
3. Do the grade levels differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
4. Does the game platform differ in their average effect size of mathematics focused game based learning courses on academic achievement as a moderator variable?
5. Do the game types differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
6. Do subject subdomains differ in their average effect size of mathematics focused game based learning courses on academic achievement?
7. Does the frequency of game-based learning activity differ in their average effect size of mathematics focused game-based learning courses on academic achievement?
Significance of the Study
With the increasing need for students to join the STEM field and the rate of which students fall behind in mathematics as they progress through grade levels (Department of Education, 2015), this study seeks to provide insight from trends using game-based learning to improve students’ academic achievement. By conducting this research, this study analyzed and synthesized prior findings, reported effective sizes for comparison, and provided explanations for what the effect sizes suggested and the trends regarding how teachers and researchers are using games in the classrooms. The collection of studies reviewed are from 2010-2023, regarding the usage of game-based learning to investigate which game platforms and game types are being used and reported any comparisons regarding student characteristics in mathematics curriculum that include grade level, game type, and gender. This research expanded upon the work conducted by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022).
The contribution to the game-based learning field from this meta-analysis provides support for teachers, researchers, game developers, and students alike. There is a distinction between using game-like elements in the classroom, called gamification, and game-based learning that requires a game, either digital or nondigital, to be a main instrument in the learning content. Although researchers seek to study using games for learning, the vast differences of items considered as part of the game-based learning language, which includes different game types and instructional design choices, could lead to misunderstanding and complication for studies stating that game-based learning was used in a classroom. Teachers could benefit from the insight of what games are being used effectively and which mathematics subdomains use games. Researchers could benefit from the additional quantitative data that are analyzed in this study. Game developers could benefit from learning more about which platforms students are more likely to be engaged in and which mathematics subdomains would benefit from more games. Finally, students could benefit from this study as the information presented here provides help to alleviate the challenges to learning.
Definition of Terms
When reviewing studies Lamb et al. (2018) noted that there is a complication of terms that are used interchangeably when referring to the study of games in the classroom. To ensure clarity and specificity, the following list of terms and their definitions are referred to in this meta-analysis.
Academic achievement represents performance outcomes that indicate the extent to which a person has accomplished specific goals that were the focus of activities in instructional environments (Steinmayr et al., 2014, p.1). For this meta-analysis, academic achievement is measured by reported test score, grade point average (GPA), or other indicators of assessment completion.
Digital games are a digital experience in which the participants (a) strive to achieve a set of goals within the constraints of a set of rules that are enforced by the software, (b) receive feedback toward the completion of these goals (e.g., score, progress, advancement, win condition, narrative resolution), and (c) are intended to find some recreational value (Clark et al., 2016, p. 87)
Game-based learning is any instructional activity based on the use of games (Talan, 2019, p. 476).
Gamification occurs when there is a usage of game mechanics, but the foundational structure of a learning experience does not change (Nadolny et al., 2017, p. 816).
Mobile games for this study refer to games played via using a mobile device (Wang et al., 2022, p. 2). These games can range in various designs from anything simple like an interactive math worksheet to a completely complex video game. For this study, mobile devices include cell phones, laptops, e-readers, and digital touchpad.
Nondigital games are the usage of physical, board, and card games for instructional purposes (Naik, 2014).
Serious educational games are a specific form of video game played within a virtual immersive three-dimensional environment used for educational purposes that includes a directed and a pedagogical approach (Lamb et al., 2018, p. 159).
Serious Games are games designed to train a broad series of tasks using real life examples (Lamb et al. 2018, p. 158).
Simulations are usually activities that allow for near infinite practice of a task. These activities provide the user the opportunity to interact at almost any scale or environment regardless of the feasibility in the real world (Lamb et al., 2018, p. 160)
Summary
According to the 2019 reporting data from the NAEP, mathematics education in the United States is a difficult subject. Game-based learning could improve math instruction needed for STEM careers. To meet this need, the usage of game-based learning has been suggested by educational practices, but more research is needed to establish effectiveness of using game-based learning to meet the student’s mathematical needs. A game-based learning activity is the usage of any game as part of the learning process. In addition, game-based learning uses intrinsic and extrinsic motivation to engage, challenge, and provide support for students (Plass et al., 2015) but has some complicated results in prior meta-analyses. A goal of this meta-analysis is to review more empirical studies that have quantitative data to identify trends regarding the design, implementation, or assessment of game-based learning.
The following chapter II is a review of the literature that has shaped the need for more studies regarding game-based learning and includes details regarding moderator variables.
Afterward, chapter III contains details of the method for which this meta-analysis was conducted. Results of the meta-analysis and moderator variables are found in chapter IV. The dissertation ends with summary, limitations, discussion, recommendations, and suggestions for future research in chapter V.
CHAPTER II
REVIEW OF THE LITERATURE
The purpose of this meta-analysis was to investigate the effectiveness of the usage of games as part of mathematics instruction on academic achievement in grades Kindergarten to 12. The main goal of this meta-analysis was to obtain the average effect size of mathematic achievement scores after the usage of game-based learning activity and if the effect sizes were found not to be homogeneous, then to compare effect sizes for differences between gender, grade level, and game type, with students’ academic achievement scores. This meta-analysis fills the gap in the knowledge by examining courses that are using game-based learning across three platforms of instruction: digital on computers, digital games played on mobile devices and games played without the inclusion of technology, which for this meta-analysis is classified as nondigital games.
Conventional instructional methodologies for teaching mathematics to 21 century students have not been effective for preparing students for STEM careers. (Annetta et al., 2006). To improve this situation, researchers have attempted to engage students the same way they are engaged at home by using games as instructional tools (Echeverria et al., 2011). This chapter contains a review of the literature first by expanding on the background and complications of game-based learning and then reviewing studies that include effect size comparisons between gender, grade level, game type, and the frequency of game-based learning activities, with students’ academic achievement scores. To fill this gap in knowledge, this study identified, summarized, and synthesized research that has used game-based learning for mathematics education.
Game-Based Learning
Game-based learning, for this meta-analysis, is the usage of any instructional activity that includes the use of games (Talan, 2019). The usage of game-based learning games in the classroom has had partial success (Echeverría et al., 2011). A common critique of using game based learning is the usage of simple educational computer games that lack both the engagement factor for students and a level of quality equal compared with the commercial video game market (Echeverría et al., 2011). Several studies have concluded that it is possible to implement a curriculum based on games (Hanghoj and Hautopp, 2016) if the educators have clear learning goals for using games in a classroom (Craft, 2016).
According to the study conducted by Hanghoj and Hautopp (2016), there are three instructional ways that teachers can present games as a lesson. The first way is the executive approach, where the instructor is most familiar with nondigital sources such as textbooks and prior lesson plan content. The instructor would be showing the video game just for the visual effects. Possible complications could arise as the instructor may lack contextual language of how to use the game only at a surface level. An example of this would be planning to buy and use a game in the classroom but not sure how the progression system works, which can result in creating a divide between the class and the learning goal. The second way is that of an improvisational approach, which relies on the concept of a flipped classroom where the students are the experts teaching the educators. This method relies upon the idea that the students have some familiarity with the game being used. Educators here can be familiar with the game and role play what they need the student to explain in more detail. The third methodology is called transformational approach, where educators “go all-in” on learning the context of the game and become highly familiar with the functionality of the game. Educators here may start to adapt and redesign whichever game they are using to fulfill their needs.
Research by Qian and Clark (2016) indicate an increasing interest in how games may have influenced learning and need for measuring the effectiveness of game-based learning in disciplines connected to STEM. According to Qian and Clark, (2016), there is a lack of clear consensus on the positive effect of game-based learning. Some results suggest that game-based learning is more effective compared with traditional classroom instruction as it could increase students’ motivation for learning and provide them with opportunities to explore and acquire new knowledge and skills whereas other research gathered did not find strong evidence that supports association between game-based learning and students’ high academic achievement or psychological development.
The following section relates several game-based learning studies to the dependent variables of mathematics achievement scores. Afterward, the additional sections reviewed the moderator variables.
Moderator Variables
Card (2012, p. 30) stated that the goal of a literature review is to gather a good number of moderators to help to guide the research topic. To add in this search, this meta-analysis has included the following moderator variables. The first moderator variable is the category of games, which refers to one of three main game types, serious games, serious educational games, and simulations. The second moderator of the game platform refers to how the game is accessed or played. These results can range from digital games that are accessed on computers or laptops to mobile games that are based on cell phones or e-reader to nondigital games that are pen-and paper based. The third moderator variable is grade level comparisons for students in grades kindergarten to high school. The fourth moderator variable is frequency of game-based activity, as to discover how often activities need to be used in a curriculum and with the correct level of regularity to be effective. The fifth and final moderator variable is gender, as the comparisons found here should provide some context to answer the call for increasing female students’ overall learning in mathematics and improvement in the retention of female students in the STEM fields.
Category of Game
The category of game is a moderator variable for this meta-analysis as it relates to the three different types of games that are used in game-based learning environments: these game types are “serious games, serious educational games, and simulations.” Additionally, each of these game types can be implemented either digitally on a computer or mobile device or as a nondigital activity. The definitions for this meta-analysis come from Lamb (2017, p. 1) and are summarized as serious games (SG) that are used to help individuals learn a series of tasks, and as serious educational games (SEG) that add a connection to a lesson plan and are used primarily by students, whereas simulations are environments that attempt to replicated real-world scenarios in which the user is allowed near-infinite attempts to practice a task. Additionally, this meta-analysis recognizes that the terms listed above can often be found interchangeably.
Lamb’s (2017) meta-analysis was based on 46 peer-reviewed studies from 2002 to 2015 that included students in primary to postsecondary schools. Each of the studies compared game based learning instruction that used either SEG, SGs, or simulations with traditional instructional methods. Lamb investigated if there was any increase in affective, cognitive, or achievement outcomes from students based on the type of instruction they received. The duration of the game based learning activity could have taken place between one lesson to a series of lessons over a full school year. The purpose of Lamb’s research was to characterize and compare outcomes related to these three categories of games (p. 159). The studies included for Lamb’s meta analysis had to have both an experiment group and a comparison group. Each of the included studies had reported effect sizes. According to the findings, the three game categories of serious games (0.79), serious educational games (0.50), and educational simulations (0.63) supported the suggestion that game-based learning approaches promote learning and are useful in the classroom. Lamb noted as a limitation, the constant usage of game-based learning activities was often used in a “short term and non-persistent” way (p. 166). There is a possibility that lack of regularity or over regularity of using game-based learning could remove the potential for academic impact.
Game Platform
As game-based learning has a range of how the game could be played there should be comparison between the game platforms. Digital games are games that can be played on one of several different platforms. The main considered platform is that games are played on either a computer, or other stationary devices such as an Xbox or PlayStation and mobile digital games are often played on phones or e-readers. Finally, nondigital games such as board games or tabletop games do not use a digital platform. Wang et al. (2022) investigated the effects of digital game-based learning activities on students in traditional learning environments. The author compared digital games to nondigital games activities to what would be considered traditional instruction of lectures, textbooks, and laboratories. In this meta-analysis, Wang et al. reviewed 33 studies from 2010-2022. These studies focused on STEM courses for students in K-12 in the United States. The gameplay activity could have lasted between a one-time activity to repeated activities over the course of several months.
Although the usage of digital games was statistically higher than the traditional instruction methods, there was no statistically difference between digital games completed on computers compared to digital games using mobile devices in addition to computers being reported as the most common device. The authors suggested that as both the growth of mobile games and further reliance on mobile devices such as phones, there is a need for future investigation of this comparison.
Regarding nondigital games comparison with traditional instruction, Cicchino (2015) conducted a study in a U.S. Social Studies class that investigated the effects of a game-based learning environment. This study consisted of 62 eighth-grade students from three different social studies classes from a middle-class New Jersey school. There were five social-studies classes, three with this game activity, and two of the “traditional” courses were taught by the participating teacher. This participating teacher had experience using game-based learning activity before. To compare the game activity, there were 115 students who received traditional instruction during this study. Game play was recorded for 35 minutes-per-day for 4 days. Students were given a posttest consisting of a five-question quiz a month upon completion of the game activity and delayed posttest of the same exact quiz was presented again another 6 months after the study was concluded.
According to the results, there as an increase to the mean (M=3.66) for those with the game-based game activity on their posttest compared with the traditional students mean of (M=3.60) and on the delayed posttest the game activity group had a mean of (M=2.11) compared with the traditionally students delayed posttest mean of (M=2.00) (p. 13-14). These findings suggest that the game-based activity and the traditional instructional method had a lack of statistically significant difference. Cicchino noted the limitations of time requirements, planning, resources, and training as all factors to consider when implementing game-based learning approaches instead of a traditional teaching approach.
Hung et al. (2014) insisted that the game-based learning approach is proposed by integrating mathematics content into computer games on mobile devices in elementary school mathematics courses as the students were found to have a significantly higher self-efficacy when using a mobile device for game-based learning (p.162).
Abdul Jabbar and Felicia (2015) stated information regarding the game platform in which 42 students used desktop computers compared to the 10 that used some sort of mobile device such as a cell phone. An observation from this study was that although mobile devices are being used, studies still may overly rely upon the usage of desktop computers for game-based learning activities. Gao et al. (2020) commented that 3 of the 27 studies included in their review suggested that game-based learning via mobile games resulted in higher academic achievements compared with traditional instruction.
The research focus of most studies included in the systematic review by Gao et al. (2020) investigated the effectiveness of using mobile games for learning. This systemic review found that student learning performance was measured mainly by assessment tests and that the studies “consistently showed positive effects of the mobile game-based approach on student learning performance” (p. 1803).
The rationale from these studies is to identify studies that have investigated the usage of different methods for providing game-based learning activities.
Grade Level
The grade level of students, being primary-, middle-, and high-school students, is necessary for this meta-analysis as the student interest level in STEM may reduce as they process through the grades. According to NAEP, one third of students will lose interest in STEM classes during the primary grades and half of students will not seek to take STEM classes when entering high school (U.S. Department of Education, 2019). The consistent usage of games and game based learning in primary-, middle-, and high-school classrooms may result in more students developing their mathematics understanding and improve the retention rate of students who stay in the STEM-related fields.
Tsai and Tsai (2020) conducted a meta-analysis regarding game-based science learning against other instructional methods from 2010 to 2018 and was inspired by the suggestion from Young et al. (2012) for more empirical research regarding the usage of game-based learning in primary-, middle-, school science classes. Tsai and Tsai sought to examine quantitative data from studies that focused on students’ ability to problem solve and students’ processing skills. Additionally, these researchers compared a game activity group to another group of students that received game-mechanics as part of the curriculum, also known as gamification. The game- activity group received gameplay as part of the lesson plan as compared with the game mechanics, or gamification of the curriculum, where game elements are applied to the classroom and the students are not necessarily playing a game. An example of gamification is a badge collection system in which students’ complete tasks and achievements in traditional methods for a reward.
This meta-analysis consisted of 26 studies, 14 were in the gameplay design focused group and 12 were in the game-mechanics design group. The researchers’ data for the gameplay design group found a medium effect size for using gameplay design in the classroom, specifically 0.68 for primary students with a confidence interval of [0.32–1.04] and 0.51 with a confidence interval of [0.10-0.92] for middle students (pp. 289). Additionally, these authors stated that students improved across all grade levels but the comparisons between each of these groups did not result in any statistically significant differences.
Abdul Jabbar and Felicia (2015) conducted a systematic review of 91 studies that focused on game design features that promote engagement and learning in game-based learning (GBL) settings from 2003 to 2012. The student populations that were measured for this study were in primary, middle, and high school. Although the meta-analysts did not present effect-size data for comparison, they did provide information regarding comparisons between different grade levels and various game platforms. They found that a large portion of studies were focused on students between ages 8 and 14, which includes primary and middle school and noted that research regarding these grade levels was increasing in publication globally.
Chen et al. (2022) created an integrated bibliometric analysis and systematic review from 1990 till 2020 in science and mathematics education. The meta-analysts reviewed 146 articles that included the moderator variables of game platform and grade levels. They found that game based learning research in science and mathematics focused on elementary education (40.3%), and secondary education (38.2%) and suggested that the focus for study for game-based learning was younger students (pp. 468).
According to the Wang et al.’s (2022) findings, there was no reported difference in the usage of games to each of the STEM subjects of Science, Technology, Engineering, and Mathematics, as the authors found that students in the primary-grade levels benefited the most from game-based learning activities. Clark et al. (2016) stated that the research focused on K-16 grade, the exact number of studies included per moderator variable was not included. Instead, Clark et al., reported the average age of a student out of the 57 studies reviewed to be 12.05 and 13.38, which would make the average student grade level to be in grades six and seven.
Grade level comparison is one of the main goals of this meta-analysis to respond to the needs as indicated by NAEP results and the call for more empirical research from this field.
Gender
In 2015, the Department of Education’s National Center for Science and Engineering Statistic data found that among 15-year-old students, only 7% of all students who are aiming to join a stem related career are female students (U.S. Department of Education, 2020). A possible interpretation of these data that was stated in chapter I is the decrease in mathematics performance for students during their K-12 progression. This trend was also noted in a 2023 report regarding the characteristics of workers in the field of STEM and the trends identified from 2011 to 2022. According to this report, women are underrepresented in STEM workforce, with only 35% working in STEM-related field.
The STEM field has a problem of maintaining more female students’ interest compared with male students’ interest as students progress through grade levels (Zhao et al., 2021). When reviewing any gender comparisons, Abdul Jabbar and Felicia (2015) stated conflicting findings from studies included. Due to the lack of studies that compare the gender of students, only six studies were included in the gender comparisons and the findings suggested that there most likely was no difference, this systematic review reinforces the need for more investigating into this moderator variable. This finding supported the work of Vu and Feinstein (2017), a meta analysis regarding STEM and game-based learning courses that found there was no gender difference for student achievement for primary-grade level students in game-based learning classes. Chen et al., (2022) work on game-based learning studies over the past 30 years and across multiple countries including the United States did not find any gender difference for high school students with respect to game-based learning. This meta-analysis investigated any new trends between game-based learning and gender comparisons with academic achievement for primary-, middle-, and high-school students.
Frequency of game-based learning activity
As presented in chapter I, Nadolny et al. (2017) and Talan et al. (2020) suggested that creating game-based learning environments varies between each classroom and each subject domain and that as a discipline, game-based learning should identify clear instructional learning components with their corresponding game element or game attributes. For this study’s meta analysis, one of the learning components is length of instructional time and frequency of a game based learning activity. The following studies used different measures of time that yielded different results.
According to the Wang et al. (2022) findings, the usage of the digital game-based instruction for less than a week had the largest effect on students’ academic achievement. Wang et al. considered four categories for game activity duration, less than a week (0.95), between a week and a month (0.47), between a month and 3 months (0.58), and 3 months or longer (0.60). One aspect of this information to review is why the “between a week and month” effect size is smaller than the shorter and longer periods of time. While different categories of instructional time yield different results, each were statistically significant.
A study conducted by Echeverria et al. (2011) sought to identify any trends for science learning with a game activity. During this study, students participated in 3 one-hour-long game sessions regarding electrostatics for 27 students who were high-school seniors. After playing the game over the course of the 3 sessions, the results of the pre- and post-tests showed an increase in the average number of correct answers from 6.11 to 10.00, with standard deviations of 5.03 and 7.54. They concluded that the game activity was effective for students’ science learning.
The systematic review from Gao et al. (2020) of mobile game-based learning in STEM classrooms, they reviewed 30 studies from 2010 to 2019. From these studies, 28 included data regarding students between preschool and high school. Twenty-seven studies stated the duration of the game-based learning activity with 14 of the studies reporting that the game activity was conducted within one day, seven studies were less than a week, two studies were longer than a month and one study was 3 months long. The systematic review did not clearly state the amount of time per lesson or course or any other regulatory information regarding how the time was spent. The review did state that some of the game playing was conducted outside of the classroom hours or remotely from home. A possible limitation identified from this study is the inclusion of game-based learning activities occurring outside the in-classroom instructional period.
Summary
The field of game-based learning has created a large amount of insight into the question “can we use games in the classroom” and is now starting to focus on other applications for developing this field. One of the research questions in Wang et al. (2022) study was “are the learning gains higher when using the digital game to support STEM education as compared to non-digital game-based methods?” With the resulting effect size found was 0.56 and the suggestion was that yes, game-based learning was more effective than other instructional strategies, Wang et al. (2022) referenced prior studies from Tsai and Tsai (2020) and Tokac et al. (2019). Tokac et al. (2019) conducted a meta-analysis to measure the impact of learning video games on mathematics achievement compared with traditional methods. They found that mathematics video games contributed to learning compared with traditional methods. It is with these findings and the moderators listed above that this meta-analysis seeks to identify any new trends in development for using game-based learning and mathematics in K-12 grade students during the data collection process of chapter IV and the discussion of findings in chapter V.
Researchers and educators have been interested in the potential of computer programming and software design in creating authentic and meaningful contexts to teach students design and problem-solving skills (Akcaoglu, 2016). In general, grade level, game type, gender, and frequency of game-based activities have been investigated in many studies and have various effects on game-based learning. The above listed variables are used as the moderator variables for this meta-analysis. The next chapter consists of the methodology and coding sheet.
CHAPTER III
METHODOLOGY
The purpose of this meta-analysis was to investigate the effectiveness of the usage of games as part of mathematics instruction on academic achievement in grades kindergarten (K) to 12. The main goal of this meta-analysis was to obtain the average effect size and compare the effect sizes for differences between the moderator variables of gender, grade level, and game type, with students’ academic achievement scores. Experimental, quasi-experimental, and pre and posttest comparison research were investigated to identify connects between game-based learning and student-achievement outcomes. This meta-analysis fills the gap in the knowledge by examining courses that are using game-based learning across three platforms of instruction: nondigital, digital on computers, and mobile devices.
This chapter contains the following sections: a description for the search requirements for acceptable studies on game-based learning, the inclusion and exclusion criteria, a description of the coding process and the general characteristics of the studies to be collected, a description of the data collection, and an explanation of how the data were analyzed and how effect sizes were calculated. The meta-analysis was conducted in accordance with processes from Card (2012), Lipsey and Wilson (2001), and Borenstein et al. (2009).
Research Design
This research was conducted using a meta-analysis design approach. A meta-analysis is “a quantitative method of synthesizing empirical research in the form of effect sizes” (Card, 2012, p. 7). An effect size, according to Card (2012) is used to compare studies by using a standardized measurement. The goal of this study was to search the literature from 2010 to 2023 for experimental studies regarding game-based learning for mathematics content, to identify and synthesize effect sizes findings that were reported for academic-achievement scores and moderator variables of category of games, game platformed used, grade level, gender, subject discipline, and frequency of game-based activities. By completing a meta-analysis design approach, this dissertation met the needs mentioned in chapter I from Young et al. (2012) and Wang et al. (2022) for additional quantitative research data. This meta-analysis has reviewed published, peer-reviewed scholarly works, conference papers, and dissertations from 2010-2023. The studies included have taken place in public schools in the United States due to the standards and expectations from the U.S. Department of Education’s policies and the NAEP testing population. The inclusion of conference papers and dissertations was to prevent publication bias. Expanding on prior research, this meta-analysis search included databases relating to Science, Technology, Engineering, Mathematics, Computers, and Education.
Search Strategies
To be included in this meta-analysis the following steps and conditions were considered:
1. Publications of studies must have been conducted between 2010 and 2023, used game based learning as part of the learning process, have been reported in English, and the sample included only students from the United States.
2. Student participants must have been in grades K -12.
3. The research must have included mathematics curriculum as part of the study’s design plan.
4. The study must have included at least one game-based learning activity.
5. The research must explain how the game(s) were used as part of the course plan and how or if the game activity was assessed.
6. The study included quantitative data for quantitative data analysis to be conducted.
When reviewing prior studies and meta-analysis, the required search terms had to be included in either the title or the abstract and were broken into three levels of requirements. Each study that was kept for final review met all three of these level requirements.
The first level was confirming the study relates to the topic with the following terms to identify studies that were on topic for review: “game-based learning,” “gamification,” “games in learning,” “educational games,” “serious games,” “simulations,” “digital games,” “nondigital games,” “serious educational games,” in addition to the limiters of ““mathematics,” and “statistics.” Using these terms allowed for the selection of studies that focused on game-based learning studies that were connected to the subject domain of mathematics. The second level of requirement included reviewing study abstracts that confirmed the study focused on students from the United States that were in grades K through 12 that can be broken into categories of “primary-school students,” “middle-school students,” and “high-school students.” The third level of requirement was that studies also must have collected quantitative data that should have been effect-size reports regarding achievement since completing the game-based learning activity or intervention.
Data Sources
After reviewing previous meta-analysis, the selection of journals and databases for the first wave of the search was conducted using Academic Search Complete, American Psychological Association (APA) PsycINFO, Education Source, Educational Resource Information Center (ERIC), and SCOPUS. Due to the nature of reviewing studies that use game based learning as part of the instructional design, the scope was extended to include studies found from the following journals: Computers in Human Behavior, Science Direct, Tech Trends, Journal of Educational Technology Systems, Simulation and Gaming International Journal of Research in Education and Science, and Computers and Education. The third wave of article searches was completed using Google Scholar to investigate cited sources from the first wave of articles. Finally, this search included reviewing works found in the Dissertation Abstract International database and used ERIC to identify additional conference papers.
Figure 1. Flow diagram of the search, screening, and selection process
Inclusion and Exclusion Criteria
Studies that were included in this meta-analysis if they met the following criteria:
1. The research investigated the usage of game-based learning as part of the mathematics curriculum.
2. The student population was described based on a student population in grades K-12.
3. The research reviewed conducted in the United States.
4. The research analyzed quantitative data.
5. The dependent is academic achievement scores that can consist of grade point average (GPA), final grade, or an academic assessment score.
6. The moderator (independent) variables included are either grade level, game type, game platform, gender, or frequency of game-based learning activity.
7. The studies were published between 2010 and 2023.
Exclusion criteria
Studies that were not used for this meta-analysis due to the following conditions:
1. The research was conducted outside of the United States
2. The research was not published in English.
3. The student population was described as either preschool or postsecondary education level.
4. The research did not use or include mathematics as part of the curriculum as the focus of the study.
5. The research did not report effect-sizes comparisons or statistical data that could be used to obtain an effect size.
6. The research focused on qualitative or anecdotal data analysis.
7. The research was only focused on theoretical concepts.
After reviewing the criteria list above, there were a total of 17 eligible studies for the final selection in which the researcher examined for data comparisons. The data-collection process included several working sessions between the researcher and the chair of the dissertation committee to confirm eligibility.
Publication Bias
As the field of game-based learning is relatively new, some of the data collected involved small sample sizes that could effect the way statistical significance was reported. This study has conducted a meta-analysis on research studies that followed the inclusion criteria listed above. During the data review, appropriate caution was taken when completing comparisons. Card (2012, p. 262) stated that another way to ensure there is no publication bias is to include code for different types of publication. This study includes dissertations that met the above inclusion criteria. Of the selected 17 studies, 6 were dissertations. According to Card (2012), there are six approaches that can control publication bias, including moderator-analysis, funnel plots, regression analysis, fail-safe N, trim and fill, and weighted selection. This meta-analysis used both funnel plots and fail-safe N to investigate publication bias.
Funnel plots
According to Card (2012, p. 263), “funnel plots represent a graphical way to evaluate publication bias.” By using funnel plots, this research presents a collection of scatterplots of effect sizes in studies related to each study’s sample sizes (Card, 2012, p. 263). This research analyzed data from effect sizes and has displayed this information in funnel plots, since the number of studies included in the meta-analysis is considered small by including only 17 studies. An asymmetrical funnel plot could refer to publication bias due to the small number of studies having small effect sizes that is less than the average. The average effect size found was 1.06 as indicated by the vertical line. Only five studies above the average effect size and of those 5, only 3 would be classified as outliers, as seen in Figure 2.
Although the majority of the effect sizes are in the lower left side of the funnel plot, there is an even spread of results moving across the chart toward the right side. A possible rationale for why Figure 2 looks this way is due to the small number of studies that were used to calculate the effect sizes. As seen here in Figure 2, there are two main outliners that have either a large effect size or a large standard error.
Fail-safe n
The goal of fail-safe n, or the number of studies with null results, is to identify publication bias. According to Card (2012, p. 268), the fail-safe is the number of excluded
studies all averaging an effect size of zero that would have to exist for their inclusion in the study to lower the overall effect size to nonstatistically significant. The main formula for considering fail-safe n consists of finding z scores of each study involved. Cooper et al. (2009) commented that the usage of z scores did not fact into account the sampling sizes of the studies included.
Additionally, Card (2012, p. 270) noted that there is room for interpretation of how to identify publication bias, the widely accepted formula for publican bias is 5(k)+10 regarding the number of needed studies to conduct the meta-analysis to help with publication bias concerns. The result of this equation for this meta analysis was 5(17)+10 so the fail-safe N should be greater than 85.
For this dissertation, the k was equal to 17 studies. After reviewing the data collected and computing Hedges’s g, the number of studies with a null result should be 85 or more. One rationale for the small number of published studies that were excluded from this meta-analysis was specifically the requirement of game-based learning taking place in the United States rather than worldwide, publication bias is unlikely for this meta-analysis.
Instrumentation Reliability
This meta-analysis includes studies and dissertations that were based on the inclusion criteria. All the studies selected were in accordance with the following coding section. According to Card (2012), there is a need to ensure that the empirical data can be replicated. To ensure replication, one method is to have more than one person conducting the coding independently (intercoder), This study controlled intercoder reliability by having a second coder who evaluated the empirical data of five studies and the coding sheet. The second coder holds a master’s degree in social welfare with prior experience conducting quantitative data research in the San Franisco Bay Area. The researcher and the external coder met over virtual meetings twice, once to review directions and later to compare and evaluate the information gathered. There was a 1-to-7-point system assigned to the studies in which any study that gained 5 or more points was deemed problematic unless there was a factor of the paper to be included. There was a 93% agreement rate when each of the data sets were compared. During the review period, the coder and the researcher reviewed and discussed the disagreements, which ultimately resulted in a 100% agreement.
Coding
The goal of the coding sheet was threefold: the first was to provide a guide with clear directions for other reviewers of studies included in this meta-analysis, the second is to establish consistency for multiple reviewers and finally, the last reason was to create a documented process of evaluation and to provide transparency (Card, 2012, p. 77). The key words and the database used for the initial search are listed above in the Data Sources and Search Strategies sections. The data collected was conducted in accordance with the coding sheet. Each study was given a number when collected and 5 of the studies were reviewed by the second coder. The coding sheet outline for this study is listed in the following section and a copy of the sheet itself is in Appendix A.
General information
In this section, the coder identified general characteristics that included the title of the study, the author, the publication year, the American Psychological Association (APA) citation, the report type (whether it was a journal article, dissertation, book chapter, conference paper, or other), the study code number, and the peer-review status.
Setting and participation characteristics
The coder reported data that included the sample sizes and details regarding the student characteristics reported such as gender and grade level.
Game-based learning characteristics
The game-based learning tool that was used in the study was coded. Included are characteristics such as if the game is already an available product, whereas it was adapted, or was specifically made for the course itself. Additionally, the coder identified what game type was provided as either digital or nondigital and if the game was a commercial game that was adapted for the lessons, a serious game, educational game, or simulation. Finally, the coder identified if the game was classified as a mobile game and if a device such as a smartphone, tablet, or laptop was needed.
Program or course characteristics
This section sought to identify descriptive information about the course. Data collected here involved the mathematics domain, the frequency of game-based learning activities during the experiment and prior attempts for using game-based learning in the course. The reported mathematics content domain for the selected studies were reviewed and then synthesized into the categories of algebra, fractions, ratios, problem solving, and arithmetic.
By knowing the frequency of game-based learning activities, researchers can check in on the repeatability and standardization of a game-based lesson plan. Data collected here included the length of game-based instruction and how often the activities were conducted for the research period. Frequency was measured by the number of minutes per session, the number of sessions during a game-based learning experiment in which games were played and the number of weeks for the study. Finally, this section included a measurement regarding the prior usage of game based instruction for prior versions of the course.
Instructor characteristics
In this section, the data collected reported if the instructor(s) reviewed have conducted game-based learning lessons and if the instructor reported a level of familiarity with the game(s) used in the experiment. For the coding process, limited familiarity would be considered having less than 10 hours of reported time with the game, medium familiarity would be over 10 hours and less than 50 hours with the game(s) and great familiarity would be over 50 hours reported interaction with the game(s). This measurement of familiarity originally was considered to identify any potential bias in a study but ultimately was labeled as a limitation for the study. Additionally, there was a collection of who conducted the experiment if it was the teacher or researcher or both.
Research-design characteristics
In this section, data were collected regarding what the design plans were in the studies collected. The main goal for this section was to identify the method of measurement for the game-based learning activities. The two main categories reported were if the researchers used a standardized achievement test comparison, pre-posttest comparison, and posttest only report as part of the experiment. Other information gathered here included the sampling type and if the teacher or researcher conducted the experiment.
Moderator variables found
In this section, the coder reported which of the moderator variables were compared for the dependent variable of academic achievement in mathematics, if there was a subdomain for mathematics identified, if it was a gender comparison, a grade-level comparison, a game-type comparison, or a frequency of game-based activity comparison, and if there were any cross comparisons.
Effect sizes and statistical information reported
In this section, the coder reported what type of effect sizes that were reported such as sample sizes, means, standard deviations, t-test values, p values, any raw differences, and a section for “other” was available to mark any effect sizes missed above.
Study quality
In this section, the coder gave a rating between 0 to 7. For each study, if there is insufficient information from one of eight above listed sections in the coding of this meta analysis in the body of the research, the study was assigned a point. Once a study reached 7 points, it was marked as “not specific enough, low quality.” Any study that had between 1 and 6 points was marked as “medium quality” study and any study with 0 points was marked as “good quality.”
Research Questions
This meta-analysis addressed the following research questions:
1. What is the average effect size of using game-based learning on academic achievement in K- to -12 th- grade mathematics classes?
2. Does gender differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
3. Do the grade levels differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
4. Does the game platform differ in their average effect size of mathematics focused game based learning courses on academic achievement as a moderator variable?
5. Do the game types differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
6. Do subject subdomains differ in their average effect size of mathematics focused game based learning courses on academic achievement?
7. Does the frequency of game-based learning activity differ in their average effect size of mathematics focused game-based learning courses on academic achievement?
Data Analysis
Upon completing the coding of the studies, effect sizes were calculated regarding the variables listed above regarding game-based learning in mathematics courses. To compare one dataset to another, the data were first converted to a standardized measurement. Based on the suggestions by Card (2012, p. 90), the researcher conducted the required steps needed to calculate Cohen’s d and Hedges’s g. Both methods tools are used to calculate meaningful effect size information, but Hedges’s g is often used regarding smaller sample sizes to correct for bias. There was testing conducted to check for homogeneity and analysis of variance (ANOVA) was used to identify any trends from the moderator variables on academic achievement scores.
CHAPTER IV
RESULTS
The purpose of this meta-analysis was to investigate the effectiveness of the usage of games as part of mathematics instruction on academic achievement in grades kindergarten (K) to 12. Although the field of game-based learning is a newly growing research field, game-based learning expands across multiple dimensions and prior studies have suggested that results of experiments using game-based learning have complicated results. According to Card (2012), the process of conducting and reporting findings from a meta-analysis includes the following two main aspects. First, the researcher attempts to generalize findings from specific studies. Second, the researcher attempts to settle conflicting evidence. These two main aspects were present in the process of this meta-analysis. The main goal of this meta-analysis was to obtain the average effect size and compare the effect sizes for differences between the moderator variables of gender, grade level, and game type, with students’ academic achievement scores.
This chapter of the meta-analysis fills the gap in the knowledge by reviewing quantitative research on studies that conducted game-based learning activities and identifying any patterns that occurred from the moderator variables relationships to the subject matter.
Overview of Results
When conducting a meta-analysis and combining results from multiple studies, there are two models for consideration when calculating results and the providing interpretations, those models are the fixed effects and the random effects. According to Card (2012), the interpretation of results using these two models differ based on the number of studies. When using a random- effects model, the goal of the results would be suggesting larger generalizations of the literature or research as a whole on the topic and is an estimate of the average of the distribution of treatment effects across various study settings. Fixed-effects model focuses on the studies that were included in the meta-analysis and will draw conclusions from those studies only. Fixes effects model is the usual average of all the effect sizes. Because this meta-analysis consisted of only 17 studies, the usage of the fixed-effects model was selected.
This meta-analysis investigated, reviewed, and synthesized data and results from 17 peer- reviewed and published studies. Of the 17 studies, 6 were dissertations and are available on scholarly databases. Each study included here worked with digital games in the classroom. Table 3 contains descriptive information for the 17 studies that initially were selected for this meta analysis.
An initial overview of the information in Table 3 indicates the following observations. The learning measure for the experiments were split between the usage of a standardized achievement test and pretest, posttest comparison with only one study conducting a posttest only comparison. Excluding the outlier due to extremely large sample population, the sample population range was 11 students to 437 students. Gender was assessed for female numbers with the eight studies have the majority female students. Nine studies would have the majority male students. The grade level focus was 50% middle-school students, whereas the remaining 50% was equally split between primary and high school. Regarding mathematics domain, algebra was the majority of the mathematics domain for these studies. Algebra and problem solving were the only mathematics-domain subjects taught at the high-school level. At the primary-school level, problem solving, and arithmetic were the mathematics domains. At the high-school level, algebra, ratios, and problem solving were the mathematical domains. Last, as the year of publication was from 2010 to 2023, 69% of the studies reviewed were before 2017 with three studies published in 2023.
Table 3
Descriptive Information for the 17 Studies Included in the Meta-Analysis
a=F=female students, NA=no clear information was provided,
b=HS (High school Grades 9-12), MS (Middle school Grades 5-8), PR (primary school Grades K-4)
c Gibbs (2020) and Hunt et al. (2023) both included more than one grade category but in the writing their focus was on middle school
Research Questions
This meta-analysis addressed the following research questions:
1. What is the average effect size of using game-based learning on academic achievement in K-to-12 th-grade mathematics classes?
2. Does gender differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
3. Do the grade levels differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
4. Does the game platform differ in their average effect size of mathematics focused game based learning courses on academic achievement as a moderator variable?
5. Do the game types differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
6. Do subject subdomains differ in their average effect size of mathematics focused game based learning courses on academic achievement?
7. Does the frequency of game-based learning activity differ in their average effect size of mathematics focused game-based learning courses on academic achievement?
Research Question 1
The overall effect size for the 17 studies regarding the effect of game-based learning on mathematics achievement scores for students in kindergarten through 12th grade in the United States was 0.30, with a standard error of 0.03. The 95% confidence interval (CI) is (0.24, 0.36). Because zero is not in the CI, this result indicates that the average effect size is statistically significant at the .05 level. The test for heterogeneity resulted in a Q of 283.30, with 16 degrees of freedom that is statistically significant at the 0.95 level of significance. Table 4 includes the effect sizes, sample size, Hedges’s g, and the confidence intervals for each of the studies testing measurements.
Table 4
Overall Effect Sizes of the 17 Studies Included in the Meta-Analysis
After completing the first analysis, the sample-size population of one study was vastly too large compared with the other sample populations and was removed from the rest of the analyses. It should also be noted that the overall effect size only changed by 0.01. Schenke et al.’s (2014) study was only used to compute the overall average g and then removed from further analysis. The overall effect size for the 16 studies regarding the effect of game-based learning on mathematics achievement scores for students in kindergarten through 12th grade in the United States was 0.29, with a standard error of 0.03. The 95% confidence interval (CI) is (0.24, 0.36). Because zero is not in the CI, this result indicates that the average effect size is statistically significant at the .05 level. Testing for heterogeneity of effect sizes resulted in a Q of 283.04, with 15 degrees of freedom that is statistically significant at the 0.95 level of significance. Given this large value for Q, investigating the effects of moderator variables was undertaken.
Moderator Variable Results
To answer the remaining six research questions, data were collected regarding the moderator variables including gender, grade level, game platform, game type, mathematics domain, and frequency of instruction.
Research question 2
The comparison conducted for this question was regarding the reported gender of students in each study and if there were any data about outcomes in that relation. Overall, 14 studies reported male and female students in sample population whereas 3 studies were not clear about the descriptives for their students. Furthermore, almost none of the studies directly reported the outcomes of student’s achievement scores in relationship to the moderator variable. Therefore, for this question, the data were measured by calculating the effect size of each study and applying it to the largest reported percentage of that moderator on the sample size. The studies were separated into three categories, studies that had a larger male sample size, studies that had a larger female sample size, and studies were gender was not reported. The results of the analysis suggest that there was statistical significance for all three categories are found in Table 5.
Out of the 17 studies, 13 reported descriptive data regarding the gender composition of the student in the study respectfully. According to results in Table 5, there was statistically significant for each category. There was an overlap in the 95% CI between the not reported category and the studies that reported a large female composition of the study. The 95% CI not reported category and the large male composition category overlap totally.
Table 5
Results of Effect-Size Comparison for Gender
*Statistically significant at the 0.05 level
Research question 3
Grade level was investigated as a measure to discover which student population would benefit the most from game-based learning. Studies were separated into three categories: primary (grades kindergarten through 4th grade), middle school (grades 5-8) and high school (grades 9-12). All 17 studies reported grade levels for the students in the population. According to Table 6, the largest effect size was 0.51 for the primary-grade category. An initial reading of this result shows that the high-school students and the middle-school students had the smaller effect sizes but with a wider range. Additionally, the CI indicated an overlap between the three categories.
Research question 4
The comparison of game platform was to identify the usage of technology that was tied to the game-based learning activities and identify if any of the tools had a significant effect. According to Table 7, of the three categories (computers, mobile devices, and not reporting how
Table 6
Results of Effect-Size Comparison for Grade Level
*Statistically significant at the 0.05 level
the digital games were played), mobile devices had the lowest effect size of a 0.07 with a negative CI range. The QB of 13.22 is significant regardless of the mobile category. There is a wide range for the confidence interval for computer usage and the not reported category, in which it could be considered to use the same device. From the studies selected, the information was missing.
Table 7
Results of Effect-Size Comparison for Game Platform
*Statistically significant at the 0.05 level
Research question 5
This comparison was regarding the definitions of terms for game-based learning. Sixty three percent of the studies selected for this meta-analysis did not use one of the terms such as serious game or [serious] educational game. There were zero studies to be classified as simulations. Regardless of the definition, there was some statistical significance for studies that were not classified as a serious game. Note that the QB also was statically significant for this variable.
Table 8
Results of Effect-Size Comparison for Game Category
*Statistically significant at the 0.05 level
Research question 6
Mathematics domain was broken into five categories, all of which resulted in a statistically significant effect size. The largest effect size was found to be arithmetic, which was the focus of only one study. Of the seven studies classified as algebra, that category included algebra I, II, and pre-algebra. Algebra is also the largest percentage of selected studies and has a small average effect size. This variable also had the largest QB of 63.51.
Table 9
Results of Effect-Size Comparison for Mathematics Domain
*Statistically significant at the 0.05 level
Research question 7
The effect-size comparison for frequency was broken into three categories to measure the amount of time spent playing the game-based learning activity, the number of times the students conducted the activities each week, and finally the number of weeks to complete the experiment.
The number of minutes had the following three categories: under an hour, over an hour, and not reported. The number of sessions per week was one session, 2 sessions, 3 sessions, or the number was not reported. Each of these sessions had statistical significance. Finally, the number of weeks was divided into under 10 weeks, 10 weeks or more and not reported. All three categories were statistically significant. Only the QB for minutes and weeks was found to be statistically significant. The largest effect size in Table 10 was found to be spending an hour or more of instructional time on the game-based learning activity and it would have the largest CI range as a group of studies. The CI the total number of weeks not reported category was only one study, thus leading to be smaller overall result insight comparatively.
Table 10
Results of Effect-Size Comparison for Frequency
*Statistically significant at the 0.05 level
Additional Analysis
Publication years were compared to identify how the studies were related to one other in terms of heterogeneity. The year 2016 was selected as the “middle” point for this meta-analysis and the findings of 9 studies with an average g of 0.22, SE of 0.04 and a 95% CI [0.14, 0.29] from before 2016 compared with the 7 studies from 2017 to 2023 with an average g of 0.47, standard error of 0 and a 95% CI [0.46, 0.47]. This comparison suggests that the studies post 2016 are more similar in their design or measurement or their effect. From the aspect of learning over time, although the 2010-2016 studies had a smaller effect size overall, the lack of overlap from the CI refers to a larger range of results in the first category.
Summary
This meta-analysis included 17 studies that met the inclusion and exclusion criteria. Although some of the studies investigated multiple variables, there were only 17 effect sizes used to measure the pretest and posttest comparison of using game-based learning content in classrooms. The overall average effect size was 0.29, which is classified as a small effect size. In these selected studies, digital games were used as part of the game-based learning instruction mathematics content. Comparisons were calculated regarding the game platform, game type, gender, grade level, mathematic domain, and three measures of frequency of the instructional activity. Differences were found regarding game platform, game category, gender, math domains, time spent per session and number of sessions per week. Although the overall effect size is found to be small, the effect size is still considered to be statistically significant at the 0.05 level. The test for homogeneity was statistically significant and in addition, when investigating the 17 studies included, one study had a large disproportion sample population relative to the other studies, therefore this study was removed from moderator analysis. The difference in overall effect was less than 0.01, thus the overall effect size of 16 and 17 studies is considered the same.
The results are summarized in chapter V along with implications for practice, the limitations of the meta-analysis and proposal for future work.
CHAPTER V
SUMMARY, DISCUSSION, IMPLICATIONS, AND RECOMMENDATIONS
This study was a meta-analysis of the effectiveness of using game-based learning on academic achievement and possible variables that effect academic achievement in kindergarten (K)-12 classrooms in the United States. The study of using games in a classroom setting is classified as game-based learning, which is the process and practice of learning using games from the learner’s point of view. Game-based learning is any instructional activity based on the usage of games (Talan, 2019).
Meta-analysis is the collection and investigation of multiple studies in which each study’s effect sizes are evaluated, converted to a series of standardized measurements, and then compared to find generalizable results. Depending on the number of studies included the results from a meta-analysis could suggest what a group of studies are reporting or make a generalized inference regarding the field of the subject matter (Card, 2012). Meta-analysis was conducted to investigate research conducted on game-based learning studies that occurred in the United States from 2010-2023.
To investigate the effects of moderator variables in this meta-analysis, the following moderator variables were selected: game category, game platform, grade level, gender, mathematics domain, and frequency of instruction.
Summary of the Study
The search for information into the field of game-based learning resulted in 917 studies to be reviewed. Before screening, 193 were removed due to duplication, 44 were removed by automation tools, and 6 were removed due to errors. During the screening process, 683 articles did not meet the first set of requirements regarding the inclusion criteria in the abstract. Once the process of retrieval and assessment for eligibility was conducted, 150 studies were selected for additional screening. The process of removing studies had to do with the location of where the research was conducted and the usage of measurement, in most cases the lack of quantitative processes. This collection of studies was narrowed to a small sample of 17 studies that followed the main criteria of consisting of quantitative research, involving K-12 students, the study being conducted in the United States, and finally the course in which the game-based learning activity was conducted focused on mathematics education.
The final studies that were selected have a range of student grade levels, only three studies did not report gender descriptions. To review student’s academic achievement in mathematics, each of these studies conducted tests regarding mathematics content. Sixteen of the studies conducted either a standardized achievement test or a pretest, posttest comparison. One study specifically conducted only a posttest comparison.
Of the final 17 studies, 6 were nonpeer-reviewed dissertations, whereas the other 11 were from peer reviewed journal articles.
Summary of the Results
In this section, results of the meta-analysis for each of the seven research questions are provided.
Research Questions
This meta-analysis addressed the following research questions:
1. What is the average effect size of using game-based learning on academic achievement in kindergarten- to-12 th- grade mathematics classes?
The average effect size overall for this meta-analysis was 0.29, which is considered a small effect size but is still statistically significant. According to Card (2012), because the sample size of studies is small, the outcome that can be suggested here is that this collection of literature found significance. The test for homogeneity was statistically significant and in addition, when investigating the 17 studies included, one study had a large disproportionate sample population relative to the other studies, therefore this study was removed from moderator analysis. The difference in overall effect was less than 0.01, thus the overall effect size of 16 and 17 studies is considered the same.
2. Does gender differ in their average effect size of mathematics focused game-based learning courses on academic achievement as a moderator variable?
Although gender was reported for 13 of the studies, there were statistically significant average effect sizes found for both categories and for the nonreported studies. There was a small difference identified between male and female students, as studies with a larger male percentage of sample size had an average effect size of 0.21, whereas studies with a larger female percentage of the sample size had an average effect size of 0.47.
3. Do the grade levels differ in their average effect size of mathematics focused game based learning courses on academic achievement as a moderator variable?
Grade level was reported for each study included in this meta-analysis, and there was an identifiable difference between the three categories of primary-, middle-, and high-school students. Primary-grade-level students had a medium average effect size of 0.51, whereas the other two categories had small average effect sizes of 0.20.
4. Does the game platform differ in their average effect size of mathematics-focused game-based learning courses on academic achievement as a moderator variable?
From the selected studies, digital games were part of the game-based learning activity and in which the platform of a computer or a mobile device was identified for 14 of the studies with the remaining two using the term “game-based application.” Ten of the 14 studies used computers in some way. The computer category had an average effect size of 0.40. Although most of the reported studies were classified as using a computer, there was not sufficient information regarding whether the computers were laptops being used in the classroom or in a computer lab. From the selected articles, mobile devices included phones, e-readers, iPads, and “Gameboy” devices.
5. Do the game types differ in their average effect size of mathematics focused game based learning courses on academic achievement as a moderator variable?
The studies selected were reviewed for what definition was used to categorize the type of game that was being used in the classroom for the mathematics instruction. The categories for game type, according to Lamb et al. (2018), were serious games, serious educational games, and simulations. Ten of the studies did not use one these terms and instead just referred to the instructional method as playing a game-activity during the instructional process. The 0.40 average effect size was found for those with no term present, whereas the two studies that did state “educational game” as a term, which was the coded second category of game, had an average effect size of 0.24. A possible rationale for this overlap of the confidence intervals for educational games category and the not reported type is that many study reporting games could be classified as educational games.
6. Do subject subdomains differ in their average effect size of mathematics focused game-based learning courses on academic achievement?
Question 6 sought to identify what kind of mathematics domain was present in the studies gathered for this meta-analysis. The coding for this section consisted of five categories, including algebra, fractions, ratios, problem solving, and arithmetic. Of the five categories, all five were found to have some statistical significance. Arithmetic, consisting of only the study by Shin et al. (2011), had the largest effect size of 3.10, whereas the range average effect sizes of the other four categories was 0.18-0.82. Ratios and problem solving both resulted in a large effect size of 0.80 and a medium effect size of 0.50.
7. Does the frequency of game-based learning activity differ in their average effect size of mathematics focused game-based learning courses on academic achievement?
Frequency was investigated to provide guidance for future study by gathering information on the number of minutes used per game-based session, how many sessions per week, and how many weeks in a game-based learning experiment. Thirteen studies reported the number of minutes in the game-based learning session with sessions that were an hour or longer having the larger average effect size of 0.90. Additionally, 13 studies reported the number of sessions per week, but the difference was marginal between the average effect sizes of 0.40 and 0.50. Finally, the number of weeks, both reported and not reported, resulted in small average effect sizes.
Limitations
According to Card (2012, p. 328) limitations usually are related to the data report, the publication bias, or in the case of a meta-analysis, the way the data were interpreted. Limitations can hinder the researcher’s ability to draw conclusions on the data for this meta-analysis. Limitations that were reported by the meta-analysis of the obtained studies include too small of a sample size, not sufficient time to prepare the game-based lesson, not enough time using the activity and meeting the course goals, lack of planning or familiarity with the tools from both students and teachers, a lack of a procedural feedback, and the rigidness of the selected games.
Regarding identifying whether there was publication bias, this study used Hedges’s g to compare average effect sizes and an intercoder collaboration system to review articles. According to Cooper et al. (2009) and Card (2012), it is difficult to conclude if there is publication bias with a small sample size of 17 studies. In addition to the selected studies also calling for an increased sample size, the Schenke et al. (2014) study had a very large sample population that results in a very small effect size.
The following sections include a discussion of the results, implications for instruction, suggestions for future research, and finally a conclusion.
Discussion of the Results
As found in chapter I, the meta-analyses by Clark et al. (2016), Lamb et al. (2018), Tokac et al. (2019), and Wang et al. (2022) sought to compare game-based learning with traditional instruction. These four studies were considered as supporting information to drive this meta-analysis. All four found statistically significant average-effect-size results with Lamb et al. (2018) and Wang et al. (2022) finding medium to large effect sizes, which is contrary to the finding of a small average effect size in this study. One possible reason for the smaller effect size is that previous meta-analyses were not limited to studies published based on students in the US only. Clark et al., Lamb et al., and Wang et al. did not consider country of origin for the students in their meta-analysis, but a review of the studies in the meta-analysis indicated that there were many countries represented in addition to the US. Tokac et al. investigated country as a moderator variable comparing US with other countries. The results based on 17 U.S. studies with an average effect size of 0.10, standard error .08, and 95% confidence interval (CI) [-0.06,0.26] and based on 22 studies outside of the US with average effects size of 0.16 with standard error of .08 and 95% CI [0.00, 0.32]. Tokac et al.’s results differ from the current study in that neither US nor other countries’ effect sizes could be considered statistically significant.
When designing this goal for the meta-analysis, an observation from prior research was that there was research produced about the usage of computer games in education; however, the research into the usage of games and game designs that support learning is an ever-growing and overall, a new field (Johnson & Mayer, 2010). Because the field of game-based learning is shifting constantly definitions and research goals, this meta-analysis is a contribution regarding recent findings that agrees with the findings from prior meta analyses.
Another one of the goals of this meta-analysis was to identify and synthesize more quantitative research for game-based learning and processes that could be applied by future researchers and instructors. When designing this meta-analysis, several factors for how to fill the gap in literature were considered. Young (2012) noted that findings for game-based learning were complicated at best, and overall, the critics or comments for that study was that game-based learning struggles to be seen as effective. A solution to that problem is to provide more quantitative studies. The call for more quantitative was found in many of the articles consulted during the beginning of this process such as Byun and Joung (2018).
During the process of identifying articles and synthesizing the findings of this meta analysis, an observation indicates the lack of quantitative research work for game-based learning in the United States. When conducting the search for articles, search results in the thousands were populated when searching. As the purpose of this meta-analysis was to focus on students in the United States, there were many of the research articles that were eliminated due to the location of the study. The main reason for narrowing the studies by region was that students in the United States are being measured by standardized tests that follow a unified standardization and according to those results, are struggling in mathematics education (Schwartz, 2023). The United States as a requirement for the search, lead to a limit to the number of usable studies.
Following in the call for more quantitative studies, researchers sought to identify the tools for game-based learning and comparing their usability (Gao et al. 2020). Fourteen of the selected articles identified the technology used in the classroom and a possible limitation to the lack of mobile phones. A rationale for why mobile devices were not a larger category in the research selected and why mobile devices resulted in a negative CI range would be that phones may not be allowed in schools or are not compatible with the software used for educational games. In addition, this compatibility problem also could be connected to any of the technologies used for game-based learning. The costs of acquiring properly working technologies is a factor that is beyond this meta-analysis.
A possible reason for the lack of game types appearing in the field is the ever-changing definitions (Echeverria et al. 2011; Lamb et al., 2018; Johnson & Mayer, 2010). Lamb et al. (2018) defined three terms for games in education. Serious games were supposed to teach students how to complete tasks that used examples to connect with the students. Serious educational games were video games played in immersive (3D) environments that had a direct teaching approach. Simulations were games used as activities that allow for near infinite practice on a task. Although Lamb et al.’s (2018) study resulted in a positive effect sizes for each category– serious games (0.79), serious educational games (0.63), and simulations (0.50) — this meta analysis can conclude that no matter the term used, game-based learning has a positive effect size.
The investigation conducted in this meta-analysis was to identify what terms were being used when classifying or describing the usage of games in classroom courses. Lamb et al. (2018) suggested that the development of games in regarding immersion, such as three-dimensional games that allow students more interaction with the game space or environment compared with basic movement or simple actions may be an attempt to keep students engaged with the content. Although eight studies used a term to define the game that was used, it is possible that researchers consider the usage of a game for education could just be called an “educational game” that does not match with the definitions suggested by Lamb et al.
As the technology field continues to advance, so do the games, the instructional software, and the platforms that could be used for a game to be conducted on. Wang et al. (2022) found that using a mobile device for game-based learning had a large average effect size of 0.77 with a CI of [0.50, 1.04] This CI did overlap with the CI for computer users, but the finding suggests that students using their phones for game-based learning were improving in the classroom. An inference can be made regarding the usage of the word “app” or application. This word was found in more than one study but was not clearly defined as to what device was used to run the application. Finally, regarding the word “app,” the development of technology between 2010 and 2023 would most likely refer to an iPad or phone rather than a computer application.
Grade-level comparison was one of the main goals of this meta-analysis. The findings from Vu and Feinstein (2017) stated that as students progressed in their grade level, their performance in Science, Technology, Engineering, and Mathematics (STEM), in this case mathematics, decline. The results from this study concluded that the most important effect on mathematics was for the primary students. Although Wang et al. (2022) found that primary students had the larger effect size of 0.84 with a standard error of 0.11 and a CI of [0.63, 1.04], the results from these 17 studies found a consistency with that result. The 17 included studies found that the primary students had the largest effect size result of 0.51 with a standard error of 0.07 and a CI of [0.38, 0.64]. A possible limitation to the results found for the grade level comparison is that content designed for primary grade levels may not be as complex as content designed for higher grade levels.
As students advanced in their mathematics courses, a comparison was made between the mathematics domains to investigate the relationship between students’ performance and the content. This study found that problem solving, ratios, and arithmetic had the potential for larger average effect size compared with Algebra. A limitation to immediately consider is how those domains were instructed with the games included. The information was not clear on the activities for each course that taught Algebra.
Gender was selected as a moderator variable so as to identify any trends of students between male and female students who struggle with the mathematics as they move forward in grade levels. Although most of the studies selected for this meta-analysis did not directly state outcomes regarding gender, as the findings were calculated based upon the reported statistics of the population in each sample. Although the findings by Vu and Feinstein (2017) suggested that gender differences did not occur with interest in STEM, in this case mathematics, for students in fourth grade, which in this case applies to primary-grade students, this meta-analysis has a different result. The findings of this meta-analysis were that the usage of game-based learning had a larger average effect size for the studies that were classified as having a larger percentage of females in the sample population, as the average effect size was 0.47 with a standard error of 0.50 and a CI of [0.36, 0.57] compared with the male participants of an average effect size of 0.21, standard error of 0.50 and a CI of [0.11, 0.31]. It should be noted that comparing the CIs for the two categories there is no overlap, which indicates the two average effect sizes are statistically different. The findings and inferences can be made here to game-based learning has a positive effect size for students regardless of gender, and female students perform well in mathematics courses that use game-based learning activities.
After reviewing the descriptive information for the game-based learning course, the game types used, and the mathematical content that was covered, the next measurement to consider was the frequency of game usage. The idea for this measurement was to give a direct answer of how long a game-based learning session needs to be and how often to result in a positive effect. The prior studies by Clark et al. (2018), Tokac et al. (2019), and Wang et al. (2022) compared the length of instruction regarding the game-based learning activity. Clark et al. found that multiple sessions an average effect size of 0.44 with a CI of [0.29, 0.59], and Wang et al. found that the frequency of sessions of less than a week were found to have the average effect size of 0.95 with a CI of [0.65, 1.26]. This meta-analysis found that using a game-based activity for at least an hour had an average effect size of 0.94 with a standard error of 0.15 and a CI of [0.65, 1.23] and that the students should participate in the lesson one to two times per week.
Although Clark et al. (2018) found no difference in effect size regarding instructors account for how the game was included in the lesson plan– the game could be integrated with instruction either in-game or around the game session or the game is the “main” instruction –, this meta-analysis noted that the findings were not clear with what was being due within with the hour of time with the game, that is, if there is any scaffolding or hybrid instruction taking place, or what is the form of checking in on the students during the hour of game time.
Implications for Practice
According to these results, game-based learning has a statistically significant effect overall on mathematics education for students in the United States. Is using game-based learning a good or bad idea? Factors to consider for the next game-based learning module would be what platform the games are played on, the repeatability of the game used, the tools of measurement and structure that connects the game to the lesson goals, and finally the instructor’s training or practice of using the game.
As mentioned earlier in this chapter, both the technology used in the classroom and the language of using technology is evolving. An “app” or application is the correct term for any program that runs on a device, the main usage of the word “app” refers now to smartphone or a tablet device rather than a computer or laptop.
When designing a game-based lesson or unit, the “how to do it” process must be considered, and all the studies in this meta-analysis found their own way to instruct with games. Planning to complete the lesson in the appropriate amount of time is important as many of the studies selected found that there was not enough time.
An additional comparison was made regarding if the researchers stated how the games were used in the classroom and placed into three categories. The games could be the default setting or out of the box with no changes, the games could be modified to fit the lesson goals, or the games were created specifically for the lesson by the instructor or the researcher, by a company, or by a government entity. The finding when comparing the three categories was no statistical significant regarding the modifications of the games themselves, but an indirect observation is that most of the games were just the default setting and that the teachers had little to not interaction or training with the games. The information to expand on these two points was not in the studies selected. For educators and researchers looking to use games for their next unit or course, a suggestion would be to ensure some familiarity with how the game works, what problems could appear in the game, and the lesson plan structure tied to the game.
Recommendations for Future Research
Observations relating to these findings suggest that there is space for more development for game-based learning in education. This meta-analysis sought to provide information for future researchers in the game based learning field. As the need for more quantitative data continues beyond this meta-analysis, there are four main suggestions consisting of descriptive information gathering, subject matter, grade level, game platform usage, and finally designing the game-based learning activity in the studies.
Based on the limitations of this study, there was a lack of information that needed to be gathered at both the beginning and the end of the study identifying the gender composition of the student sample groups and of the treatment groups between a control and one experiment including their pretest and posttest scores would provide opportunities to gain gender insights in the outcomes. Additional information to gather would be the instructor’s familiarity or training with the game being used. As students are more likely to have used a game outside of the school, an investigation into the instructor’s familiarity would help to identify how the game-based actives are facilitated.
Subject matter, in the terms of STEM versus mathematics, was a limitation due to the inclusion criteria of this study. As suggested by Byun and Joung (2017), a focus for the mathematics domain needs to move away from just algebra and instead involve more complicated subject matter. As this study found algebra had the highest frequency of studies, the larger effect size was identified to be other mathematics domains. When more research is conducted, a clearly defined subject domain, especially when focusing on primary- and middleschool students, is necessary to make informed comparisons. More than one study included in the meta-analysis used the terms tied to state standards rather than clearly identifying the type of mathematics and the domain in which the mathematics lesson was conducted.
Research regarding how games are played in the classroom over several grade levels needs further analysis. The U.S. Department of Education noted that students start to fall behind in their grade level in mathematics grade levels as they pass the fourth grade. These meta analysis findings are in support that primary students had the greatest average effect size regarding game-based learning and that middle- and high-school students both had positive average-effect-size results. One aspect for investigation is what factors result in primary-school students to have the largest outcome for game-based learning in mathematics and how can those factors be generalized to an eighth-grade or twelfth-grade level. Some of these factors for why this finding occurred could be the lesson content, the games that were played, or the novelty of the game that was selected. This information could be considered for future research.
The choice for what device to use in a classroom requires more investigation. When deciding what game to use in a classroom is the technology that would be included for the instructional method. The default platform for digital game-based learning is a computer but there are some socio-economic factors that would need to be considered by the researchers, teachers, and school districts. As the student population moves to using advanced technology, so too must the educational world. Mobile devices and mobile apps are being released more often, and the students are the overall experts in using these digital technologies in their own time.
Finally, more empirical studies that state the structure of the game-based learning activity from a pedagogical point of view are necessary. Lesson plans, inclusion of daily and weekly goals, statements regarding the time spent before and after the game-based learning activity, and descriptions of what is occurring in each game-based learning session would provide insight to create a generalized game-based learning strategy.
Conclusion
Although a small sample of studies resulted for this meta-analysis, the findings were found to be related to prior meta-analyses conducted in the mathematics field. Observations, implications, and suggestions for more detailed quantitative data, clearly defining the ways in which games are used in a mathematics course would be beneficial. Based on the research results found here and the suggestions presented above, it is concluded that game-based learning is an effective tool for helping students learn in primary-, middle-, and high-school mathematics classes.
REFERENCES
*Abdelhafez, A. (2016). The effects of game-based technology on high school students’ algebraic learning in an urban school classroom (Order No. 10240334). Available from ProQuest Dissertations & Theses Global; Publicly Available Content Database. (1865956661). Retrieved from https://www.proquest.com/dissertations-theses/effects game-based technology-on-high-school/docview/1865956661/se-2
Abdul Jabbar, A. I., & Felicia, P. (2015). Gameplay engagement and learning in game-based learning. Review of Educational Research, 85(4), 740-779. https://doi.org/:10.3102/0034654315577210
Akcaoglu, M. (2016). Design and implementation of the game-design and learning program. Techtrends: For Leaders in Education & Training, (2), 114. https://doi.org/10.1007/s11528-016-0022-y
Alaswad, Z., & Nadolny, L. (2015). Designing for game-based learning: the effective integration of technology to support learning. Journal of Educational Technology Systems, 43, 389–402. https://doi.org/10.1177/0047239515588164
Annetta, L. A., Murray, M. R., Laird, S. G., Bohr, S. C., & Park, J. C. (2006). Serious games: Incorporating video games in the classroom. EDUCAUSE Quarterly, 29(3), 16-22.
Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learning outcomes? evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 30, 100322. https://doi.org/10.1016/j.edurev.2020.100322 (Bai et al., 2020)
*Bai, H., Pan, W., Hirumi, A., & Kebritchi, M. (2012), Assessing the effectiveness of a 3-D instructional game on improving mathematics achievement and motivation of middle school students. British Journal of Educational Technology, 43: 993- 1003. https://doi.org/10.1111/j.1467-8535.2011.01269.x
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta‐analysis. https://doi.org/10.1002/9780470743386
Byun, J., & Joung, E. (2018). Digital game-based learning for K-12 mathematics education: A meta-analysis. School Science and Mathematics, 118(3–4), 113–126.
Card, N. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford Press
*Carr, J. (2012). Does math achievement h’app’en when iPads and game-based learning are incorporated into fifth-grade mathematics instruction? Journal of Information Technology Education: Research, 11, 269–286. https://doi.org/10.28945/1725
*Chang, M., Evans, M. A., Kim, S., Norton, A., & Samur, Y. (2015). Differential effects of learning games on mathematics proficiency. Educational Media International, 52(1), 47- 57. https://doi.org/10.1080/09523987.2015.1005427
Chen, P., Hwang, G., Yeh, S., Chen, Y., Chen, T., & Chien, C. (2022). Three decades of game-based learning in science and mathematics education: An integrated bibliometric analysis and systematic review. Journal of Computers in Education, 9(3), 455-476. https://doi.org/10.1007/s40692-021-00210-y
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research, 86, 79–122. http://www.jstor.org/stable/24752870
Cicchino, M. (2015). Using game-based learning to foster critical thinking in student discourse. Interdisciplinary Journal of Problem-Based Learning, 9, 57-74. https://doi.org/10.7771/1541-5015.1481
Colby, R. (2017). Game-based pedagogy in the writing classroom. Computers and Composition, 43, 55-72, https://doi.org/10.1016/j.compcom.2016.11.002.
Cooper, H., Hedges, L.V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). Russell Sage Foundation
Craft, J. (2016). Rebuilding an empire with minecraft: bringing the classics into the digital space. The Classical Journal, 111, 347-364. https://doi.org/10.5184/classicalj.111.3.0347
Denham, A. R., Mayben, R., & Boman, T. (2016). Integrating game-based learning initiative: Increasing the usage of game-based learning within K-12 classrooms through professional learning groups. Techtrends. https://link.springer.com/article/10.1007/s11528-015-0019-y
*Early, K. N. (2023). Gamification of mathematics instruction: A quantitative study on student growth and proficiency in a suburban middle school (Order No. 30488148). Available from ProQuest Dissertations & Theses Global; Publicly Available Content Database. (2811112888). Retrieved from https://www.proquest.com/dissertations theses/gamification-mathematics-instruction-quantitative/docview/2811112888/se-2
Echeverría, A., García-Campo, C., Nussbaum, M., Gil, F., Villalta, M., Améstica, M., & Echeverría, S. (2011). A framework for the design and integration of collaborative classroom games. Computers and Education, 57(1), 1127-1136. https://doi.org/10.1016/j.compedu.2010.12.010
Ellison, T. L., Evans, J. N., & Pike, J. (2016). Minecraft, teachers, parents, and learning: What they need to know and understand. School Community Journal, 26(2), 25–43.
*Ferguson, T. L. K. (2014). Mathematics achievement with digital game-based learning in high school algebra 1 classes (Order No. 3613196). Available from ProQuest Dissertations & Theses Global. (1509130712). Retrieved from https://www.proquest.com/dissertations theses/mathematics-achievement-with-digital-game-based/docview/1509130712/se-2
*Foster, A., & Shah, M. (2015). The ICCE framework: Framing learning experiences afforded by games. Journal of Educational Computing Research, 51(4), 369–395. https://doi.org/10.2190/ec.51.4.a
Gao, F., Li, L. & Sun, Y. (2020). A systematic review of mobile game-based learning in STEM education. Education Tech Research Dev 68, 1791–1827 https://doi.org/10.1007/s11423-020-09787-0
Gee, J. P. (2005). Learning by design: Good video games as learning machines. E-Learning and Digital Media, 2, 5–16. https://doi.org/10.2304/elea.2005.2.1.5
*Gibbs, P. S. (2020). Game based learning: The effects of DragonBox 12+ on algebraic performance of middle school students (Order No. 28154801). Available from ProQuest Dissertations & Theses Global. (2511938530). Retrieved from https://www.proquest.com/dissertations-theses/game-based-learning-effects-dragonbox 12-on/docview/2511938530/se-2
Hanghoj, T., & Hautopp, H. (2016). Teachers’ pedagogical approaches to teaching with minecraft. Proceedings of the European Conference on Games Based Learning, 1, 265-272.
*Hieftje, K., Pendergrass, T., Kyriakides, T., Gilliam, W., & Fiellin, L. (2017). An evaluation of an educational video game on mathematics achievement in first grade students. Technologies, 5(2), 30. MDPI AG. Retrieved from http://dx.doi.org/10.3390/technologies5020030
*Hunt, J. H., Taub, M., Marino, M., Duarte, A., Bentley, B., Holman, K., & Kuhlman, A. (2023). Effects of game-enhanced supplemental fraction curriculum on student engagement, fraction knowledge, and STEM interest. Education Sciences, 13(7), 646. https://doi.org/10.3390/educsci13070646
Hung, CM., Huang, I. & Hwang, GJ. Effects of digital game-based learning on students’ self efficacy, motivation, anxiety, and achievements in learning mathematics. J. Comput. Educ. 1, 151–166 (2014). https://doi.org/10.1007/s40692-014-0008-8
Hwang, G., Chiu, L., & Chen, C. (2015). A contextual game-based learning approach to improving students’ inquiry-based learning performance in social studies courses. Computers & Education, 81, 13-25. https://doi.org/10.1016/j.compedu.2014.09.006
Johnson, C., & Mayer, R. (2010) Applying the self-explanation principle to multimedia learning in a computer-based game-like environment, Computers in Human Behavior, 26, 1246-1252. https://doi.org/10.1016/j.chb.2010.03.025
*Kebritchi, M., Hirumi, A., & Bai, H. (2010). The effects of modern mathematics computer
games on mathematics achievement and class motivation. Computers & Education,
55(2), 427-443. https://doi.org/10.1016/j.compedu.2010.02.007
*Kim, H., & Ke, F. (2016). Effects of game-based learning in an OpenSim-supported virtual environment on mathematical performance. Interactive Learning Environments, 25(4), 543–557. https://doi.org/10.1080/10494820.2016.1167744
*King, A. (2011). Using interactive games to improve math achievement among middle school students in need of remediation (Order No. 3468187). Available from ProQuest Dissertations & Theses Global; Publicly Available Content Database. (889144147). Retrieved from https://www.proquest.com/dissertations theses/using-interactive-games- improve-math-achievement/docview/889144147/se-2
Lamb, R. L., Annetta, L., Firestone, J., & Etopio, E. (2018). A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations. Computers in Human Behavior, 80, 158-167. https://doi.org/10.1016/j.chb.2017.10.040
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
*Martin, L. N. (2018). The effect of game-based learning on Title 1 elementary students’ math achievement (Order No. 10931295). Available from ProQuest Dissertations & Theses Global. (2099751018). Retrieved from https://www.proquest.com/dissertations theses/effect-game-based-learning-on-title-1 elementary/docview/2099751018/se-2
Nadolny, L., Alaswad, Z., Culver, D., & Wang, W. (2017). Designing with game-based learning: Game mechanics from middle school to higher education. Simulation and Gaming, 48, 814–831. https://doi.org/10.1177/1046878117736893
Naik, N. (2014). A Comparative Evaluation of Game-Based Learning: Digital or Non-Digital Games? Proceedings of the European Conference on Games Based Learning, 2, 437–445.
Nebel, S., Schneider, S., & Rey, G.D. (2016). Mining learning and crafting scientific experiments: A literature review on the use of minecraft in education and research. Journal of Educational Technology & Society, 19(2), 355-366.
*Pan, Y., & Ke, F. (2023). Effects of game-based learning supports on students’ math performance and perceived game flow. Educational Technology Research and Development, 71(2), 459–479. https://doi.org/10.1007/s11423-022-10183-z
Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of game-based learning. Educational Psychologist, 50, 258-283.
Qian, M., & Clark, K. R. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50-58. https://doi.org/10.1016/j.chb.2016.05.023
*Schenke, K., Rutherford, T., & Farkas, G. (2014). Alignment of game design features and state mathematics standards: Do results reflect intentions? Computers & Education, 76, 215–224. https://doi.org/10.1016/j.compedu.2014.03.019
Steinmayr, R., Meiǹer, A., Weideinger, A. F., & Wirthwein, L. (2014). Academic achievement
(pp. 9780199756810-0108). Oxford, UK: Oxford University Press.
*Shin, N., Sutherland, L. M., Norris, C. A., & Soloway, E. (2011). Effects of game technology on elementary student learning in mathematics. British Journal of Educational Technology, 43(4), 540–560. https://doi.org/10.1111/j.1467-8535.2011.01197.x
Steinmayr, Ricarda , Meißner, Anja , Weidinger, Anne F. , & Wirthwein, Linda (2014). Academic Achievement. obo in Education. doi: 10.1093/obo/9780199756810-0108
Sung, H., & Hwang, G. (2013) A collaborative game-based learning approach to improving students’ learning performance in science courses. Computers & Education, 63, 43-51. https://doi.org/10.1016/j.compedu.2012.11.019.
Susaeta, H., Jimenez, F., Nussbaum, M., Gajardo, I., And reu, J., & Villalta, M. (2010). From MMORPG to a classroom multiplayer presential role playing game. Educational Technology and Society, 13, 257-269.
Sweller, J. (2011). Cognitive load theory. Psychology of Learning and Motivation, 55, 37-76. https://doi.org/10.1016/b978-0-12-387691-1.00002-8
Talan, T., Doğan, Y., & Batdı, V. (2020). Efficiency of digital and non-digital educational games: A comparative meta-analysis and a meta-thematic analysis. Journal of Research on Technology in Education, 52(4), 474–514. https://doi.org/10.1080/15391523.2020.1743798
Tsai, Y-L & Tsai C-C. (2020). A meta-analysis of research on digital game-based science learning. J Comput Assist Learn.; 36: 280–294. https://doi.org/10.1111/jcal.12430
Tokac, U., Novak, E., & Thompson, C. G. (2019). Effects of game‐based learning on students’ Mathematics achievement: A meta‐analysis. Journal of Computer Assisted Learning, 35(3), 407-420. https://doi.org/10.1111/jcal.12347
U.S. Department of Education, Institute of Education Sciences, National Center For Education Statistics (2019). NAEP report card: NAEP 2019 Mathematics Assessments. https://www.nationsreportcard.gov/highlights/mathematics/2019/
U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics (2020). Health and STEM career expectations and science literacy achievement of U.S. 15-year-old students. Stats in brief. https://nces.ed.gov/pubs2020/2020034.pdf
von Gillern, S., & Alaswad, Z. (2016). Games and game-based learning in instructional design. The International Journal of Technologies in Learning, 23(4), 1–7. doi:10.18848/23270144/cgp/v23i04/1-7
Vu, P., & Feinstein, S. (2017). An exploratory multiple case study about using game-based learning in STEM classrooms. International Journal of Research in Education and Science, 3(2), 582-582. https://doi.org/10.21890/ijres.328087
Vygotsky, L. S. (1980). Mind in society: The development of higher mental processes. Cambridge, MA: Harvard University Press.
Wang, L., Chen, B., Hwang, G., Guan, J., & Wang, Y. (2022). Effects of digital game-based STEM education on students’ learning achievement: A meta-analysis. International Journal of STEM Education, 9(1). https://doi.org/10.1186/s40594-022-00344-0
Yang, Y. (2015). Virtual CEOs: A blended approach to digital gaming for enhancing higher order thinking and academic achievement among vocational high school students. Computers and Education, 81, 281-295
Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., Simeoni, Z., Tran, M., & Yukhymenko, M. (2012). Our princess is in another castle: A review of trends in serious gaming for education. Review of Educational Research, 82, 61-89. https://doi.org/10.3102/0034654312436980
Zhao, D., Muntean, C. H., Chis, A. E., & Muntean, G.-M. (2021). Learner attitude, educational background, and gender influence on knowledge gain in a serious games-enhanced programming course. IEEE Transactions on Education, 64(3), 308–316.
APPENDIX A
Coding Sheet
Coding Sheet
General Information
Coder Name (First and Last Name):
Date of coding:
Example: December 8, 2023
Identification of Studies
Study Code Number:
Title of study:
Author of the study:
APA citation of study:
Year of Publication:
What type of report (circle below choice)
a. Journal article
b. Book or book chapter
c. Dissertation
d. Private report
e. Conference paper
f. Other (specify written in here)
g. Not clearly stated
Was the study Peer Reviewed?:
1. Yes
2. No
3. Unknown/Not Stated
Setting and Participation Characteristics
Gender (Insert numbers, if reported).
1. Female ______
2. Male ______
3. No gender information given ______
School Level and sample size of the students in the study (Insert numbers, if reported).
1. PR (primary school Grades K-4)
2. MS (Middle school Grades 5-8)
3. HS (High school Grades 9-12)
4. M (Mixed includes grade levels from more than one category)
5. Not reporte
Game-Based Learning Characteristics
Game Difference
1. Digital game
2. Nondigital game
3. Study used both digital and nondigital games.
Game Used
1. Off the shelf/out of the box with no modification
2. Game was modified to meet the course standards and unit goals
3. Game was specifically designed/created by the authors/researchers for this study
4. Not reported
Did the study classify the game(s) used by any of the following terms:
1. Serious Game (SG)
2. Serious Educational Game (SEG)
3. Educational game
4. Simulation
5. No term was used
6. Other, state the game type term used _________
Game Platform
1. Game played on smartphone
2. Game played on a computer/laptop
3. Game played non digitally
4. Both
5. Other
6. Not reported
Program or Course Characteristics
What Mathematics content was being studied as part of the research?
_____________________________
Has game-based learning been used for this class or course before?
1. Yes
2. No
3. Not reported
How many sessions is the game-based activity used?
___________ (write out answer here or mark N/A if not stated)
How long (in minutes) is each session?
___________ (write out answer here or mark N/A if not stated)
Instructor Characteristics
Has this instructor used games before in the classroom?
1. Yes
2. No
3. Not clearly stated.
Did the teacher report the level of familiarity with using their type of game in the classroom?
Research Design Characteristics
Type of Design
1. Experimental
2. Quasi Experimental
3. Pretest only
4. Pre-Post comparison
5. Posttest only
6. Other
Sampling
1. Random selection
2. Random assignment
3. Convenience sample
How was data measured?
1. Standardized achievement test
2. A custom molded test
3. Observational comparison
4. Final product
5. Pretest/Posttest comparison
6. Survey data
7. Other
Who conducted the game-based learning activity?
1. Teacher
2. Researcher
3. Other (specify) _____________________
Were the validity and reliability of the lesson established?
1. Yes
2. No
3. No information provided
Moderator Variables Found
Did this game-based study report any comparison between grade levels?
1. Yes
2. No
3. Not Specific
Did this game-based study report any comparison between gender?
1. Yes
2. No
3. Not Specific
Did this game-based study report any comparison between game types?
1. Yes
2. No
3. Not Specific
Did this game-based study report any comparison between frequency of game-based instruction?
1. Yes
2. No
3. Not Specific
Effect Sizes and Statistical Information Reported
Page number where the effect size data is listed
Type of data effect size
1. Means and Standard Deviations
2. t test and Degrees of Freedom
3. F test and Degrees of Freedom
4. alpha and p value
5. Confidence intervals
6. Effect Size
7. Effect Size measure
8. Correlation coefficient (r)
9. Other
The effect difference favors
1. Control group//the traditional instruction
2. Game-based instruction
3. Other
Study Quality
Of the 7 sections above, assign 1 point if the section cannot be completed
How many points did this study receive?
Did this study state specifically if there were any limitations to their findings and if so, what
were they?