Designing Intelligent Tutors That Adapt to When Students Game the System Ryan Shaun Baker December, 2005 Doctoral Dissertation Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA USA Carnegie Mellon University, School of Computer Science Technical Report CMU-HCII-05-104 Thesis Committee: Albert T. Corbett, co-chair Kenneth R. Koedinger, co-chair Shelley Evenson Tom Mitchell Submitted 1n partialfulfillment of the requirements for the degree ofDoctor ofPhilosophy Copyright © 2005 by Ryan Baker. All rights reserved.
This research was sponsored in part by an NDSEG (National Defense Science and Engineering Graduate) Fellowship, and by National Science Foundation grant REC-043779. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies or endorsement, either express or implied, of the NSF, the ASEE, or the U. UMI Number: 3241593 INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. ® UMI UMI Microform 3241593 Copyright 2007 by ProQuest Information and Learning Company. All rights reserved.
This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P. Box 1346 Ann Arbor, MI 48106-1346 Carnegie Mellon DOCTORAL THESIS in the field of HUMAN-COMPUTER INTERACTION School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Designing Intelligent Tutors That Adapt to When Students Game the System Ryan Shaun Baker Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy LIC be [2-(S70oS Thesis Committee Co-Chair Date Thesis Committee Co-Chair Date «= : ke 2.Ÿ) l 2a¬j€-át Department Head Date APPROVED: -WA “⁄⁄ Ýí Dean L2 /12/24 Date Keywords: intelligent tutoring systems, educational data mining, human-computer interaction, gaming the system, quantitative field observations, Latent Response Models, intelligent agents Abstract Students use intelligent tutors and other types of interactive learning environments in a considerable variety of ways. In this thesis, I detail my work to understand, automatically detect, and re-design an intelligent tutoring system to adapt to a behaviorI term “gaming the system”.
Students who game the system attempt to succeed in the learning environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. Within this thesis, Ï present a set of studies aimed towards understanding what effects gaming has on learning, and why students game, using a combination of quantitative classroom observations and machine learning. In the course of these studies, I determine that gaming the system is replicably associated with low learning. I use data from these studies to develop a profile of students who game, showing that gaming students have a consistent pattern of negative affect towards many aspects of their classroom experience and studies.
Another part of this thesis is the development and training of a detector that reliably detects gaming, in order to drive adaptive support. In this thesis, | validate that this detector transfers effectively between 4 different lessons within the middle school mathematics tutor curriculum without re-training. This detector uses Latent Response Models (Maris 1995), combining labeled and unlabeled data at different-grain sizes, in order to train a model to accurately indicate both which students were gaming, and when they were gaming, and uses Fast Correlation-Based Filtering (Yu and Liu 2003) to efficiently search the space ofpotential models. The final part of this thesis is the re-design of an existing intelligent tutoring lesson to respond to gaming.
The re-designed lesson incorporates an animated agent (“Scooter the Tutor”) who indicates to the student and their teacher whether the student has been gaming recently, and gives students supplemental exercises, in order to offer the student another chance to learn material he/she gamed through. Scooter reduced the frequency of gaming by over half, and Scooter’s supplementary exercises were associated with substantially improved learning; Scooter appeared to have little effect on non-gaming students. Acknowledgements The list of people that I should thank for their help and support in completing this dissertation would fill an entire book. Here, instead, is an incomplete list of some of the people I would like to thank for their help, support, and suggestions.
Angela Wagner, Ido Roll, Mike Schneider, Steve Ritter, Tom McGinnis, and Jane Kamneva assisted in essential ways with the implementation and administration of the studies presented in this dissertation. None of the studies presented here could have occurred without the support of Jay Raspat, Meghan Naim, Dina Crimone, Russ Hall, Sue Cameron, Frances Battaglia, and Katy Getman, in welcoming me into their classrooms. The ideas presented in this dissertation were refined through conversations with Ido Roll, Santosh Mathan, Neil Heffernan, Aatish Salvi, Dan Baker, Cristen Torrey, Darren Gergle, Irina Shklovski, Peter Scupelli, Aaron Bauer, Brian Junker, Joseph Beck, Jack Mostow, Carl diSalvo, and Vincent Aleven. My committee members, Shelley Evenson and Tom Mitchell, helped to shape this dissertation into its present form, teaching me a great deal about design and machine learning in the process.
My advisors, Albert Corbett and Kenneth Koedinger, were exceptional mentors, and have guided me for the last five years in learning how to conduct research effectively, usefully, and ethically — I owe an immeasurable debt to them. Finally, I would like to thank my parents, Sam and Carol, and my wife, Adriana. Their support guided me when the light at the end of the dissertation seemed far. Table of Contents Introduction II Gaming the System and Learning 11 IH Detecting Gaming 20 IV Understanding Why Students Game 42 Adapting to Gaming 55 Conclusions and Future Work 88 References 92 Appendices A Cognitive Tutor Lessons 97 B Learning Assessments 104 C Gaming Detectors 118 Chapter One Introduction In the last twenty years, interactive learning environments and computerized educational supports have become a ubiquitous part of students’ classroom experiences, in the United States and throughout the world.
Many such systems have become very effective at assessing and responding to differences in student knowledge and cognition (Corbett and Anderson 1995; Martin and vanLehn 1995; Arroyo, Murray, Woolf, and Beal 2003; Biswas et al 2005). Systems which can effectively assess and respond to cognitive differences have been shown to produce substantial — and statistically significant — learning gains, as compared to students in traditional classes (cf. Koedinger, Anderson, Hadley, and Mark 1997; vanLehn et al 2005). However, even within classes using interactive learning environments which have been shown to be effective, there is still considerable variation in student learning outcomes, even when each student’s prior knowledge is taken into account.
The thesis of this dissertation is that a considerable amount of this variation comes from differences in how students choose to use educational software, that we can determine which behaviors are associated with poorer learning, and that we can develop systems that can automatically detect and respond to those behaviors, in a fashion that improves student learning. In this dissertation, I present results showing that one way that students use educational software, gaming the system, is associated with substantially poorer learning — much more so, in fact, than if the student spent a substantial portion of each class ignoring the software and talking off-task with other students (Chapter 2). J then develop a model which can reliably detect when a student is gaming the system, across several different lessons from a single Cognitive Tutor curriculum (Chapter 3). Using a combination of the gaming detector and attitudinal questionnaires, I compile a profile of the prototypical gaming student, showing that gaming students differ from other students in several respects (Chapter 4).
I next combine the gaming detector and profile of gaming students, in order to re-design existing Cognitive Tutor lessons to address gaming. My re-design introduces an interactive agent, Scooter the Tutor, who signals to students (and their teachers) that he knows that the student is gaming, and gives supplemental exercises targeted towards the material students are missing by gaming (Chapter 5). Scooter substantially decreases the incidence of gaming, and his exercises are associated with substantially better learning. In Chapter 6, I discuss the larger implications of this dissertation, advancing the idea of interactive learning environments that effectively adapt not just to differences in student cognition, but differences in student choices.
Gaming the System I define “Gaming the System” as attempting to succeed in an educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. Gaming strategies are seen by teachers and outsiders as misuse of the software the student is using or system that the student is participating in, but are distinguished from cheating in that gaming does not violate explicit rules of the educational setting, as cheating does. In fact, in some situations students are encouraged to game the system — for instance, several test preparation companies teach students to use the structure of how SAT questions are designed in order to have a higher probability of guessing the correct answer. Cheating on the SAT, by contrast, is not recommended by test preparation companies.
Gaming the System occurs in a wide variety of different educational settings, both computerized and offline. To cite just a few examples: Arbreton (1998) found that students ask teachers or teachers’ aides to give them answers to math problems before attempting the problems themselves. Magnussen and Misfeldt (2004) have found that students take turns intentionally making errors in collaborative educational games in order to help their teammates obtain higher scores; gaming the system has also been documented in other types of educational games (Klawe 1998; Miller, Lehman, and Koedinger 1999). Cheng and Vassileva (2005) have found that students post irrelevant information — in large quantities — to newsgroups in online courses which are graded based on participation.
Within intelligent tutoring systems, gaming the system has been particularly well-documented. Wood and Wood (1999) found that students quickly and repeatedly ask for help until the tutor gives the student the correct answer, a finding replicated by Aleven and Koedinger (2000). Mostow and his colleagues (2002) found in a reading tutor that students often avoid difficulty by re-reading the same story over and over. Aleven and his colleagues (1998) found, in a geometry tutor, that students learn what answers are most likely to be correct (such as numbers in the givens, or 90 or 180 minus one of those numbers), and try those numbers before thinking through a problem.
Murray and vanLehn (2005) found that students using systems with delayed hints (a design adopted by both Carnegie Learning (Aleven 2001) and by the AnimalWatch project (Beck 2005) as a response to gaming) intentionally make errors at high speed in order to activate the software's proactive help. Within the intelligent tutoring systems we studied, we primarily observed two types of gaming the system: 1. quickly and repeatedly asking for help until the tutor gives the student the correct answer (as in Wood and Wood 1999; Aleven and Koedinger 2000) 2. inputting answers quickly and systematically.
For instance, entering 1,2,3,4,. or clicking every checkbox within a set of multiple-choice answers, until the tutor identifies a correct answer and allows the student to advance. In both of these cases, features designed to help a student learn curricular material via problem- solving were instead used by some students to solve the current problem and move forward within the curriculum. The Cognitive Tutor Classroom All of the studies thatI will present in this dissertation took place in classes using Cognitive Tutor software (Koedinger, Anderson, Hadley, and Mark 1995).
In these classes, students complete mathematics problems within the Cognitive Tutor environment. The problems are designed so as to reify student knowledge, making student thinking (and misconceptions) visible.
