ABSTRACT Title of dissertation: Developing Cost-Effective Model-Based Techniques for GUI Testing Qing Xie, Doctor of Philosophy, 2006 Dissertation directed by: Professor Atif Memon Department of Computer Science University of Maryland, College Park Most of today’s software users interact with the software through a graphical user interface (GUI), which constitutes as much as 45-60% of the total code. The correct- ness of the GUI is necessary to ensure the correctness of the overall software. Although GUIs have become ubiquitous, testing GUIs for functional correctness has remained a ne- glected research area. Existing GUI testing techniques are extremely resource intensive primarily because GUIs have very large input spaces and evolve frequently.
This dis- sertation overcomes the limitations of existing techniques by developing a process with supporting models, techniques, and tools for continuous integration testing of evolving GUI-based applications. The key idea of this process is to create three concentric testing loops, each with specific GUI testing goals, resource usage, and targeted feedback. The innermost fully automatic loop called crash testing operates on each code change of the GUI software. The second semi-automated loop called smoke testing operates on each day’s GUI build.
The outermost loop called comprehensive GUI testing is executed after a major version of the GUI is available. The primary enablers of this process, also devel- oped in this dissertation, include an abstract model of the GUI and a set of model-based techniques for test-case generation, test oracle creation, and continuous GUI testing. The model and techniques were obtained by studying GUI faults, interactions between GUI events, and why certain event interactions lead to faults. The continuous testing process and associated techniques are shown to be useful, via several large experiments involving millions of test cases, on both in-house and open-source GUI applications.
Developing Cost-Effective Model-Based Techniques for GUI Testing by Qing Xie Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2006 Advisory Committee: Professor Atif Memon, Chair/Advisor Professor Victor Basili Professor Rance Cleaveland Professor Michel Cukier Professor Adam Porter Professor Gang Qu UMI Number: 3241432 UMI Microform 3241432 Copyright 2007 by ProQuest Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P.
Box 1346 Ann Arbor, MI 48106-1346 c Copyright by Qing Xie 2006 ACKNOWLEDGMENTS First and foremost I would like to thank my advisor, Professor Atif Memon for giving me an invaluable opportunity to work on challenging and extremely interesting projects over the past several years. He has always made himself available for help and advice and there has never been an occasion when I have knocked on his door and he has not given me time. It has been a pleasure to work with and learn from such an extraordinary individual. This experience will benefit me forever.
Thanks are due to Professor Victor Basili, Professor Rance Cleaveland, Professor Michel Cukier, Professor Adam Porter, and Professor Gang Qu for agreeing to serve on my thesis committee and for sparing their invaluable time reviewing the manuscript. I thank all my teachers in schools, colleges, and universities whose dedication and hard work helped lay the foundation for this work. My colleagues at the Software Testing Group have enriched my graduate life in many ways and deserve a special mention. My interaction with Bin Gan, Xun Yuan, Jaymie Strecker, Cyntrica Eaton, Adithya Nagarajan, Ishan Banerjee, and Lorin Hochstein has been very fruitful.
I would also like to acknowledge help and support from some of the staff members in the department. I owe my deepest thanks to my family - my mother, father, and brother who have always stood by me and guided me through my career; my husband, Feng, for always ii being there to support me and be constant source of encouragement during my Ph.D; my daughter, Katie, for bringing me endless happiness. I would like to acknowledge financial support from the NSF grant CCF-0447864 and the Office of Naval Research grant N00014-05-1-0421, for all the projects discussed herein. Last, but far from least, I want to express my thanks to all the people who have made this thesis possible and because of whom my graduate experience has been one that I will cherish forever.
iii TABLE OF CONTENTS List of Tables vi List of Figures vii 1 Introduction 1 1.1 What is a GUI? .2 GUI Testing Process .3 Challenges of GUI Testing .4 Existing Approaches and their Limitations .5 A New Continuous GUI Testing Process .6 Structure of the Dissertation. 12 2 Background and Related Work 13 2.2 Event-flow Graphs .2 Test Case Generation .2 Model-based Approaches .4 Test Coverage Criteria .6 Rapid Feedback-based QA mechanisms. 32 3 A Continuous GUI Testing Process 33 3.4 Instantiating the Loops .1 Minimized Effective Event Context .2 Pilot Study - Understanding the MEEC .2 Step 1: Study Subjects .3 Step 2: Fault Seeding .4 Step 3: Test-Case Generation .5 Step 5: Studying Predecessor Events .3 Dissecting the MEEC .4 Threats to Validity .5 Event-Interaction Graph .6 Crash Test Cases .7 Feasibility Studies - Evaluating Crash Test Cases .1 Feasibility Study - Crash Testing on TerpOffice Applications .2 Feasibility Study - Crash Testing for Open-Source Applications .1 Designing Different Test Oracles .2 Evaluating the GUI Test Oracles .2 Modeling Cost and Fault Detection Effectiveness. 113 6 Comprehensive GUI Testing 115 6.1 Experiment - Studying the Characteristics of a “Good” Comprehensive Test Suite .3 Part 1: Effect of Test Suite Size .4 Part 2: Effect of Test Case Length .5 Part 3: Effect of Event Composition .2 Experiment - Developing Test Oracles for Comprehensive Testing.
136 7 Summary and Future Work 138 7.1 Summary of Contributions. 142 Bibliography 146 v LIST OF TABLES 4.2 Classes of Seeded Faults .3 Seeded Faults Classified by Functionality .4 Regular Expression Table .5 Sizes of Event-Interactions Graph .6 Number of Test Cases Generated for Each Version of Each Application .7 Number of Crashes Detected for Each Version of Each Application .1 The Data Table Cleanup Steps .2 Friedman Test Results .3 Wilcoxon Test Results .4 Average Number of Widget Comparisons Per Test Case. 110 vi LIST OF FIGURES 1.1 Different Loops of Continuous GUI Testing .1 (a) Open GUI, (b) its Partial State .2 Example of an Event-Flow Graph .3 (a) A Simple GUI and (b) Example of a JFCUnit Test Case .4 An Overview of the GUI Oracle .1 Activities to Support Continuous GUI Testing .1 Total Number of Event Sequences .4 MEEC for TerpCalc .5 MEEC for TerpPaint .6 MEEC for TerpSpreadSheet .7 MEEC for TerpWord .8 EIG for the EFG of Figure 2.9 Generate Event-Inteaction Graph from Event-Flow Graph .10 Total Execution Time .11 Number of Software Crashes .12 Number of Crash-Causing Bugs .13 Number of Bugs vs. Number of Test Cases .14 Effectiveness of the Rotating Algorithm for TerpCalc .15 Effectiveness of the Rotating Algorithm for TerpPaint .16 Effectiveness of the Rotating Algorithm for TerpPresent .17 Bug History Over Versions .1 Oracle Information for the Cancel Event .2 Oracle Procedure Algorithm .3 L1 Compares Widget-Relevant Triples after Each Event in the Test Case .4 Distribution of Values by Test Oracle .5 Histogram for TerpPresent .6 Histogram for TerpWord .7 Histogram for TerpPaint .8 Histogram for TerpSpreadSheet .9 Values for All Test Cases .10 Position Where the Fault is Detected vs.
Oracle for (a) TerpPresent, (b) TerpWord, and (c) TerpSpreadSheet .1 Event Distribution for Each Application .2 Fault Detection Effectiveness vs. Test Suite Size for TerpCalc .3 Fault Detection Effectiveness vs. Test Suite Size for TerpWord .4 Fault Detection Effectiveness vs. Test Suite Size for TerpSpreadsheet .5 Fault Detection Effectiveness vs.
Test Suite Size for TerpPaint .6 Fault Detection Effectiveness vs. Test Case Length for TerpCalc .7 Fault Detection Effectiveness vs. Test Case Length for TerpWord .8 Fault Detection Effectiveness vs. Test Case Length for TerpSpreadsheet .9 Fault Detection Effectiveness vs.
Test Case Length for TerpPaint .10 New Faults Detected with Length Increase .11 Number of Failures .12 Errors for TerpPaint .13 Errors for TerpSpreadSheet .14 Errors for TerpWord .15 Event Classes and Error Types .16 Error detection of .17 Time Required for . 136 ix Chapter 1 Introduction Testing is widely recognized as a key quality assurance (QA) activity in the software development process. Although research in testing has received considerable attention in the last two decades [22], testing of graphical user interfaces (GUIs), which constitute as much as 45-60% of the total software code [45], has remained until recently, a neglected research area [32]. Because GUI software has become nearly ubiquitous, neglecting the quality of GUI software has the potential to have a negative impact on all of today’s software.
A software with a GUI front-end consists of two parts : (1) the underlying code that implements the “business logic” and (2) the GUI front-end that facilitates user interaction with the underlying code. A software user interacts with the GUI by performing events, such as button clicks, menu selections, and text inputs. The GUI uses the input events to interact with the underlying code via messages and method invocations. During GUI testing, test cases, modeled as sequences of events are executed on the GUI and its output is compared to an “expected output.” The goal of GUI testing is to reveal GUI faults (defined as one that manifests itself on the visible GUI at some point of time during the software’s execution).
Several researchers have exploited the event-driven nature of GUIs to develop auto- mated model-based GUI testing techniques (e. However, these techniques have not been adopted by GUI testers because of several problems: (1) the models are expensive to obtain (ex- cept for event-flow graphs); they are typically created manually, (2) the number of per- mutations of all possible GUI interactions (event sequences) with the user is enormous; these techniques test the GUI for a small sub-space of user interactions; it remains unclear whether testing this sub-space reveals any GUI faults, and (3) GUIs are typically devel- oped using agile processes, which are known for their simple planning, short iterations, and are driven by frequent customer feedback. It becomes expensive to update the models and test artifacts (e., test cases, test oracles) during frequent software/GUI updates. Moreover, because modern software is typically developed by multiple program- mers, another GUI testing challenge largely ignored by existing techniques is that the programmers are likely to “break” the GUI software during their local code updates.
Programmers are generally unwilling and, due to limited resources, unable to setup an expensive GUI testing process for each update. If left undetected, the cascading effect of these updates may lead to integration faults that cause substantial delays during GUI integration testing. The research presented in this dissertation overcomes the limitations of existing techniques. Specifically, the contributions of this research include: the development of new cost-effective, automated GUI testing techniques that are applicable to rapidly evolving GUI software, development of new GUI models that are inexpensive to obtain and maintain, demonstration of the fault detection effectiveness of the new techniques, and 2 development of a continuous GUI testing process that targets feedback to specific developers.
The remainder of the chapter outlines the steps necessary for GUI testing and the challenges that GUI testers face for each step, followed by a discussion of existing GUI testing techniques and their limitations, and a high-level overview of the research pre- sented in this dissertation.1 What is a GUI? Most of today’s software users interact with the software through a GUI. The user typically uses a mouse and a keyboard to interact with GUI widgets. Widgets of a GUI include elements such as windows, pull-down menus, buttons, scroll bars, text boxes, and icons. The software user performs events on these widgets, such as clicking a button, selecting a menu item, and typing in a text box.