Introduction to Software Testing CSCI 5828: Foundations of Software Engineering Lecture 05 — 01/31/2012 © Kenneth M. Anderson, 2012 1 Goals • Provide introduction to fundamental concepts of software testing • Terminology • Testing of Systems • unit tests, integration tests, system tests, acceptance tests • Testing of Code • Black Box • Gray Box • White Box • Code Coverage © Kenneth M. Anderson, 2012 2 Testing • Testing is a critical element of software development life cycles • called software quality control or software quality assurance • basic goals: validation and verification • validation: are we building the right product? • verification: does “X” meet its specification? • where “X” can be code, a model, a design diagram, a requirement, … • At each stage, we need to verify that the thing we produce accurately represents its specification © Kenneth M. Anderson, 2012 3 Terminology • An error is a mistake made by an engineer • often a misunderstanding of a requirement or design specification • A fault is a manifestation of that error in the code • what we often call “a bug” • A failure is an incorrect output/behavior that is caused by executing a fault • The failure may occur immediately (crash!) or much, much later in the execution • Testing attempts to surface failures in our software systems • Debugging attempts to associate failures with faults so they can be removed from the system • If a system passes all of its tests, is it free of all faults? © Kenneth M. Anderson, 2012 4 No! • Faults may be hiding in portions of the code that only rarely get executed • “Testing can only be used to prove the existence of faults not their absence” or “Not all faults have failures” • Sometimes faults mask each other resulting in no visible failures! • this is particularly insidious • However, if we do a good job in creating a test set that • covers all functional capabilities of a system • and covers all code using a metric such as “branch coverage” • Then, having all tests pass increases our confidence that our system has high quality and can be deployed © Kenneth M. Anderson, 2012 5 Looking for Faults All possible states/behaviors of a system © Kenneth M. Anderson, 2012 6 Looking for Faults As you can see, its not very comprehensive Tests are a way of sampling the behaviors of a software system, looking for failures © Kenneth M. Anderson, 2012 7 One way forward? Fold The testing literature advocates folding the space into equivalent behaviors and then sampling each partition © Kenneth M. Anderson, 2012 8 What does that mean? • Consider a simple example like the greatest common denominator function • int gcd(int x, int y) • At first glance, this function has an infinite number of test cases • But lets fold the space • x=6 y=9, returns 3, tests common case • x=2 y=4, returns 2, tests when x is the GCD • x=3 y=5, returns 1, tests two primes • x=9 y=0, returns ?, tests zero • x=-3 y=9, returns ?, tests negative © Kenneth M. Anderson, 2012 9 Completeness • From this discussion, it should be clear that “completely” testing a system is impossible • So, we settle for heuristics • attempt to fold the input space into different functional categories • then create tests that sample the behavior/output for each functional partition • As we will see, we also look at our coverage of the underlying code; are we hitting all statements, all branches, all loops? © Kenneth M. Anderson, 2012 10 Continuous Testing • Testing is a continuous process that should be performed at every stage of a software development process • During requirements gathering, for instance, we must continually query the user, “Did we get this right?” • Facilitated by an emphasis on iteration throughout a life cycle • at the end of each iteration • we check our results to see if what we built is meeting our requirements (specification) © Kenneth M. Anderson, 2012 11 Testing the System (I) • Unit Tests • Tests that cover low-level aspects of a system • For each module, does each operation perform as expected • For method foo(), we’d like to see another method testFoo() • Integration Tests • Tests that check that modules work together in combination • Most projects on schedule until they hit this point (MMM, Brooks) • All sorts of hidden assumptions are surfaced when code written by different developers are used in tandem • Lack of integration testing has led to spectacular failures (Mars Polar Lander) © Kenneth M. Anderson, 2012 12 Testing the System (II) • System Tests • Tests performed by the developer to ensure that all major functionality has been implemented • Have all user stories been implemented and function correctly? • Acceptance Tests • Tests performed by the user to check that the delivered system meets their needs • In large, custom projects, developers will be on-site to install system and then respond to problems as they arise © Kenneth M. Anderson, 2012 13 Multi-Level Testing • Once we have code, we can perform three types of tests • Black Box Testing • Does the system behave as predicted by its specification • Grey Box Testing • Having a bit of insight into the architecture of the system, does it behave as predicted by its specification • White Box Testing • Since, we have access to most of the code, lets make sure we are covering all aspects of the code: statements, branches, … © Kenneth M. Anderson, 2012 14 Black Box Testing Input System Actual Output == ?? Spec Expected Output A black box test passes input to a system, records the actual output and compares it to the expected output Note: if you do not have a spec, then any behavior by the system is correct! © Kenneth M. Anderson, 2012 15 Results • if actual output == expected output • TEST PASSED • else • TEST FAILED • Process • Write at least one test case per functional capability • Iterate on code until all tests pass • Need to automate this process as much as possible © Kenneth M. Anderson, 2012 16 Black Box Categories • Functionality • User input validation (based off specification) • Output results • State transitions • are there clear states in the system in which the system is supposed to behave differently based on the state? • Boundary cases and off-by-one errors © Kenneth M. Anderson, 2012 17 Grey Box Testing • Use knowledge of system’s architecture to create a more complete set of black box tests • Verifying auditing and logging information • for each function is the system really updating all internal state correctly • Data destined for other systems • System-added information (timestamps, checksums, etc.) • “Looking for Scraps” • Is the system correctly cleaning up after itself • temporary files, memory leaks, data duplication/deletion © Kenneth M. Anderson, 2012 18 White Box Testing • Writing test cases with complete knowledge of code • Format is the same: input, expected output, actual output • But, now we are looking at • code coverage (more on this in a minute) • proper error handling • working as documented (is method “foo” thread safe?) • proper handling of resources • how does the software behave when resources become constrained? © Kenneth M. Anderson, 2012 19 Code Coverage (I) • A criteria for knowing white box testing is “complete” • statement coverage • write tests until all statements have been executed • branch coverage (a. edge coverage) • write tests until each edge in a program’s control flow graph has been executed at least once (covers true/false conditions) • condition coverage • like branch coverage but with more attention paid to the conditionals (if compound conditional, ensure that all combinations have been covered) © Kenneth M. Anderson, 2012 20 Code Coverage (II) • A criteria for knowing white box testing is “complete” • path coverage • write tests until all paths in a program’s control flow graph have been executed multiple times as dictated by heuristics, e., • for each loop, write a test case that executes the loop • zero times (skips the loop) • exactly one time • more than once (exact number depends on context) © Kenneth M. Anderson, 2012 21 A Sample Ada Program to Test 1 function P return INTEGER is 2 begin 3 X, Y: INTEGER; 4 READ(X); READ(Y); 5 while (X > 10) loop 6 X := X – 10; 7 exit when X = 10; 8 end loop; 9 if (Y < 20 and then X mod 2 = 0) then 10 Y := Y + 20; 11 else 12 Y := Y – 20; 13 end if; 14 return 2 ∗ X + Y; 15 end P; © Kenneth M. Anderson, 2012 22 P’s Control Flow Graph (CFG) 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F 12 © Kenneth M. Anderson, 2012 23 White-box Testing Criteria • Statement Coverage • Create a test set T such that • by executing P for each t in T • each elementary statement of P is executed at least once © Kenneth M. Anderson, 2012 24 All-Statements Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F 12 © Kenneth M. Anderson, 2012 25 All-Statements Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-statements-adequate test set: 12 © Kenneth M. Anderson, 2012 26 All-Statements Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-statements-adequate test set: 12 (X = 20, Y = 10) © Kenneth M. Anderson, 2012 27 All-Statements Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-statements-adequate test set: 12 (X = 20, Y = 10) (X = 20, Y = 30) © Kenneth M. Anderson, 2012 28 White-box Testing Criteria • Edge Coverage • Select a test set T such that • by executing P for each t in T • each edge of P’s control flow graph is traversed at least once © Kenneth M. Anderson, 2012 29 All-Edges Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F 12 © Kenneth M. Anderson, 2012 30 All-Edges Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-edges-adequate test set: 12 © Kenneth M. Anderson, 2012 31 All-Edges Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-edges-adequate test set: (X = 20, Y = 10) 12 © Kenneth M. Anderson, 2012 32 All-Edges Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-edges-adequate test set: (X = 20, Y = 10) 12 (X =15, Y = 30) © Kenneth M. Anderson, 2012 33 White-box Testing Criteria • Condition Coverage • Select a test set T such that • by executing P for each t in T • each edge of P’s control flow graph is traversed at least once • and all possible values of the constituents of compound conditions are exercised at least once © Kenneth M. Anderson, 2012 34 All-Conditions Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F 12 © Kenneth M. Anderson, 2012 35 All-Conditions Coverage of P 6 7 10 F T T T F T 2,3,4 5 9 9′ 14 F F Example all-conditions-adequate test set: 12 © Kenneth M.
