Practical Language Testing Glenn Fulcher For all the inspiring teachers I have been lucky enough to have and especially Revd Ian Robins Who knows where the ripples end? First published in Great Britain in 2010 by Hodder Education, An Hachette UK Company, 338 Euston Road, London NW1 3BH © 2010 Glenn Fulcher All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronically or mechanically, including photocopying, recording or any information storage or retrieval system, without either prior permission in writing from the publisher or a licence permitting restricted copying. In the United Kingdom such licences are issued by the Copyright Licensing Agency: Saffron House, 6–10 Kirby Street, London EC1N 8TS. Hachette UK’s policy is to use papers that are natural, renewable and recyclable products and made from wood grown in sustainable forests.
The logging and manufacturing processes are expected to conform to the environmental regulations of the country of origin. The advice and information in this book are believed to be true and accurate at the date of going to press, but neither the author nor the publisher can accept any legal responsibility or liability for any errors or omissions. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress ISBN: 978 0 340 984482 1 2 3 4 5 6 7 8 9 10 Cover Image © Anthony Bradshaw/Photographer’s Choice RF/Getty Images Typeset in 10 on 13pt Minion by Phoenix Photosetting, Chatham, Kent Printed and bound in Great Britain by Antony Rowe, Chippenham, Wilts What do you think about this book? Or any other Hodder Education title? Please send your comments to educationenquiries@hodder.uk http://www.com Contents Acknowledgements vii List of figures ix List of tables xi Preface xiii 1 Testing and assessment in context 1 1. Tests in educational systems 4 3.
Testing and society 8 6. The politics of language testing 12 8. Historical interlude II 15 9. Professionalising language education and testing 17 10.
Validity 19 Activities 21 2 Standardised testing 31 1. Testing as science 32 3. What’s in a curve? 35 4. The curve and score meaning 36 5.
Putting it into practice 37 6. Test scores in a consumer age 42 7. Testing the test 44 8. Living with uncertainty 54 11.
Reliability and test length 57 12. Relationships with other measures 57 13. Measurement 59 Activities 60 iv Contents 3 Classroom assessment 67 1. Life at the chalk-face 67 2.
Assessment for Learning 68 3. Self- and peer-assessment 70 4. Assessment and second language acquisition 77 7. Criterion-referenced testing 79 8.
Some thoughts on theory 87 Activities 90 4 Deciding what to test 93 1. The test design cycle 93 2. Where do constructs come from? 102 4. Models of communicative competence 105 5.
From definition to design 118 Activities 120 5 Designing test specifications 127 1. What are test specifications? 127 2. Specifications for testing and teaching 134 3. A sample detailed specification for a reading test 139 4.
Target language use domain analysis 149 7. Moving back and forth 154 Activities 155 6 Evaluating, prototyping and piloting 159 1. Investigating usefulness and usability 159 2. Evaluating items, tasks and specifications 159 3.
Guidelines for multiple-choice items 172 4. Operational item review and pre-testing 188 Activities 190 7 Scoring language tests 197 1. Scoring items 197 Contents v 2. Scoring constructed response tasks 208 4.
Corrections for guessing 218 6. Avoiding own goals 219 Activities 220 8 Aligning tests to standards 225 1. It’s as old as the hills 225 2. The definition of ‘standards’ 225 3.
The uses of standards 226 4. Unintended consequences revisited 228 5. Using standards for harmonisation and identity 229 6. How many standards can we afford? 231 7.
Performance level descriptors (PLDs) and test scores 233 8. Some initial decisions 234 9. Standard-setting methodologies 236 10. Evaluating standard setting 241 11.
The special case of the CEFR 244 13. You can always count on uncertainty 248 Activities 250 9 Test administration 253 1. Controlling extraneous variables 254 3. Standardised conditions and training 259 5.
Planned variation: accommodations 262 6. Unplanned variation: cheating 264 7. Scoring and moderation 267 8. Data handling and policy 268 9.
Reporting outcomes to stakeholders 269 10. The expense of it all 272 Activities 274 10 Testing and teaching 277 1. The things we do for tests 277 2. Washback and content alignment 282 4.
Preparing learners for tests 288 5. Selecting and using tests 292 6. The gold standard 295 vi Contents Activities 298 Epilogue 300 Appendices 301 Glossary 319 References 325 Index 343 Acknowledgements I am deeply indebted to the Leverhulme Trust (www.uk), which awarded me a Research Fellowship in 2009 in order to carry out the research required for this book, and funded study leave to write it. The generosity of the Trust provided the time and space for clear thinking that work on a text like this requires.
The University of Leicester was extremely supportive of this project, granting me six months’ study leave to work entirely on the book. I would also like to thank staff in the School of Education for help and advice received while drafting proposals and work schedules. I am grateful to the people, and the institutions, who have given me permission to use materials for the book. Special thanks are due to Professor Yin Jan of Shanghai Jiao Tong University, and Chair of the National College English Testing Committee of the China Higher Education Department.
Her kindness in providing information about language testing in China, as well as samples of released tests, has enriched this book. I have always been inspired by my students. While I was working on the develop- ment of Performance Decision Trees (see Chapter 7), Samantha Mills was working on a dissertation in which she developed and prototyped a task for use in assessing service encounter communication in the tourist industry. In this book the two come together to illustrate how specifications, tasks and scoring systems, can be designed for specific purpose assessment.
I am very grateful to Samantha for permission to reproduce sec- tions of her work, particularly in Chapters 5 and 6. Test design workshops can be great fun; and they are essential when brainstorming new item types. I have run many workshops of this kind, and the material used to illus- trate the process of item evaluation in Chapter 6 is taken from a workshop conducted for Oxford University Press (OUP). I am grateful to OUP, particularly Simon Beeston and Alexandra Miller, for permission to use what is normally considered to be confi- dential data.
The book presents a number of statistical tools that the reader can use when design- ing or evaluating tests. All of the statistics can be calculated using packages such as SPSS, or online web-based calculators. However, I believe that it is important for people who are involved in language testing to understand how the basic statistics can be calcu- lated by hand. My own initial statistical training was provided by Charles Owen at the University of Birmingham, and I have always been grateful that he made us do calcula- tions by hand so that we could ‘see’ what the machine was doing.
However, calculation by hand can always lead to errors. After a while, the examples in the text became so familiar that I would not have been able to spot any errors, no matter how glaring. I am therefore extremely grateful to Sun Joo Chung of the University of Illinois at Urbana- Champaign for the care with which she checked and corrected these parts of the book. viii Acknowledgements The content of the book evolved over the period during which it was written.
This is because it is based on a research project to discover the language testing needs of teachers and students of language testing on applied linguistics programmes. A survey instrument was designed and piloted, and then used in the main study. It was deliv- ered through the Language Testing Resources website (http://languagetesting.info), and announced on the language testing and applied linguistics discussion lists. It was also supported by the United Kingdom’s Subject Centre for Languages, Linguistics and Area Studies.
The respondents came from all over the world, and from many different back- grounds. Each had a particular need, but common themes emerged in what they wished to see in a book on practical language testing. The information and advice that they provided has shaped the text in many ways, as my writing responded to incoming data. My thanks, therefore, to all the people who visited my website and spent time complet- ing the survey.
My thanks are also due to Fred Davidson, for a continued conversation on language testing that never fails to inspire. To Alan Davies and Bernard Spolsky, for their help and support; and for the constant reminder that historical context is more important than ever to understanding the ‘big picture’. And to all my other friends and colleagues in the International Language Testing Association (ILTA), who are dedicated to improving language testing practice, and language testing literacy. Every effort has been made to obtain the necessary permission with reference to copyright material.
The publishers apologise if inadvertently any sources remain unacknowledged and will be glad to make the necessary arrangements at the earliest opportunity. Finally, acknowledgements are never complete with recognition for people who have to suffer the inevitable lack of attention that writing a book generates. Not to mention the narrowing of conversational topics. My enduring thanks to Jenny and Greg for their tolerance and encouragement.1 Jeremy Bentham’s Panopticon in action 2.1 Distribution of scores in typical army groups, showing value of tests in identi- fication of officer material 2.2 The curve of normal distribution and the percentage of scores expected between each standard deviation 2.3 A histogram of scores 2.4 The curve of normal distribution with raw scores for a particular test 2.5 The curve of normal distribution with the meaning of a particular raw score 2.6 A scatterplot of scores on two administrations of a test 2.7 Shared variance between two tests at r2 = .1 Continuous assessment card 3.2 An item from an aptitude test 3.3 A negatively skewed distribution 4.1 The test design cycle 4.2 The levels of architectural documentation 4.3 Language, culture and the individual 4.4 Canale’s expanded model of communicative competence 4.5 Bachman’s components of language competence 4.6 The common reference levels: global scale 5.1 Forms and versions 5.2 Popham’s (1978) five-component test specification format 7.1 Marking scripts in 1917 7.2 The IBM 805 multiple-choice scoring machine 7.3 Example of a branching routine 7.4 An Item–person distribution map 7.5 EBB for communicative effectiveness in a story retell 7.6 A performance decision tree for a travel agency service encounter 8.1 The distributions of three groups of test takers 9.1 An interlocutor frame 10.1 An observation schedule for writing classes This page intentionally left blank Tables 2.2 Proportion of test takers from two groups answering individual items correctly 2.3 Calculating a correlation coefficient between two sets of scores 2.4 Item variances for the linguality test 2.5 Descriptive statistics for two raters, rating ten essays 2.6 Descriptive statistics for combined scores 2.7 Correlations of group with individual linguality test scores 2.8 The relation between the two tests 3.2 Results of a reading test 6.2 Responses of 30 students to items 67–74 6.4 Means for p and q for item 70 7.1 Correlations between human and machine scores on PhonePass SET-10 8.2 Classifications of students into three levels by two judges 9.1 Observed values by conditions and outcomes on a language test 9.2 Expected values by outcomes on a language test 9.3 Critical values of chi-square 10.1 Standards for formative writing, language arts, grades 9–12 10.