VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES NGUYỄN THỊ MAI HỮU AN INVESTIGATION INTO THE COGNITIVE VALIDITY OF THE SPEAKING SECTION OF THE VIETNAMESE STANDARDIZED TEST OF ENGLISH PROFICIENCY (VSTEP.3-5) Nghiên cứu giá trị xác thực đối với quá trình tư duy của thí sinh khi thi phần thi nói bài thi chuẩn hóa đánh giá năng lực tiếng Anh từ bậc 3 đến bậc 5 theo Khung năng lực ngoại ngữ 6 bậc dùng cho Việt Nam (VSTEP.3-5) A thesis submitted in fulfillment of the requirements for the Degree of Doctor of Philosophy Major: English Language Teaching Methodology Code: 9140231.01 Hanoi, 2022 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF POST-GRADUATE STUDIES NGUYỄN THỊ MAI HỮU AN INVESTIGATION INTO THE COGNITIVE VALIDITY OF THE SPEAKING SECTION OF THE VIETNAMESE STANDARDIZED TEST OF ENGLISH PROFICIENCY (VSTEP.3-5) Nghiên cứu giá trị xác thực đối với quá trình tư duy của thí sinh khi thi phần thi nói bài thi chuẩn hóa đánh giá năng lực tiếng Anh từ bậc 3 đến bậc 5 theo Khung năng lực ngoại ngữ 6 bậc dùng cho Việt Nam (VSTEP.3-5) A thesis submitted in fulfillment of the requirements for the Degree of Doctor of Philosophy Major: English Language Teaching Methodology Code: 9140231.01 Supervisors: Professor Hoa Nguyen Professor Fred Davidson Hanoi, 2022 STATEMENT OF AUTHORSHIP I certify my authority to the research entitled Submitted in fulfillment of the requirements for the Degree of Doctor of Philosophy An investigation into the cognitive validity of the speaking section of the Vietnamese Standardized Test of English Proficiency (VSTEP.3-5) Except where the reference is indicated, no other person‘s work has been used without the acknowledgement in the text of the thesis. Hanoi, June 2022 Nguyễn Thị Mai Hữu ACKNOWLEDGEMENT First and foremost, I would like to express my great gratitude to Professor Nguyen Hoa and Professor Fred Davidson, my supervisors at the University of Languages and International Studies, for their valuable guidance and comments on my dissertation, without which I would have encountered much difficulty in accomplishing it. My deep thanks also go to Dr. Karen Ashton, my supervisor when I worked as a visiting scholar at Massey University, who guided me much when I was at Massey University to do research relating to this dissertation.
Without the time I was at Massey University, I could hardly be able to complete the theoretical framework, research methodology, and data analysis parts of this dissertation. Next, my deep and special thanks go to Dr. Huynh Anh Tuan, Dean of the Faculty of Graduate and Post-graduate Studies, whose supports and encouragements were of crucial importance to me during the time I conducted the thesis. I am profoundly grateful to the members of the Faculty of Graduate and Post-graduate Studies: Tran Thi Phuong Nhung, Trinh Hong Nam, Nguyen Thi Ai Anh, Ta Bich Dao, whose help and encouragements were of great contribution to the completion of the dissertation.
I am also grateful to my colleges Nguyen Thi Hong Nhung, Ta Bich Lien, Nguyen Thi Quynh Yen, Nguyen Xuan Khanh, Nguyen Thi Ngoc Quynh, Tran Thi Hien for their supports to me when I conducted this dissertation. Also, I am much obliged to the Vietnamese participants who were willing to help me with the data collection, especially those who joined the stimulated recall interviews. They were so nice and patient in listening to the audio of their own speaking performance, and in answering all my questions. Last but not least, the love, supports and back-up of my husband, my parents, brothers and sisters, and my children provided me with intense energy, without which I could hardly complete this dissertation.
ii ABSTRACT This study was designed to craft a validation study for the Vietnamese Standardized Test of English Proficiency (VSTEP.3-5) in relation to the speaking cognitive processing. For such an aim, the cognitive processing model applied in the language domain of speaking was studied following the socio-cognitive framework developed by Weir (2005). Validation evidence was then be elicited via various methods of documents analysis, focus group interviews, survey questionnaires, and stimulated recall interviews. The techniques were used to study the extent to which the cognitive processes that the VSTEP.3-5 speaking test takers are supposed to have in test conditions resemble the ones in non-test conditions.
These methods were designed for the speaking cognitive processes that contribute to the target construct of the test, which is reflected in the input, the test format, and the test tasks (Field, 2013). The results of the study are that validity evidence was found to establish the cognitive validity of the VSTEP speaking section following the speaking cognitive processes introduced by Field (2013) in the socio-cognitive framework developed by Weir (2005), the cognitive demands were calibrated in the speaking rating scale to reflect the proficiency levels of the descriptors of the scale, and the cognitive processes that the test takers went through in the test condition resemble the ones in non-test conditions, which provide significant implications for the commissioning of the VSTEP.3-5 speaking tasks, the calibration of the VSTEP.3-5 rating scale, and the administration of the VSTEP. iii CONTENTS STATEMENT OF AUTHORSHIP. iii LIST OF TABLES.
viii LIST OF FIGURES. ix CHAPTER I: INTRODUCTION. Objectives of the study. Significance of the study.
Scope of the study. Organization of the study. 3 CHAPTER II: LITERATURE REVIEW. The concept of validity.
Validation in language testing. The concept of cognitive validity.4 Cognitive validity and testing speaking. Cognitive validity in assessing speaking. A framework for establishing the cognitive validity in testing speaking.
The socio-cognitive framework in different studies. 33 CHAPTER III: THE VIETNAMESE STANDARDIZED TEST OF ENGLISH PROFICIENCY VSTEP. Decision to provide the VSTEP. The National Foreign Languages Project.3-5 test taker characteristics.
The construct of the VSTEP.3-5 item writer training .3-5 marking, grading and reporting results. 52 CHAPTER IV: RESEARCH METHODOLOGY. Selection of participants. Data collection instruments .3-5 test format and related documents .3-5 speaking test form.
Stimulated-recall interviews. 61 CHAPTER V: THE COGNITIVE PROCESSES SUPPOSEDLY REPRESENTED IN THE VSTEP. The cognitive processes presented in the VSTEP.3-5 test development reports and related documents. Focus group interviews.
Phonetic encoding, articulation. 101 CHAPTER VI: THE CALIBRATION OF COGNITIVE DEMANDS IN THE VSTEP.3-5 SPEAKING RATING SCALE. Many-facet Rasch Model. Descriptor difficulty arrangement.
114 CHAPTER VII: THE SPEAKING COGNITIVE PROCESSES IN VSTEP.3-5 SPEAKING TEST AND NON-TEST CONDITIONS. General information about the survey and stimulated recalls. The cognitive processes in VSTEP.3-5 speaking test and non-test conditions118 7. Stimulated Recall Interviews.
137 CHAPTER VIII: CONCLUSION. Summary of the findings. Cognitive processes that the VSTEP.3-5 test takers are supposed to experience. Calibration of cognitive demands across different levels of the VSTEP.
Similarities between cognitive processes in VSTEP.3-5 speaking test and non-test conditions. Implications to VSTEP.3-5 speaking section administration, new language test development and validation, and Weir‘s socio cognitive framework .3-5 development and administration. Language test development and validation. Weir‘s socio cognitive framework.
Limitations and further studies. 152 THESIS RELATED PUBLICACTIONS. I Appendix 2: Stimulated-recall interview protocol. III Appendix 3: Discussion points–Focus group of VSTEP.3-5 speaking test designers .XII Appendix 4: Discussion points – Focus group of VSTEP.
XIII Appendix 5: Discussion points – Focus group of VSTEP.XIV Appendix 6: Survey questionnaires with 288 VSTEP. XV Appendix 7: Stimulated recall interview questions. XXV vi Appendix 8: VSTEP. XXVI Appendix 9: VSTEP.3-5 speaking mark sheet.
XXVIII Appendix 10: Summary of transcripts – Focus group of VSTEP.3-5 speaking test designers. XXIX Appendix 11: Summary of transcripts–Focus group of VSTEP.XXXIV Appendix 12: Summary of transcripts–Focus group of VSTEP.3-5 oral examinersXL Appendix 13: Stimulated recall session transcripts. XLVI Appendix 14: Consent form. LXXVIII vii LIST OF TABLES Table 1: The VSTEP.
41 Table 2: Summary of the VSTEP.3-5 speaking test construct .3-5 overall language ability .3-5 oral production ability .3-5 oral Interaction ability. 44 Table 6: Data collection instruments. 58 Table 7: Item facet summary of statistics .3-5 speaking section correlation data. 107 Table 9: T-test result between VSTEP.3-5 overall scores and speaking scores (C1 students)*.
108 Table 10: T-test result between VSTEP.3-5 overall scores and speaking scores (B2 students)*. 108 Table 11: T-test result between VSTEP.3-5 overall scores and speaking scores (B1 students). 108 Table 12: T-test result between VSTEP.3-5 overall scores and speaking scores (lower than B1 students)*. 109 Table 13: Summary information of 30 test takers who attended the stimulated recall sessions.
118 Table 14: Summary of survey results of general information about VSTEP. 119 Table 15: Summary of survey results about Part 1 of the VSTEP. 121 Table 16: Summary of survey results about Part 2 of the VSTEP. 124 Table 17: Summary of survey results about Part 3 of the VSTEP.
128 Table 18: Summary of stimulated recall interviews about the cognitive processes that the test takers may experience in the test condition. 132 Table 19: Summary of stimulated recall interviews about cognitive processes in everyday speaking situations. 135 viii LIST OF FIGURES Figure 1: Field‘s model of separating levels of processing from outputs of processing (adapted version of the Levelt model, 1989). 22 Figure 2: The socio-cognitive framework for validating speaking (Weir, 2005).
24 Figure 3: A framework for conceptualizing speaking test validity (Taylor, 2011). 25 Figure 4: Field‘s information sources feeding into the phases of the processing system26 Figure 5: VSTEP. 37 Figure 6: The VSTEP. 40 Figure 7: Test use stages (adapted from the Manual for language testing) .3-5 test assembling stages .3-5 speaking mark sheet.
51 Figure 10: The research design of the study. 55 Figure 12: All facets table of VSTEP.3-5 speaking scores of 288 test takers. 106 Figure 13: Arrangement of difficulty levels of all the descriptors of the VSTEP. 110 Figure 11: Stimulated-recall interview protocol stages adapted from O‘Brien (1993) and Dempsey (2010).
VIII ix CHAPTER I: INTRODUCTION 1. Rationale The University of Languages and International Studies (ULIS) is a research university in languages, linguistics, international studies and related social sciences and humanities. ULIS‘s mission is to contribute to the national development through the pursuit of excellence in education, research, and the provision of quality services relevant to social demands. ―We particularly value creativity, dynamics, honesty, responsibility, and capability to work and gain success in a multicultural and competitive globalizing world.‖1 ULIS was appointed by the National Foreign Languages Project of Vietnamese Ministry of Education and Training to develop the English language test for the three levels of English language competences, which are corresponding to the B1, B2, and C1 levels of the CEFR (see Appendix 1 for the VSTEP.3-5 development team members).
The test was named as the Vietnamese Standardized Test of English Proficiency (VSTEP.3-5 was designed to target the population of test-takers of post- secondary school levels such as undergraduate, graduate students of the Vietnam‘s colleges and universities, and working people of different fields.3-5, like other high-stake language tests, shall be used predictively, e. to show that an individual is capable of performing in a particular job, class or academic setting. This places a responsibility on the test designers to ensure that the test elicits behaviors similar to the behaviors that happen in a real-world context. When developed, the VSTEP.3-5 test was validated and validation results showed evidence of construct validity of the test, which was significant for the test stakeholders to confidently use the test scores.
Nevertheless, like all other language tests2, validity evidence should continuously be developed when the tests are in use.3-5, though was validated at the development stages and validation results were widely published, further validation studies have been conducted3. For such purpose, the cognitive validity of the speaking section of the VSTEP.3-5 test, therefore, was investigated to establish the 1 Retrieved from http://ulis.vn/english/taxonomy/term/1/6 on Feb 9th 2014. 2 Cambridge research notes series are a typical example.