An investigation of experienced EAP teachers’ marking processes Author: Lijun Yin University of St Andrews British Council ELT Master’s Dissertation Awards 2019 Commendation An investigation of experienced EAP teachers’ marking processes Date Submitted: 17 August 2018 Table of Contents Acknowledgements.1 The context of research .2 Working definitions of key terminology .3 The outline of this dissertation. 4 Chapter 2 Literature Review .1 Criteria-based assessment in the HE context .1 Theoretical Rationale for criteria-based assessment in the HE context .2 Empirical investigations into criteria-based grading .2 The assessment of writing in the HE context.1 The expectation of stance in student writing .2 Issues in providing effective feedback to students .3 Summary of literature .4 Gap in the literature and research question.3 Data collection and analysis procedures .1 Research design and pilot of study .2 Teacher marking and interview .3 Coding of qualitative data .1 Evaluation of the essay sample .1 Ratings of the level of essay on assessment criteria.2 Perceptions of writing quality .3 Common themes in the essay evaluation .4 inter-rater differences in the essay evaluation.2 General marking practices of coursework essays .1 Scoring decision-making process .1 Discussion of writing constructs for EAP teachers in the essay assessment .1 The effect of stance on the perception of writing quality .2 Interaction of implicit construct and explicit criteria .2 Discussion of the application of assessment criteria in marking practices .3 Discussion of feedback provision .4 Implications for EAP writing assessment and instruction .1 Summary of the key findings .2 Contributions of my research .4 Directions for future research. 54 Appendix A Essay sample. 57 Appendix B Marking Criteria.
61 Appendix C Semi-structured Interview Protocol. 62 Appendix D Interview transcript of teacher A. 63 List of Tables and Figure Table 1 Participants’ ratings of the essay on each criterion……………………………………………….24 Figure 1 Teacher B’s comment on the essay introduction………………………………………………….26 Table 2 Strengths and weaknesses of the essay sample…………………………………………………….30 Table 3 Participants’ comments on and ratings of Language and Vocabulary…………………….33 Acknowledgements Abstract Assessment is of central importance in education in that it not only serves the purpose of certifying student achievement but also has the potential of promoting student learning (e. Research on assessment in tertiary contexts (e.
Bloxham et al., 2011; Bloxham et al., 2016; Tomas, 2014) illustrates the complexity of marking as a professional judgment and problematises the transparency of marking processes in spite of the availability of assessment criteria. The elusiveness of marking criteria (e. Bloxham et al., 2011) and the intangibility of tutor expectations limit the learning purpose of assessment. As a response, this study reports on an investigation into experienced EAP instructors’ coursework essay marking processes, an area awaiting to be explored despite the importance of writing instruction in EAP programmes (e.
Alexander et al., 2008), with the aim of identifying what factors play in EAP instructors’ perception of writing quality. 5 EAP teachers from 2 UK universities were asked to mark an essay sample and interviewed about their opinions of that essay sample and their general marking practices. The research highlights the effect of the construct of stance, as revealed in the thesis statement, developed through critical evaluation of sources and presented in a coherent manner, on the perception of writing quality, possibly because of its interrelation with explicit criteria. In addition, the application of assessment criteria is also affected by personal marking standards and student profile.
Pedagogical implications of the findings of this study for the EAP instruction are also discussed. Chapter 1 Introduction Assessment plays an inherent part in student learning. The judgements that teachers make about student work carry high stakes for students in that they form the basis of certification for learning and impact on students’ future prospects (Sadler, 2009). Scholarship on Higher Education (HE) assessment (e.
Sadler 2009; Bloxham et al., 2011; Bloxham et al., 2016) suggests that despite the availability of assessment criteria, marking processes may still not be transparent to students, since marking, by nature, is subjective and a professional judgment. How a piece of writing is assessed involves many factors, including teachers’ interpretation of criteria publicised in the institutions (Tomas, 2014), expectation of student writing which may incorporate criteria not listed in the written criteria document (Sadler, 2009; Bloxham et al., 2011; Bloxham et al., 2016;) and student profile (Bloxham et al. Considering that criteria and writing constructs are often open to interpretation, students are likely to find difficulty in understanding teachers’ expectations and feel confused with the feedback (e. Lea and Street, 1998; Carless, 2006) wrapped in evaluative discourses such as “be more critical” and “your essay lacks a structure”.
Under this circumstance, students may not know how to act on teachers’ feedback, and hence the learning purpose of assessment (Carless, 2007) is undermined. Against this background, the current research investigates how EAP student coursework essays are assessed, in hopes of revealing tutor expectations and thus shedding light on the EAP instruction. This introduction first sets the context of study and then provides working definitions of key terms. The outline of this dissertation is provided at the end of this chapter.1 The context of research The internationalisation of higher education attracts an increasing number of students who use English as a second language (L2) to come to study in universities where English is the medium of instruction.
With the linguistic and cultural diversity of student background, English for Academic Purposes (EAP), defined broadly as “teaching English with the aim of assisting learners’ study or research in that language” (p. 1) by Hyland (2006), has now already expanded to be a major force in English language teaching. Various formats of EAP programmes are available today and cater to the needs of students at different study levels, ranging from one-year long foundation courses for pre-university students to a few weeks of summer pre-sessional programmes for postgraduates. According to Schmitt and Hamp- Lyons (2015), many EAP programmes tend to have a dual focus: one is to improve the language proficiency of learners, the other is to help students to close any gaps between their previous educational experiences and the new academic community culture that they wish to join.
Hyland (2018) in his most recent publication indicates that the latter target goes far beyond than the grasp of basic language competence, and the communicative demands imposed by target disciplinary community on students can become formidable. In the light of the prestige of written communication as constructing, presenting and communicating knowledge in tertiary education (Hyland, 2013a), academic writing dominates the format of assessment in university courses and becomes the centre of EAP teaching and learning (Alexander et al. The importance of developing L2 students’ ability in academic written communication in EAP programmes inspired me to look at the experienced EAP teachers’ expectation of academic discourse features in student essays so as to better address student writing needs. 2 To make the findings of this research more generalisable, this study recruited 5 eligible EAP teachers working at two different HE institutions.
Both universities provide a variety of EAP programmes including year-around EAP foundation programmes, summer EAP courses, and in-sessional language service. The 2 research sites differ in the scale, with one located in a big city enrolling the total number of students more than 4 times higher than does the other one based in a small town.2 Working definitions of key terminology In the assessment literature, various terms, which may essentially mean the same thing, are used in different occasions. To clarify the terms used in this study, the following paragraphs give working definitions of key terminology. Assessment refers to “the process of forming a judgement about the quality of student work and extent of student achievement or performance” (p.
177), according to Sadler (2005). While formulating the judgement, teachers are also likely to provide feedback on student performance to facilitate learning, which is the key characteristic of formative assessment (Sadler, 1998). Given that evaluating and improving student learning are both the main purposes of assessment (Carless, 2007), assessment in this research is used in broad terms including the processes of scoring student work and providing feedback. Assess and mark are also used interchangeably in this dissertation to refer to such processes.
Assessment/marking criteria, also termed “rubric,” “marking scheme”, “marking scale”, refers to “a tool used for marking student performance in assessment tasks and to provide feedback” (University of Wollongong Australia, 2015). Typically, marking criteria, as a scoring guide published to teachers and students, contain levels of performance (such as merit, high merit, distinction), cut-off point for each level (such as scores ranging from 16.5- 3 20 on a 20-point scale fall into the level of distinction), level descriptors (such as the key characteristics for each level of performance). In this study, criteria specifically refer to the explicit written criteria presented in a document available to teachers and students, unless modified by the adjective “implicit”. Criterion-referenced assessment /criteria-based grading means that “each student is judged against predetermined criteria without regard to other students” (Lok et al.
By contrast, norm-referencing refers to that “a predetermined percentage of students would obtain a certain grade” (Lok et al. 450) and often involves the comparison of student performances (Sadler, 2005).3 The outline of this dissertation The rest of the dissertation is arranged as follows. In chapter 2, literature on both HE assessment in general and academic writing assessment in particular is reviewed, and the research question, emerged from the gap in existent literature, is raised at the end of the chapter. Chapter 3 provides a rationale for the use of interviews as the data collection method and presents how data are analysed.
The analysis of interview results is provided in Chapter 4. Chapter 5 discusses the findings of the current study with reference to the previous literature and suggests the implications for writing instruction and assessment. Chapter 6 summaries the key findings and points out caveats as well as directions for future research. 4 Chapter 2 Literature Review As pointed out in chapter 1, this study sets out to probe into experienced EAP instructors’ assessment practices of student coursework essays.
Informed by the research context of academic writing assessment in tertiary education, this literature review starts with presenting the theoretical rationale of ethical considerations and educational benefits underpinning the current criteria-based assessment framework in HE contexts. What follows is an examination of empirical studies on criteria-based grading, pointing to the complexity of marking. The second part of this chapter presents literature on academic writing assessment including the necessity of stance taking in written communication and the issues in effective feedback provision arising from the elusiveness of marking criteria and evaluative discourses. After reviewing the two research tracks of educational assessment in tertiary context and academic writing, this chapter provides a summary of literature bringing together the scholarship on the two fields.
This chapter ends with identifying the gap in the existing literature and raising the research question.1 Criteria-based assessment in the HE context 2.1 Theoretical Rationale for criteria-based assessment in the HE context In the UK higher education sector, informing students of the assessment criteria that will be used to mark their assignment is regarded as a condition for good assessment practice and thus attached much importance to by various stakeholders. For example, UK Quality Assurance Agency for Higher Education (QAA) proposes that “publicising and using clear assessment criteria are key factors in assuring that marking is carried out fairly and consistently” (QAA, 2006, p. In the National Student Survey questionnaire, which is issued by Office for Students (OfS) with the aim of seeking students’ opinions of academic quality, the category Assessment and feedback contains an item named “the criteria used in 5 marking have been made clear in advance” (OfS, 2017). In a similar vein, students themselves also request for “clear assessment criteria linked to learning outcomes and easily accessible”, as revealed in the document “Assessment and feedback benchmarking tool” published by National Union of Students (NUS, 2015).