Indicators of quality in natural language composition by Barry John Donahue A thesis submitted in partial fulfillment of the requirements for the degree of DOCTOR OF EDUCATION Montana State University © Copyright by Barry John Donahue (1982) Abstract: This study was designed to: (1) examine the relationships that exist between various commonly used measures of writing quality; and (2) determine to what extent experienced English teachers and prospective English teachers agree in their opinions of writing quality. The measures of writing quality chosen for comparison were Holistic scoring, Atomistic scoring, Mature Word Index, Type/Token Index, Mean T-unit Length, and Syntactic Complexity. The Holistic and Atomistic methods are subjective and thus required several human raters, while the other four methods are objective and could be scored using mechanical procedures. Four groups of raters were used in the study, corresponding to all possible combinations of subjective methods (Holistic and Atomistic) with experience levels (experienced teachers and prospective teachers).
Both the Holistic and Atomistic methods provided very high reliability coefficients for all groups of raters, but there was a large range of reliabilities for the categories of the Atomistic method. The conclusions of the study were: (1) The Atomistic scoring method is more time-consuming and no more reliable or informative than Holistic scoring. (2) Many of the factors generated by Diederich do not provide reliable results between raters. (3) The Mature Word Index and Type/Token Index are accurate measures of writing quality, while the Mean T-unit Length and Syntactic Complexity Index are not.
(4) Writers do not misuse or misplace mature words as they often do syntactic structures. (5) Student raters judge writing as a whole in essentially the same manner as do expert raters, but are slightly less able to distinguish the various factors of quality writing. The recommendations made in the study included preference of Holistic methods over Atomistic methods, distrust of the Mean T-unit Length and Syntactic Complexity methods, and the need to convey to prospective teachers their competence as judges of writing quality. INDICATORS OF QUALITY IN NATURAL LANGUAGE COMPOSITION BARRY JOHN DONAHUE A thesis submitted in partial fulfillment of the requirements for the degree of DOCTOR OF EDUCATION Approved: chair MONTANA STATE UNIVERSITY Bozeman, Montana August, 1982 iii ACKNOWLEDGEMENT' The author would like to thank several people for their assis tance in the preparation of this study.
First, thanks to Lynne Jermunson, Roxanna Davant, and Merry Fahrman for helping with the initial editing and grading of the essays. Sara Jane Steen's assistance in allowing the author to use the members of her English methods classes as raters is greatly appreciated. Thanks to those raters as well as the expert raters. The staff of the Word Processing Center at Montana State Univer sity, especially Judy Fisher and Debbie LaRue must be thanked for their very competent and professional typing and revising of the manuscript of this study.
The author would also like to thank Dr. Eric Strohmeyer, Dr. Robert Thibeault, Dr. Gerald Sullivan, Dr.
Douglas Herbster, and Professor Duane Hoynes for their participation on the author's Doc toral Committee. Their assistance, especially during the rush of the final weeks, has been very helpful. Finally, many thanks to Dr. Leroy Casagranda, Chairman of the Committee for his patience and guidance throughout the author's graduate work, and for his advice in the preparation of this study.
TABLE OF CONTENTS V I T A. iii LIST OF T A B L E S. A I Statement of the P r o b l e m. 3 Applications of the Results.
4 Questions Answered by the Study. 8 Limitations and Delimitations. 10 Definition of Terms. REVIEW OF LITERATURE.
16 Grading Essay Writing. 16 Holistic Methods of Evaluation. 21 Atomistic Methods of Evaluation. 47 Essay and Rater Descriptions.
47 Categories of Investigation. 51 Method of Data Collection. 55 Analysis and Presentation of Data. 64 Comparability of Student Rater Groups.
66 Comparison of Students and Experts Using Atomistic Scoring. 68 Correlations between Methods. 69 Correlations of Atomistic Categories with Methods. 76 Correlations between Categories of the Atomistic M e t h o d.
82 Correlations of Methods with Sum of Rankings of All Other Methods. 87 Analysis of Variance between Expert and Student Raters. 94 Summary of the Study. 96 Holistic Versus Atomistic Scoring.
97 Correlations between Methods. 100 Correlations between Atomistic Categories and Methods. 103 Correlations of Methods with Sum of Rankings of All Other Methods. 108 Comparison of Expert and StudentRaters.
H O Suggestions for Future Research. RAW SCORES OF R A T E R S. INTERMEDIATE RESULTSFROM CALCULATION OF MATURE WORD INDEX. 147 / vii LIST OF TABLES ( 1.
Interpretation of the Standard Frequency Index. Comparison of Grade Point Averages for Student Groups Using Holistic and Atomistic Scoring. Reliability of Average Ratings of Holistic and Atomistic Methods and Each Category of Atomistic Method. Average Scores for Methods Utilizing Raters.
Raw Scores for Methods Not Utilizing Raters. Rank Ordering of Methods and Rater Groups. Pearson Correlation Matrix of Methods and Rater Groups. Spearman Rank Order Correlation Matrix of Methods and Rater G r o u p s.
Pearson Correlations between Methods and Categories of Atomistic Scoring from Students. Spearman Rank Order Correlation between Methods and Categories of Atomistic Scoring from Students. Pearson Correlations between Methods and Categories of Atomistic Scoring from Experts. Spearman Rank Order Correlations between Methods and Categories of Atomistic Scoring from Experts.
Pearson Correlations between Categories of Atomistic Scoring for Experts. Pearson Correlations between Categories of Atomistic Scoring for Students. Pearson Correlations between Categories of Atomistic Scoring for Experts and Those for Students. Correlations between Each Method and the Sum of Rankings of All Other M e t h o d s.
Kendall Coefficients of Concordance for All Methods. Kendall Coefficients of Concordance for Holistic, Atomistic, Mature Word Index, and Type/Token Index Methods. Analysis of Variance for Holistic Rating Groups by Essays. Analysis of Variance for Atomistic Rating Groups by Essays.
Reliability of Ratings by the Student Group Using Holistic S c o r i n g. Reliability of Ratings by the Expert Group Using Holistic Scoring. Reliability of Ratings for the Category "Ideas" by the Student Group Using Atomistic Scoring. Reliability of Ratings for the Category "Organization" by the Student Group Using Atomistic Scoring.
Reliability of Ratings for the Category "Wording" by the Student Group Using Atomistic Scoring. Reliability of Ratings for the Category "Flavor" by the Student Group Using Atomistic Scoring. Reliability of Ratingss for the Category "Usage" by The Student Group Using Atomistic Scoring. Reliability of Ratings for the Category "Punctuation" by the Student Group Using Atomistic Scoring.
Reliability of Ratings for the Category "Spelling" by the Student Group Using Atomistic Scoring. Reliability of the Total of All Categories by the Student Group Using Atomistic Scoring. Reliability of Ratings for the Category "Ideas" by the Expert Group Using Atomistic Scoring. Reliability of Ratings for the Category "Wording" by the Expert Group Using Atomistic Scoring.
Reliability of Ratings for the Category "Organization" by the Expert Group Using Atomistic Scoring. Reliability of Ratings for the Category "Flavor" by the Expert Group Using Atomistic Scoring. Reliability of Ratings for the Category "Usage" by the Expert Group Using Atomistic Scoring. Reliability of Ratings for the Category "Punctuation" by the Expert Group Using Atomistic Scoring.
Reliability of Ratings^ for the Category "Puncuation" by the Expert Group Using Atomistic Scoring. Reliability of the Total of All Categories by the Expert Group Using Atomistic Scoring. Mature Words Used in the Essays. Contractions, Proper Nouns, and Slang Used in the Essays.
Topic Imposed Words Used in the Essays. Number of Types and Tokens Used in the Essays. 150 X ABSTRACT This study was designed to: (I) examine the relationships that exist between various commonly used measures of writing quality; and (2) determine to what extent experienced English teachers and prospec tive English teachers agree in their opinions of writing quality. The measures of writing quality chosen for comparison were Holistic scor ing, Atomistic scoring, Mature Word Index, Type/Token Index, Mean T-unit Length, and Syntactic Complexity.
The Holistic and Atomistic methods are subjective and thus required several human raters, while the other four methods are objective and could be scored using me chanical procedures. Four groups of raters were used in the study, corresponding to all possible combinations of subjective methods (Holistic and Atomistic) with experience levels (experienced teachers and prospective teachers). Both the Holistic and Atomistic methods provided very high reliability coefficients for all groups of raters, but there was a large range of reliabilities for the categories of the Atomistic method. The conclusions of the study were: (1) The Atomistic scoring method is more time-consuming and no more reliable or informative than Holistic scoring.
(2) Many of the factors generated by Diederich do not provide relia ble results between raters. (3) The Mature Word Index and Type/Token Index are accurate measures of writing quality, while the Mean T-unit Length and Syntactic Com plexity Index are not. (4) Writers do not misuse or misplace mature words as they often do syntactic structures. (5) Student raters judge writing as a whole in essentially the same manner as do expert raters, but are slightly less able to distinguish the various factors of quality writing.
The recommendations made in the study included preference of Holistic methods over Atomistic methods, distrust of the Mean T-unit Length and Syntactic Complexity methods, and the need to convey to prospective teachers their competence as judges of writing quality. CHAPTER I INTRODUCTION The skill of effective written communication is one of the most valuable assets which the educated person possesses. It forms the foundation upon which success in other studies may be built; it is a prerequisite to good employment and countless other tasks of social adjustment; and, it provides the means by which ideas otherwise locked tightly in one mind may be transmitted to another. Unfortunately, competent writing is an ability which develops slowly through years of practice.
As a former Chief Inspector of Primary Schools in England (cited in Maybury, 1967:19) stated: No human skill or art can be mastered unless it is constant ly practiced. A short composition once a fortnight, inter spersed with formal exercises is no good at all. There must be bulk. Furthermore, "writing is not an easy activity.
It involves the total being in a process of learning a more and more complex skill" (Carlson, 1970:vii-viii). An explanation of this complexity may be found in the dependence of the writing skill upon other, more basic skills. As Moffet and Wagner (1976:10) wrote: Teachers habitually think of literacy as first or basic, as reflected in the misnomer "basic skill," because the two Rs occur early in the school career and lay the foundation for book learning. But we do well to remind ourselves that reading and writing actually occur last— that is, not only after the acquisition of oral speech but also after considerable nonverbal experience.
The three levels of coding. mean that experience has to be encoded into thought before thought can be encoded into speech, and thought encoded into speech before speech can be encoded 2 into writing. Each is basic to the next, so that far from being basic itself literacy depends on the prior codings.adds an optional, visual medium to the necessary, oral medium. Or, simply, as Chaucer says: The Iyf so short, the craft so long to lerne.
(Parliament of Fowls, 1.1) Because of the importance of the skill of writing and the time required to attain functional mastery of it, it is essential that teachers have precise information concerning the progress of each student toward attainment of the skill.