LINGUISTICALLY MOTIVATED FEATURES FOR CCG REALIZATION RANKING DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Rajakrishnan Rajkumar, M. Graduate Program in Linguistics The Ohio State University 2012 Dissertation Committee: Professor Michael White, Advisor Professor Peter Culicover Professor William Schuler c Copyright by Rajakrishnan Rajkumar 2012 ABSTRACT Natural Language Generation (NLG) is the process of generating natural language text from an input, which is a communicative goal and a database or knowledge base. Infor- mally, the architecture of a standard NLG system consists of the following modules (Reiter and Dale, 2000): content determination, sentence planning (or microplanning) and sur- face realization. This thesis is about designing novel, linguistically motivated features for surface realization (the final NLG module mentioned above), the process by which text is created from an abstract representation of language according to the rules of syntax and morphology.
It primarily involves three interrelated problems: constituent ordering, in- flection and agreement and function word insertion. For addressing these problems, most state-of-the-art realization ranking models (Velldal and Oepen, 2005; White and Rajkumar, 2009) employ features which are based on very basic insights from linguistic theory (POS tags, rules derived from parse trees, for example). More sophisticated insights of linguis- tic theory have not been widely perceived as necessary for increased system performance, with very basic insights providing the most gains (similar to the situation Johnson (2009) describes in the context of natural language parsing). In contrast, our goal is to design features motivated by insights from theoretical linguis- tics and also based on cognitively plausible accounts of language comprehension discussed in the linguistics literature, so that the realization ranking model can better approximate human judgements of fluency and acceptability.
We show that the minimal dependency ii length theory (Gibson, 1998; Temperley, 2007) helps with the constituent ordering prob- lem in surface realization. For the problem of generating correct inflected word forms, we demonstrate that a machine learning-based approach is well-suited to encode insights from the theoretical linguistics literature on English agreement (Kathol, 1999; Pollard and Sag, 1994). This approach leads to improvements over a competitive baseline model containing n-gram and parsing features (of the kind described in Johnson, 2009). Finally, we demon- strate empirically that the uniform information density principle discussed in (Jaeger, 2010) contributes towards the that-complementizer choice in the context of surface realization.
Thus the primary contribution of this thesis is the design and evaluation of linguisti- cally motivated features that model all the three classes of linguistic phenomena described above. We demonstrate that such features further enhance the output of a state-of-the art realization model and also lead to more fluent output. The secondary contribution of this thesis is a novel and particularly appropriate evaluation method that complements standard NLG evaluation techniques, viz. BLEU scores and human evaluations.
We adopted distri- butional analyses to compare the output of ranking models with the gold standard corpus, as these analyses show how the output of these models match the dependency length and constituent order distributions in the reference corpus. We also evaluated how various types of complement clauses predicted by classification models consisting of various feature sets performed in comparison to the gold standard corpus. Thus these analyses also provided insights on how different feature sets fared in preferring constructions of interest. iii For My Linguistics 201 & 384 students iv ACKNOWLEDGMENTS This thesis evolved out of several research assistantships1 supervised by my academic advisor, Professor Michael White, who was closely involved in all aspects of this work.
Though we had several disagreements over the years, I thank my advisor for introducing me to some challenging problems in computational and experimental linguistics. I would also like to express my profound debt of gratitude to my advisor for spending a consid- erable amount of his time reviewing and critiquing my work and especially for providing detailed feedback on my writing. The chapters of this thesis borrows from and further expands conference papers we jointly authored along the course of my studies here. My sincere thanks to my thesis committee members, Professors Peter Culicover and William Schuler, whose perceptive comments and pointers to existing work have contributed to the development of the ideas discussed in my thesis.
The remaining problems of this work, however, are entirely my own contribution. My time here was spent mostly in the company of fellow CL graduate students Dennis Mehay and Dominic Espinosa. I learned a lot about the practical side of computational linguistics from them. My free wheeling discussions with Omkar Lele, Andrew Plummer, DJ Hovermale and Salena Anderson (née Sampson) on the theory, practice and sociology of linguistics have also been an invaluable part of my education.
From each person mentioned 1 This work was supported by Professor White’s grant NSF IIS-0812297, Linguistics 201 and 384 teaching support from the OSU department of linguistics and by an allocation of computing time from the Ohio Supercomputer Center. I am extremely grateful for this support. v so far, I have learned more about the English language than I did from all the English teachers who were in my life. At this point, I would also like to thank my mother for her consistent efforts to understand what I was actually doing for a living all these years.
This document will not be complete without a mention of the people I encountered during my years at Jawaharlal Nehru University (JNU), New Delhi, an institution which I feel exists within even greater contradictions than, perhaps, India itself. Nonetheless, a very unique place to study language for many reasons. I fondly remember the sustained support offered by Nabarun Roy, Kishore Jose, Suhail Ebrahim, Krishnakumar Trippunit- tura and Vinod Wayanadan all these years hence. Along with them, I would also like to acknowledge the role of my childhood friend from Trivandrum, Krishnachandran Balakr- ishnan, in lending credence to the view that laymen could often be hard to deal with in matters related to language.2 Finally, I would like to thank my former teachers at JNU, in particular, Professors Ayesha Kidwai, Pramod Pandey, Anvita Abbi and Girish Nath Jha.
Their encouragement and the wisdom imparted by their classes inspired me to pursue a doctoral degree in linguistics. A special note of thanks to JNU and OSU Linguistics alum- nus, Professor Shravan Vashist, who reviewed and helped revise my OSU graduate school application back in November 2004. Since then, his blog posts and papers have provided perspective on many a weary day. 2 “Laymen are generally lousy linguists: they do not know what questions to ask, they do not know how to look for answers to them and they are too ready to accept generalizations to which they could easily find counter examples.
McCawley vi VITA June, 2002 ., Mechanical Engineering College of Engineering Trivandrum, In- dia May, 2004 ., Linguistics Jawaharlal Nehru University, New Delhi, India November, 2010 ., Linguistics The Ohio State University, USA June, 2012 ., Linguistics The Ohio State University, USA PUBLICATIONS Eye Tracking and Speech Synthesis Evaluation Michael White, Rajakrishnan Rajkumar, Kiwako Ito, and Shari Speer. Eye Tracking for the Online Evaluation of Prosody in Speech Synthesis. In Bangalore Srinivas and Amanda Stent, editors, Natural Language Generation in Interactive Systems. Cambridge University Press, In Press.
Rajakrishnan Rajkumar, Michael White, Kiwako Ito, and Shari Speer. Evaluating Prosody in Synthetic Speech with Online (Eye-Tracking) and Offline (Rating) Methods. 7th Speech Synthesis Workshop, Kyoto, Japan, 2010. Michael White, Rajakrishnan Rajkumar, Kiwako Ito, and Shari Speer.
Eye Tracking for the Online Evaluation of Prosody in Speech Synthesis: Not So Fast! In Proc. of the 10th An- nual Conference of the International Speech Communication Association (INTERSPEECH- 09), 2009. Machine Translation vii Karthik Visweswariah, Rajakrishnan Rajkumar, Ankur Gandhe, Ananthakrishnan Ramanathan, and Jiri Navratil. A Word Reordering Model for Improved Machine Translation, In Pro- ceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 486-496, Edinburgh, Scotland, UK.
Association for Computational Lin- guistics. Natural Language Generation Michael White and Rajakrishnan Rajkumar. Minimal Dependency Length in Realization Ranking. In Proceedings of the 2012 Conference on Empirical Methods in Natural Lan- guage Processing, Jeju Island, Korea, July 2012.
Association for Computational Linguis- tics. Rajakrishnan Rajkumar and Michael White. Linguistically Motivated Complementizer Choice in Surface Realization. In Proceedings of the UCNLG+Eval: Language Genera- tion and Evaluation Workshop, pages 39-44, Edinburgh, Scotland, July 2011.
Association for Computational Linguistics. Rajakrishnan Rajkumar, Dominic Espinosa, and Michael White. The OSU System for Sur- face Realization at Generation Challenges 2011. of the 13th European Workshop on Natural Language Generation.
Rajakrishnan Rajkumar and Michael White. Designing Agreement Features for Realization Ranking, In Coling 2010: Posters, pages 1032-1040, Beijing, China, August 2010. Coling 2010 Organizing Committee. Dominic Espinosa, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant.
Fur- ther Meta-Evaluation of Broad-Coverage Surface Realization, In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 564-574, Cam- bridge, MA, October 2010. Association for Computational Linguistics. Rajakrishnan Rajkumar, Michael White, and Dominic Espinosa. Exploiting Named Entity Classes in CCG Surface Realization.
NAACL HLT 2009 Short Papers, 2009. Scott Martin, Rajakrishnan Rajkumar, and Michael White. Grammar Engineering for CCG using Ant and XSLT. In Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (SETQA-NLP 2009), pages 45- 46, Boulder, Colorado, June 2009.
Association for Computational Linguistics. viii Michael White and Rajakrishnan Rajkumar. Perceptron Reranking for CCG Realization. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Process- ing, pages 410-419, Singapore, August 2009.
Association for Computational Linguistics. Michael White and Rajakrishnan Rajkumar. A More Precise Analysis of Punctuation for Broad-Coverage Surface Realization with CCG. of the Workshop on Grammar Engineering Across Frameworks (GEAF08), 2008.
Michael White, Rajakrishnan Rajkumar, and Scott Martin. Towards Broad Coverage Sur- face Realization with CCG. of the Workshop on Using Corpora for NLG: Lan- guage Generation and Machine Translation (UCNLG+MT), 2007. FIELDS OF STUDY Major Field: Linguistics Studies in Natural Language Generation: Professor Michael White ix TABLE OF CONTENTS Page Abstract.
vii List of Tables. xiii List of Figures .1 Contributions and Outlook .1 Approaches to Surface Realization .1 Realization Ranking Approach .2 Feature Sets in Realization Ranking .3 Classification Based Surface Realization .4 Feature Sets in Classification Based Surface Realization .2 Evaluating Surface Realization Systems .1 Evaluating Realization Ranking Output .2 Evaluating the Output of Classifier Generation .3 Surface Realization with Combinatory Categorial Grammar (CCG) .1 Combinatory Categorial Grammar (CCG) .2 Realization from an Enhanced CCGbank .6 Baseline Model Features .8 Realization Results using Baseline Model Features .4 Motivation for Linguistically Motivated Features .5 Overview of System Performance .1 OpenCCG System Performance. Minimal Dependency Length in Realization Ranking .1 Minimal Dependency Length .1 Evidence from Comprehension & Corpus Studies .2 Evidence from Optimizing Grammars .3 Evidence from Language Evolution and Change .4 Dependency Length in Statistical Parsing .5 Results from Surface Realization .2 Dependency Length Minimization and Other Factors Influencing Con- stituent Ordering .5 Complement-Adjunct distinction .8 Dependency Length and Other Theories .3 Targeted Human Evaluation .1 Divergences from Canonical Word Order. Designing Inflection and Agreement Features for Realization Ranking .1 Motivation for Incorporating Inflection and Agreement Features in the Statistical Ranking Model .2 The Heterogeneous Nature of Number Agreement .1 Problems with a Purely Syntactic View .2 Problems with a Purely Semantic View .3 The Nature of Animacy Agreement .4 CCG and Balanced Punctuation .2 Features for Punctuation in CCG? .1 Animacy and Number Agreement Features .2 Balanced Punctuation Feature .1 Comparison with Hand-crafted Agreement Rules.
Linguistically Motivated Complementizer Choice in Surface Realization .1 Complementizer Choice in Surface Realization .2 Uniform Information Density Principle and that-mentioning. Conclusions and Overview .1 Conclusions and Future Work. 131 xii LIST OF TABLES Table Page 2.1 Features in LFG realization ranking (Cahill et al.2 Atomic features used in HPSG realization ranking (Nakanishi et al.3 Information status features (Cahill and Riester, 2009) .4 Information status asymmetries (Cahill and Riester, 2009) .