Applying a Framework of Interpersonal Adaptability for Assessment by Tom C. Oliver A Thesis Presented to The University of Guelph In partial fulfillment of requirements for the degree of Doctor of Philosophy in Psychology Guelph, Ontario, Canada © Tom C. Oliver, December, 2012 ABSTRACT APPLYING A FRAMEWORK OF INTERPERSONAL ADAPTABILITY FOR ASSESSMENT Tom Oliver Advisor: University of Guelph, 2012 Professor Peter A. Hausdorf In many of today’s work setting, workers are required to spend a considerable part of their day engaged in social interactions and managing social relationships with customers (Schneider, 1994), and with teams (Kozlowski & Ilgen, 2006).
Furthermore, increases in globalization (Javidan, Dorfman, de Luque, & House, 2006), boundaryless organizational structures (Macy & Izumi, 1993), and workplace diversity (Mahoney, 2005) have increased the ambiguity and complexity of workplace interpersonal interactions. As a result, in today’s workplace there is a great need for employees to be interpersonally adaptable (Klein, DeRouin, & Salas, 2006). Though many assessment practices and measures intended to assess individual effectiveness in social situations exist, many of these measures and practices do not assess the situationally-specific and goal-oriented aspects of interpersonal adaptability. There were two primary purposes for the dissertation.
First, this dissertation introduces a framework of interpersonal adaptability. In doing so, this framework was meant to highlight three opportunities to improve the assessment of interpersonal adaptability. Specifically, in order to conduct construct-valid ratings of interpersonal adaptability there is a need to design more contextualized assessments, improve the match between construct and method, and incorporate dynamic aspects. The second key purpose was to apply some of these opportunities to current assessment practices.
Two empirical studies were included in this dissertation. For the first empirical study, role play assessment exercises were revised to account for context created by role players’ portrayed disposition. It was found that the portrayed disposition of the role player accounted for a significant amount of the between-exercise variance in participants’ demonstrated interpersonal behaviors and performance. For the second empirical paper, a validation study was conducted with measures from a multi-mini interview (MMI), which is a multi-stationed interview used to assess non-cognitive skills of applicants to health professional schools.
An analysis of the MMI measures’ convergent and discriminant validity suggested that it continues to be unclear whether MMI measures assess participants’ interpersonal behaviors or interpersonal processing. Implications for future research and practice are discussed. ACKNOWLEDGMENTS As I reflect on the past three years, I find it truly humbling thinking about all of the support I have received from a number of people. I may or may not have thanked them along the way, and I hope I still get a chance to thank each of these people again in person.
In the meantime, it’s always particularly nice to spell out a big THANK YOU in writing. So here I go: To my advisor Peter Hausdorf, you truly have been a great advisor. Whether I needed a boost of enthusiasm, an alternative perspective to thinking about an issue, or a nudge to keep things more simple, you provided it. You were always available and willing to give me your time and attention, often at the last minute, and you were always prepared to support my research and professional goals and targets.
As a result of all of your support, I have a dissertation project that I am proud of, and I feel well prepared as I enter full-time into practice. I am extremely pleased to have had the opportunity to work collaboratively with all three of my committee members - Filip Lievens, Kent Hecker, and Peter Conlon. Each of you surpassed my expectations for how willing you were to be involved and active contributors throughout the different stages of this project. The breadth of expertise I was able to draw upon was awesome! I would like to particularly thank Peter Conlon for making it possible for me to complete this project at the Ontario Veterinary College.
This project often felt like it was far too big in scope to be pulled off. And really, it was. I didn’t quite realize how challenging collecting video data would be until I was learning about last resort measures for retrieving data from crashed hard drives (put the hard drive in a freezer?!?). I owe a great deal of gratitude to Elizabeth Sinclair for taking iv care of so many of the logistical steps and contingency plans for my project’s data collection.
In addition, I would still be coding all of the data if it were not for the engaged and reliable support of the 15 research assistants that were a part of my research lab. Thank you all! My project really benefited from the collaborative and collegial environment here at the University of Guelph. In particular, I would like to acknowledge David Stanley, Jeffrey Spence, Meghan McMurty, Michael Meehan, and Scott Colwell, for their insight and support. My grad school experience would not have been so positive if it were not for all the great friends I made along the way.
Dan Zdzieborski, Steve Risavy, Troy Rieck, Amanda Tobe, Christine Yip, Erica Skinner, Leann Schneider, and Sandeep Aujla; I love the work-hard / play-hard environment we created. If work takes me outside of Ontario, I look forward to seeing you all at future professional conferences! Of course, I reserve special thanks to my family. My parents, John and Valerie Oliver, thank you for modeling hard work and for providing me with an unlimited reserve of emotional support. My brother and sister, Peter and Vanessa Oliver, I am now the most respectable of us Olivers on paper, but I have no doubt that you will both find ways to surpass me soon enough! Last, but certainly not least, I thank my beautiful girlfriend Julie Rogers.
It takes a truly kind and loving girl to have put up with all of my academic shop talk, staring off into space, and my general lack of interpersonal adaptability at home. Thank you babe – I love you! v TABLE OF CONTENTS ACKNOWLEDGMENTS. iv LIST OF TABLES. x LIST OF FIGURES.
1 MANUSCRIPT 1 Conceptualizing and Assessing Interpersonal Adaptability: Towards a Functional Framework. 7 Distal-Level Variables. 13 Within-Situation Variables. 17 Characteristics of Interpersonal Situations.
22 Between-Situation Variables. 28 Implications for Assessment. 30 Appropriately Match Construct to Method. 31 Design Contextualized Assessments.
33 Design Dynamic Assessments. 37 MANUSCRIPT 2 vi Shining a Light Into the Black Box: Applying Interpersonal Theory to the Role Play Exercise. 53 Trait Activation Theory. 60 Role Play Exercises.
61 Role Player Instructions. 64 Manipulation Check of Role Plays. 70 Test of Hypotheses. 92 MANUSCRIPT 3: Assessing the Construct Validity of MMI Non-Cognitive Skill Ratings and Conducting a Validation of the MMI With Alternative Measures of Non-Cognitive Ability.
100 Non-Cognitive Skills Assessed By the MMI. 102 Personality and Non-Cognitive Skill. 136 APPENDIX A - Role Play Exercises Instructions. 140 APPENDIX B- MPCC Video Coder Training Materials.
141 APPENDIX C - Affiliation & Control Video Coder Training Materials. 142 APPENDIX D - Affiliation & Control Video Coder ‘Cheat Sheet’. 143 APPENDIX E - Affiliation & Control – Example Rating Measure. 144 APPENDIX F - MPCC Rating Measure for Role Play ‘H’ (example).
145 APPENDIX G - MPCC Content Categories for Coding Lines of Inquiry. 147 APPENDIX H - MPCC Content Categories for Coding Lines of Inquiry. 148 APPENDIX I - Role Play Performance Outcome Rating Measure. 149 viii APPENDIX J - Participant Demographic Questionnaire, With Personality, Social Skill, Consent, and Debrief.
150 APPENDIX K - Correlations Between Cognitive Ability, Social Skill, and Personality With Participant Behavior, Interpersonal Skill, and Performance Outcomes. 159 APPENDIX L - Correlations Between Knowledge and Experience With Participant Behavior, Interpersonal Skill, and Performance Outcomes. 161 APPENDIX M - HLM Analyses for Participants’ Use of Affiliation and Control Behavior Predicted By Role Player Disposition (With Personality Traits). 162 APPENDIX N - HLM Analyses for Participants’ Use of Communication Skill and Relationship Building Behaviors Predicted By Role Player Disposition (With Personality Traits).
163 APPENDIX O - HLM Analyses for Participants’ Performance Outcomes Predicted Participants’ Use of Interpersonal Behavior and Interpersonal Skills and Moderated By Role Player Disposition (With Personality Traits). 164 APPENDIX P - Summary of EFA and CFA for the MMI. 165 APPENDIX Q - Correlations Between Cognitive Ability and Personality. 168 APPENDIX R- Facets of Emotionality and Role Play Performance.
169 APPENDIX S- Interpersonal Adaptivity Rating. 170 ix LIST OF TABLES MANUSCRIPT 2 Table 1 Means, standard deviations, simple effects, and effect sizes of the mean level of agency and communion demonstrated by role players by exercise conditions. Descriptive Statistics and Correlations of Study Variables. Hierarchical Linear Models Predicting Participant Affiliation and Control Behavior.
Hierarchical Linear Models Predicting Participant Interpersonal Skill Behavior. Hierarchical Linear Models Predicting Participant Performance Outcome. Generalizability analysis for participant, station, item, and rater. CFA Fit Indices.
CFA analysis for the completely standardized estimates for the correlated uniqueness model of the 2-factor MMI model. General Descriptives and Internal Consistency Results .119 x LIST OF FIGURES MANUSCRIPT 1 Figure 1. Functional Framework of Interpersonal Adaptability. Performance outcome predicted by participant control behavior as moderated by control role player disposition.
Performance outcome predicted by participant relationship building behavior as moderated by role players’ portrayed disposition of affiliation. Performance outcome predicted by participant communication skill behavior as moderated role player’s portrayed disposition of control.83 xi APPLYING A FRAMEWORK OF INTERPERSONAL ADAPTABILITY FOR ASSESSMENT Introduction My dissertation is comprised of three separate, but conceptually related, manuscripts. The first manuscript is a conceptual paper that draws upon sections of my introduction from my proposal. It is targeted to be included as a chapter in an upcoming book.
The second and third manuscripts are empirical studies that draw upon some of the findings from my dissertation research. A general discussion was also prepared in order to highlight the key conceptual, empirical, and practical implications from my dissertation research. My dissertation aimed to contribute to the considerable literature on the assessment of individual effectiveness in social situations. One set of contributions are from my book chapter on the construct of interpersonal adaptability.
Drawing upon previous conceptual work by Klein, DeRouin, and Salas (2006) and Ployhart and Bliese (2006) I defined interpersonal adaptability as an individuals’ adjustment of interpersonal behavior, thoughts, and emotions appropriately within an interpersonal interaction in order to achieve the goals afforded by the situational demands of the interaction. The definition highlights that situational demands and goals affect how interpersonal behaviors will be related to performance within an interpersonal interaction. The book chapter “Conceptualizing and Assessing Interpersonal Adaptability: Towards a Functional Framework” will be submitted as a chapter in a book to be published on adaptability by David Chan. The objective of the chapter is to provide a 1 conceptual framework for interpersonal adaptability, which will highlight opportunities to better assess constructs included within the interpersonal adaptability framework.
The three assessment opportunities discussed in the chapter are: 1) designing more contextualized assessments; 2) improving the match between construct and method; and 3) designing dynamic assessments. The first two of these opportunities were applied to guide the empirical studies that are presented in my two empirical manuscripts. In the general discussion section of my dissertation I discuss how steps taken in my research can inform directions for future research for designing dynamic assessments of interpersonal adaptability. The empirical research for my dissertation took place at the Ontario Veterinary College (OVC).
OVC has a worldwide reputation for graduating highly qualified veterinarians (Sabitini & Conlon, 2006). In order to maintain that reputation OVC has taken steps to improve the interpersonal adaptability of graduates from the program. One of these steps has been the integration of objective structured clinical examinations (OSCE) into the training curriculum to provide students with high fidelity simulated client interactions where they can develop their interpersonal adaptability.