© Copyright 2019 Helen Kilber College Is A Big Investment. Does Receiving College Scorecard Information Help Students Evaluate Which One Offers The Best “Bang For The Buck”? Helen Kilber A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2019 Reading Committee: William M. Zumeta, Chair Crystal C. Long Program Authorized to Offer Degree: Evans School of Public Policy and Governance University of Washington Abstract College Is A Big Investment.
Does Receiving College Scorecard Information Help Students Evaluate Which One Offers The Best “Bang For The Buck”? Helen K. Kilber Chair of the Supervisory Committee: William M. Zumeta Evans School of Public Policy and Governance College Scorecard is a U. Department of Education website providing information on the average costs and returns, in terms of average earnings after leaving college, associated with all U.
colleges receiving student aid under Title IV of the Higher Education Act. The Obama Administration launched the Scorecard to help students and their families identify the colleges offering them the “best bang for the buck”, and also to encourage colleges to compete on price and quality. Limited evidence to date suggests that College Scorecard’s impact, although in the right direction, is very small, in large part because students do not seek it out. In a randomized control trial (N= 322), I find that providing well-presented summary cost, earnings and graduation rate information similar to that in College Scorecard directly to high school seniors leads them to rank colleges significantly more in line with the institutions’ expected financial returns.
Compared to a control group informed only of colleges’ names, location, sector (public or private) and type (2- or 4-year), students receiving scorecard information perform better at ranking alternative four-year public and private colleges and also in evaluating the choice between starting directly at a four-year college versus pursuing the community college two- to four-year transfer route, a common choice for low-income students in particular and one that decreases their chances of attaining a four-year degree. Using a separate, experimental design involving a think-aloud protocol (N = 31), I directly investigated how scorecard information affects individual students’ decision processes. I find that the framing of the information treatment itself acts as sufficient “advice” for students who use it to learn how to evaluate colleges’ financial investment value. The majority of students shift from heuristic-decision making focused on sector-based cost perceptions to a weighted approach including estimations of future earnings and graduation probabilities.
A large minority of students fail to improve their decision process, however, and continue to rank the colleges based on cost. These results suggest that improving the framing of the scorecard information treatment could thus increase its effect on students’ ability to evaluate the returns to alternative college options. A particular challenge identified is how to effectively present information on the probability of successfully navigating the community college transfer pathway to a four-year degree. This probability is considerably lower than most students think it is.
Future research in the experimental information-updating econometrics literature should consider the impact of relaxing the assumption that students respond to information on the costs and returns of college options by updating the accuracy of their existing estimates. Not doing so could lead this literature to overemphasize the role of non-financial preferences and underestimate the potential impact of effectively designed information interventions revealing, not just the true costs and returns associated with colleges, but also how to use that information in decision-making. TABLE OF CONTENTS Chapter 1. COLLEGE SCORECARD OVERVIEW AND POLICY CONTEXT.1 WHAT IS COLLEGE SCORECARD? .2 THE POLICY CONTEXT: BIG GOALS FOR INFORMATION TRANSPARENCY4 1.1 The Obama Administration – A Partially Achieved Vision .2 The Trump Administration Agenda – College Scorecard to the Fore.
THE RELEVANT LITERATURE ON INFORMATION AND COLLEGE CHOICE 7 2.1 STUDENTS AS ILL-INFORMED BUT RATIONAL HUMAN CAPITAL MAXIMIZERS .1 Do students know the true value of variables relevant to estimating the investment value of alternative colleges? .2 In practice, do students make choices like rational human capital investors? .3 What is the impact of providing accurate information on college costs and returns? 9 2.4 Implications for College Scorecard’s Likely Effectiveness.2 STUDENTS AS COGNITIVELY LIMITED, HEURISTIC DECISION-MAKERS .1 Students as Heuristic Decision-Makers .2 Hoxby and Turner’s Expanding College Opportunities Study .3 Implications for College Scorecard .3 STUDIES EVALUATING THE IMPACT OF COLLEGE SCORECARD .4 WHERE TO FROM HERE?. RESEARCH QUESTIONS AND DESIGN .1 OVERALL RESEARCH DESIGN .2 PRESENTING COLLEGE SCORECARD INFORMATION TO MAXIMIZE IMPACT .1 Comparison with Other Information Treatments .2 The Choice Situations. DATA COLLECTION INSTRUMENTS, PROCESSES AND SAMPLE CHARACTERISTICS .1 WRITTEN SURVEY INSTRUMENT DESIGN .2 The Ranking Tasks .2 “THINK ALOUD” PROTOCOL .1 December 2014 Data Collection Round, Written Survey - High School Seniors .2 December 2015 Data Collection Round, “Think Aloud” Interviews – High School Seniors.1 Sample Size and Demographics.2 Low-income School (December 2014 and December 2015 data) .3 Sample Characteristics: High-income School (December 2014 data only). THE IMPACT OF SCORECARD INFORMATION ON RANKING ACCURACY 46 ii 5.1 MEASURING STUDENTS’ ACCURACY .1 Reference Ranking Method .2 Reference Ranking Attributes .3 Ranking Accuracy Variables .4 Reference Ranking Sensitivity Analysis .1 Students’ Ranking Patterns .2 Changes in Ranking Accuracy, Summary Measures .3 ANOVA ANALYSIS: DID OVERALL ACCURACY IMPROVE? .4 MANOVA ANALYSIS: DIFFERENTIAL EFFECT OF SCORECARD INFORMATION ON RANKING ACCURACY .1 Evaluation of Support for Hypotheses .2 Unpacking The Black Box of Causation.
STUDENTS’ DECISION PROCESSES .1 PRIMARY ANALYTICAL STRATEGY.2 DECISION PROCESSES IN AJ’S TASK: EVALUATING FOUR-YEAR COLLEGES .1 Themes in the Basic Information Group .2 Themes in the Scorecard Information Group.3 Structured Analysis of Students’ Decision Rules and Ranking Patterns .4 Scorecard Information Changed Students’ Decision Processes .5 Was heuristic decision-making in the basic group a pragmatic response to lack of information? .6 Were students sophisticated interpreters of scorecard information? .3 DECISION PROCESSES IN JJ’S TASK: EVALUATING THE COMMUNITY COLLEGE PATHWAY .1 Themes in the Basic Information Group .2 Themes in the Scorecard Information Group.3 Students’ Community College Views and Indicator College Ranking Patterns.5 GENERALIZABILITY OF RESULTS. WHAT HAVE WE LEARNED? .1 The Impact of a Scorecard Information Intervention on Student Enrollment Decisions 111 7.2 Implications for Econometric Models of College Choice .3 Increasing the Impact of College Scorecard as an Information Intervention .4 Final Thoughts on College Scorecard. COLLEGE SCORECARD EXAMPLE. THINK ALOUD PROTOCOL, DECEMBER 2015.
REFERENCE RANKING SENSITIVITY ANALYSIS. QUALITATIVE CODING METHOD FOR 2015 THINK-ALOUD STUDY. COMPARISON OF RANKING PATTERNS, 2014 AND 2015 LOW INCOME HIGH SCHOOL. COLLEGE SCORECARD OVERVIEW AND POLICY CONTEXT “… tomorrow, my administration will release a new “College Scorecard” that parents and students can use to compare schools based on a simple criterion -- where you can get the most bang for your educational buck.” President Obama, State of the Union, 2013 1.1 WHAT IS COLLEGE SCORECARD? College Scorecard is a website, launched by the Obama administration in February 2013, which provides a standardized page of information on every college that receives student aid under Title IV of the Higher Education Act.
public, private non-profit and for- profit post-secondary institutions covered by College Scorecard range from less than two-year certificate granting, largely for-profit, institutions, to public and private two-year colleges, and to four-year private and public baccalaureate degree-granting institutions. Figure 1: College Scorecard Mobile Web Site Pages for University of Washington Campuses 1 1 Note that College Scorecard does not differentiate salary outcomes by campus. 1 As the quotation at the beginning of this section suggests, College Scorecard’s main focus is on providing a standardized, reliable set of information on each college so that students can compare the ‘bang for the buck’ offered by alternative colleges. The Scorecard highlights three key pieces of data - average annual cost to the student, graduation rates and average salary 10 years after leaving (see example data for the University of Washington as of 2015 shown in Figure 1 above).
Average annual cost to students: This is the net price for first-time, full-time students who receive federal financial aid. Net price is the total cost of attendance, less average per-student federal, state, local, or institutional aid for students receiving federal aid. This information is drawn from the Integrated Post-Secondary Education Data System (IPEDS) maintained by the National Center for Education Statistics.2 Graduation rate: This is the cohort graduation rate for first-time, full-time students and is the official measure of institutional graduation rates mandated by the Higher Education Act. It measures the fraction of first-time, full-time students from a given entry cohort who complete their program of study within 100, 150, or 200 percent of the ‘normal’ completion time.
For example, the 150 percent completion rate measures the fraction of the cohort that graduates within six years for students pursuing a four-year degree, or three years for students pursuing a two-year degree. College Scorecard reports six-year cohort graduation rates for all four-year (i. baccalaureate-granting) colleges and four-year graduation rates for all other colleges. 3 Salary after attending: This is the median earnings of students who received federal financial aid 10 years after leaving college, regardless of whether the student successfully completed a degree or certificate.
This data comes from earnings data from administrative income tax records maintained by the Department of the Treasury. Earnings are defined as the sum of wages and 2 All data definitions are from U. Department of Education (2015) and Executive Office of the President (2015). 3 Most of these other colleges are two-year-degree granting institutions, thus College Scorecard reports the percentage of their students who graduate within 200 percent of normal completion time while for four-year colleges it reports the percentage who graduate within 150 percent of normal completion time.
2 deferred compensation from all W-2 forms received for each individual, plus self-employment earnings from Schedule SE.4 Although these three pieces of information are highlighted, College Scorecard users can click through to get more information on costs, financial aid and student debt, graduation and retention, earnings after school, student body characteristics, and academic programs offered at each college. Table 1 below summarizes the data elements available. Elements in italics are those highlighted in the summary view of College Scorecard in Figure 1 above. Appendix 1 provides a full College Scorecard for the University of Washington.
4 Sources for data definitions: Better Information for Better College Choice & Institutional Performance (U. Department of Education, 2015) and Using federal Data to Measure and Improve the performance of U. Institutions of Higher Education (Office of the President, 2015). 3 Table 1: College Scorecard Data Elements Costs Graduation and Retention Earnings After School Average annual net price for First time, full time six-year Median earnings of former federal aid recipients, after graduation rate (four-year students who received all government and college colleges) or four-year financial aid, 10 years after grant aid*(1).
Compared to the Average annual net price by % of students who, 6 years national average. family income level(2 )for aid after enrolling, earn more recipients. % of full time, first time than an average high school undergraduates who return graduate Link to the college’s net after their first year*. Financial Aid and Debt Student Body Characteristics Academic Programs % of students who have Number of undergraduates Most popular programs repaid at least $1 of the Full time/part time share Available areas of study principal of their federal loans within 3 years of Race/ethnicity proportions leaving college* Percentage of students % of students receiving receiving a Pell grant Federal loans SAT and ACT score range Median federal debt of (25th to 75th percentile) undergraduate borrowers who completed college Median monthly loan payment for students who completed * The national average is also provided.
(1) In-state net price is given for public colleges.