University of South Carolina Scholar Commons Theses and Dissertations Spring 2020 Program Evaluation of a Middle School Stem/Steam Program Warren Richard Wintrode Follow this and additional works at: https://scholarcommons.edu/etd Part of the Educational Administration and Supervision Commons Recommended Citation Wintrode, W. Program Evaluation of a Middle School Stem/Steam Program. Retrieved from https://scholarcommons.edu/etd/5924 This Open Access Dissertation is brought to you by Scholar Commons. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of Scholar Commons.
For more information, please contact dillarda@mailbox. PROGRAM EVALUATION OF A MIDDLE SCHOOL STEM/STEAM PROGRAM by Warren Richard Wintrode Bachelor of Science in Computer Science United States Military Academy, 1985 Master of Education University of South Carolina, 2006 Submitted in Partial Fulfillment of the Requirements For the Degree of Doctor of Philosophy in Education Administration College of Education University of South Carolina 2020 Accepted by: Susan C. Bon, Major Professor Edward Cox, Committee Member Peter Moyi, Committee Member Michael Seaman, Committee Member Cheryl L. Addy, Vice Provost and Dean of Graduate Studies © Copyright by Warren Richard Wintrode, 2020 All Rights Reserved.
ii ACKNOWLEDGEMENTS I would like to acknowledge the help of the students, teachers, and administrators at New Ellenton STEAM Magnet Middle School. Without their input and assistance, this dissertation would not exist. Their dedication to the multi-disciplinary science, technology, engineering, arts, and mathematics curriculum and the engineering design process gives me hope for our future. I also want to acknowledge and thank the professors who have advised and encouraged me through the dissertation process, and the four-year buildup to the degree - Dr.
Susan Bon, Dr. Ed Cox, Dr. Robert Johnson, Dr. Peter, Moyi, Dr.
Michael Seaman, and Dr. Without your wisdom and counsel, I would not have seen the process through to a successful conclusion. iii ABSTRACT Beginning with the Obama administration’s “Educate to Innovate” campaign in 2009, integrated science, technology, engineering, and math (STEM) programs have flourished in our nation’s schools. Designed to increase the number of STEM professionals in the workforce and contribute to the United States’ continued viability in the global economy, these programs promote inquiry-based, technology-driven learning in collaborative, cross-curricular projects.
Some schools have added art, and liberal arts, to the curriculum, making them STEAM programs. Middle schools are a popular home for STEM and STEAM programs, serving as the connective tissue in the K-12 STEM/STEAM “pipeline.” To date, there have been relatively few program evaluations of STEM/STEAM programs in the literature on these programs. The purpose of this study was to conduct a program evaluation of a middle school program. This program evaluation examined the role of student interest, student self-efficacy, and teacher confidence in the success of a middle school STEAM program.
The methodology of the study was a pragmatic, mixed methods program evaluation. The data collection instruments included an interview with the school principal, focus groups with the school’s teachers, and surveys of all teachers and all students. The results of the evaluation indicated that faithfully implemented over the long-term, a school-wide STEAM program may contribute to student interest in STEAM professions, student self- efficacy, teachers’ confidence in their instructional capacity, and student academic achievement. iv TABLE OF CONTENTS ACKNOWLEDGEMENTS.
iv LIST OF TABLES. viii LIST OF FIGURES. ix LIST OF ABBREVIATIONS .2 STATEMENT OF THE PROBLEM/RESEARCH QUESTIONS .4 RATIONALE FOR THE STUDY.5 RESEARCH CONTEXT/BACKGROUND .8 PARTICIPANTS AND STUDY SITE .9 DEFINITIONS OF PROGRAM EVALUATION TERMS .15 CHAPTER 2: LITERATURE REVIEW.2 THE BACKGROUND OF STEM/STEAM IN THE UNITED STATES.3 RECRUITING STUDENTS INTO STEM/STEAM PROFESSIONS .5 STEM/STEAM IMPLEMENTATION .6 UNDERREPRESENTED PARTICIPATION IN STEM/STEAM .7 STEM/STEAM PROGRAM EVALUATION .8 SYNTHESIS: SCHOOL STEM/STEAM PROGRAM EVALUATION.3 RELIABILITY AND VALIDITY .2 DESCRIPTION OF PARTICIPANTS .3 DESCRIPTION OF THE STUDY SITE .4 DESCRIPTION OF STEAM PROGRAM .5 DATA COLLECTION AND ANALYSIS .169 CHAPTER 5: DISCUSSION AND IMPLICATIONS.4 IMPLICATIONS AND RECOMMENDATIONS .5 RELEVANCE TO THE LITERATURE .6 SUMMARY AND FINAL THOUGHTS .188 APPENDIX A: DISTRICT APPROVAL OF RESEARCH PROPOSAL.199 APPENDIX B: IRB APPROVAL OF RESEARCH PROPOSAL .205 APPENDIX C: STEAM GOALS & IMPLEMENTATION FOCUS GROUP GUIDE .206 APPENDIX D: STEAM SURVEY FOR STUDENTS .208 APPENDIX E: STEAM SURVEY FOR TEACHERS .211 APPENDIX F: INTERVIEW WITH SCHOOL ADMINISTRATOR .214 APPENDIX G: CLASSROOM OBSERVATION PROTOCOL .216 APPENDIX H: PERMISSION TO USE COPYRIGHTED MATERIALS .220 APPENDIX I: TRANSCRIPT OF ADMINISTRATOR INTERVIEW .225 APPENDIX J: TRANSCRIPT OF TEACHER FOCUS GROUPS .233 APPENDIX K: CLASSROOM OBSERVATIONS .245 vii LIST OF TABLES Table 1.1 Definitions of Program Evaluation Terms .1 Paradigms, Branches, and Defining Characteristics of Program Evaluation .1 Components of the Evaluation .2 Data Collection Instruments Mapped to STEM Dimensions .1 Math Scores for Targeted Middle School .2 Science Scores for Targeted Middle School .3 English Scores (SC READY) for Targeted Middle School .4 Student Survey Results for All Students (n=240).5 Student Survey Results for Female vs.6 Student Survey Results for African American vs.7 Student Survey Results for Hispanic vs.161 viii LIST OF FIGURES Figure 1.1 Components of a middle school STEM/STEAM program (CIPPP) .1 iSTEM framework of tiered integration of STEM disciplines .2 Strands of a STEM educational evaluation .3 The dimensions of the STEM common measurement system .1 Teacher efficacy in STEM/STEAM .2 Teacher sense of student efficacy in STEM/STEAM .3 School supports for the STEAM Program.4 Supportive educator practices (teacher survey) .5 Student self-efficacy in STEAM skills .6 Student interest in STEAM learning/professions .7 Supportive educator practices (student survey) .156 ix LIST OF ABBREVIATIONS AYP.Adequate Yearly Progress CIPP. Context, Inputs, Process, and Products DDE.
Deliberative-Democratic Evaluation DE. Department of Education E3. Enrichment Experiences in Education ESEA. Elementary and Secondary Education Act GTT.
Gateway to Technology HHS. Health and Human Services I3. Investing in Innovation IEP. Individualized Education Plan Joint Committee.
Joint Committee for Standards for Educational Evaluation NSF. National Science Foundation PBL. Problem-based Learning PLTW. Project Lead the Way STEAM.
Science, Technology, Engineering, Arts, and Mathematics STEM. Science, Technology, Engineering, and Mathematics UDL .Universal Design for Learning UFE. Utilization-focused Evaluation VEE. Values-engage educative case study model x CHAPTER 1 INTRODUCTION Introduction In a presentation on the science, technology, engineering, and mathematics (STEM) program implementation process, the Southern Regional Education Board (SREB, 2017) compared the problem-based learning (PBL) process of the STEM educational model with what it described as “Edutainment.” STEM problem-based learning is inquiry-based and open-ended rather than teacher directed and highly structured.
In PBL, there may be many solutions while in a teacher-directed project there is likely one best answer. In a highly structured project, the solution is the end state of the project, while in PBL the process is the launching pad for exploration, collaboration, and communication. The SREB described the goal of an “Edutainment” project as fun, but the goal of a PBL project is engagement. This distinction between the collaboration and inquiry-focus of problem-based learning and the highly structured, teacher-centered projects we find in many of our classrooms is intriguing and worth exploring.
Statement of the Problem/Research Questions There are numerous STEM programs in the country, dedicated to training the next generation of STEM professionals. These programs often integrate art, and liberal arts, into their program of study, transforming them into STEAM schools. To date, there have been relatively few program evaluations of middle school STEM/STEAM programs, exploring the distinction between “Edutainment” and true project-based 1 learning. Therefore, it is worthwhile to conduct an evaluation of a STEM/STEAM program in a middle school, to determine and measure the factors that indicate program effectiveness.
Research points to three attributes in a STEM/STEAM program that predict program effectiveness. The first attribute is student engagement in the learning and interest in STEM and STEAM professions (Milner, Horan, & Tracey, 2014; Reiss & Mujtaba, 2017; Sjaastad, 2011). The second attribute research emphasizes in STEM/STEAM program effectiveness is the development of student self-efficacy (Sithole et al. Student self-efficacy is the perception of students that they can master STEM concepts and skills and persevere through the challenges of a STEM educational program.
The third attribute is teacher self-efficacy. Teacher self-efficacy is teachers’ confidence in their ability to provide rigorous and engaging instruction using the engineering design process to guide the students in their tasks (Lesseig, Slavit, Nelson, & Seidel, 2016; Ferrara-Genao, 2015; van den Kieboom, McNew-Birren, Eckman, & Silver-Thorn, 2013). A thorough evaluation of a middle school STEM/STEAM program will contain valid and reliable measures of the state of these three attributes in the middle school program. The following research questions guided this assessment of the selected middle school STEM/STEAM program.
What is the STEAM program’s impact on student interest in STEM/STEAM professions? 2. How has participation in the STEAM program influenced the self-efficacy of students in STEAM knowledge and skills, with a focus on females and minorities involved in the program? 3. How has professional development and participation in the STEAM program impacted teacher confidence in providing STEAM education to their students? 4. What is the STEAM program’s influence on student achievement? Purpose The purpose of this dissertation is to examine the components of STEM educational programs and use this knowledge to conduct a program evaluation of a middle school program.
In this instance, the examination involves a STEM program that includes art, and liberal arts, as a component of the program, transforming it into a STEAM program. In evaluating a middle school program, the evaluator worked with the school faculty and administration to assess program effectiveness and make recommendations for program improvement. Additionally, this program evaluation of a middle school STEAM program may contribute to the literature of evaluation of STEM and STEAM programs in its exploration of student interest and self-efficacy. Specifically, this program evaluation will be one of the first uses of the STEM Common Measurement System (Saxton et al., 2014) as a framework for organizing an evaluation of a STEM/STEAM school and will contribute to its legitimacy if it proves viable.
A middle school program is the object of this study because middle school is the connective tissue in the K-12 STEM education “pipeline.” This pipeline refers to the 3 various paths students choose as they pursue STEM courses from kindergarten through the end of high school (Gonzalez & Kuenzi, 2012). Middle school is the first instance in which career education is seriously explored, and students begin to map out their choices in education and future profession. In 2010, the President’s Council on Science and Technology (PCAST) recommended to the president several goals in STEM education, among them the establishment of 1000 STEM-related K-12 schools by 2021 (Executive Office of the President: PCAST, 2010). The distribution of schools was 200 high schools and 800 middle schools.
This weighting in favor of middle schools was intentional. It reflects the understanding that if we want to grow our STEM workforce to remain economically competitive, it will occur through our recruiting and education processes. To do this, we need to engage our youth when they are beginning to explore their future. This occurs in middle school.
Rationale for the Study To meet the challenges of recruiting, training, and producing STEM/STEAM professionals for our workforce, there is a need for further inquiry into evaluating STEM/STEAM education programs. To date, the country is making gains in the construction of STEM/STEAM schools at the middle and high school levels. There are numerous, overlapping models for STEM/STEAM instruction. The model which middle schools use in South Carolina is the Advanced Career STEM model sponsored by the Southern Regional Education Board (SREB, 2017).
Higher education programs, such as the UTeach program at the University of Texas Austin (Perez & Romero, 2014), are providing teacher education programs for science and math teachers. Other programs (van den Kieboom, McNew-Birran, Eckman, & Silver-Thorn, 2013; Sanders, 2009) 4 prepare integrated STEM teachers. There is considerable research on student interest, persistence, and self-efficacy in the K-12 “pipeline,” and in higher education, particularly regarding female and minority students. One shortfall in STEM/STEAM inquiry is in evaluation of STEM/STEAM educational programs.
To date, there have been few evaluations of middle school STEM/STEAM program effectiveness.