1. Management Plan & Timeline
To implement the proposed program, UTSA has formed a highly competent and experienced Senior Leadership Team and external Advisory Board. Both the leadership team and the advisory board are characterized by intellectual and disciplinary diversity (see letters of collaboration).
UTSA’s University College will lead this project in collaboration with faculty in the College of Engineering, College of Education and Human Development, and the College of Sciences. The University College supports the undergraduate education of the academic colleges at UTSA and reports to the Provost. Oversight for the proposed project will be included in the administrative duties of the Dean of University College (PI), who reports directly to the Provost. This will ensure the sustainability of these initiatives after the completion of the grant period.
The senior leadership team is led by Dr. Heather Shipley (Professor, Department of Civil and Environmental Engineering and Interim Vice Provost and Dean of University College) at UTSA. The co-PIs are Dr. Mark Appleford (Associate Professor Biomedical Engineering, Associate Dean of Undergraduates Studies College of Engineering,); Dr. Juliet Langman (Professor in Bicultural and Bilingual Studies, Associate Dean of Graduate Studies, College of Education and Human Development); Dr. Kelly Nash (Associate Professor and Graduate Advisor of Record, Department of Physics), and; Dr. Jorge Solis (Assistant Professor, Department of Bicultural and Bilingual Studies). Additional, senior personnel include Dr. Krystel Castillo (Associate Professor of Mechanical Engineering and Director of the Texas Sustainable Energy Research Institute (TSERI), Dr. Harry Millwater (Professor of Mechanical Engineering), and; Dr. Orlando Graves Bolaños (Lecturer, College of Education and Human Development and DoSeum Art Manager). The team has the combined administrative experience and research expertise to oversee every facet of the project and ensure its success. The senior leadership team will assist in the implementation of the proposed project’s objectives and goals; foster relevant relationships between UTSA and outside communities; manage reporting and compliance; ensure that research projects progress according to plan, and verify the sustainability of the project after the grant period has ended.
In addition to the faculty involved in the senior leadership team, faculty will be recruited from Physics, Mechanical Engineering, and Biomedical Engineering. These faculty members will have a Ph.D. in their respective areas as well as experience teaching the undergraduate courses identified for both the lower and upper division. This will provide an opportunity for these faculty members to participate in faculty coaching to undergo pedagogical changes to the courses they teach.
An external advisory board, composed of national experts in STEM and Hispanic education, will be established to provide ongoing feedback and guidance to project faculty collaborators and principal investigators on the development and refinement of the faculty coaching model and the restructuring of undergraduate courses. Year 1 of the project will focus on developing and piloting the restructured course activities and materials. As more STEM majors are incorporated into the project in Years 2-4, project team members will hold annual 2-day evaluation meetings to discuss strengths and areas of improvement where members of our advisory board will participate. The goal of these discussions will be to identify emerging ideas and principles for integrating LA-STEM practices into undergraduate courses within and across fields of study. The advisory board will also provide guidance for analyzing evaluation data, planning for university sustainability and scale-up challenges, and broader policy implications of the project. Lastly, a related Year 4 objective for the project will be to disseminate a vetted pedagogical guidebook of (e.g., anchor lesson plans, exemplary video-recorded lessons).
The proposed project is for four years and will consist of planning and development, implementation of multiple cycles of lesson study that result in redesigned STEM courses, and expansion of faculty coaching and peer mentoring models to other STEM courses, beyond the four courses outlined above. Table 4 shows the proposed timeline:
2. Evaluation Plan
The independent evaluator, Dr. Mary Martinez-Wenzl, from Education Northwest will address the first two focal evaluation questions directly and the third question will be led by the PIs, see questions below. Education Northwest is a nonprofit organization with over 50 years of experience in rigorous research and evaluation, strategic communications and dissemination, and technical support and training. Dr. Martinez-Wenzl is experienced with culturally responsive evaluation, including the evaluation of math and science instruction and has published on the subject of higher education access for Hispanic students. The evaluator will conduct a formative and summative evaluation that will provide ongoing feedback to improve program implementation and evaluate UTSA’s progress toward program goals and objectives. Moreover, the proposed evaluation questions are guided by the program activities and expected outcomes described in the proposal. The evaluation questions include:
- How do STEM faculty perceive the impact of the undergraduate/lower-division course redesign and professional development?
a. How do STEM faculty perceive the effectiveness of the course redesign using the LA-STEM framework?
b. To what extent does the professional development improve faculty knowledge of integrating language?
c. To what extent were faculty able to implement course instruction using the LA-STEM framework with fidelity? - How do students perceive the impact of the program supports on STEM career prospects?
a. To what extent do students feel their STEM courses prepared them to meet the rigor of their upper-division STEM courses?
b. How do students perceive the effectiveness of the peer mentor program?
c. How do students perceive the effectiveness of the research and internship opportunities to increase their knowledge of the STEM research process, develop marketable skills and communicate their knowledge? d. What is the relationship between instructors’ fidelity of implementation (FOI) of the reform practices in redesigned courses and the academic achievement of Hispanic undergraduates in STEM fields? - What is the association between program participation and changes in academic performance, retention rates, and graduation rates?
a. What is the relationship between instructors’ fidelity of implementation (FOI) of the reform practices in redesigned courses and the academic achievement of Hispanic undergraduates in STEM fields?
b. How do observed changes in outcomes vary by implementation fidelity?
c. What factors support and/or hinder the development and implementation of the intervention in each course?
Qualitative and quantitative data will be collected to address each question (i.e., observations of courses, faculty interviews, student graduation rates, student writing samples, course grades, anchor lesson plans, etc.). The evaluator will answer the first two questions by conducting surveys and interviews with STEM faculty and of undergraduate students. The evaluator will also conduct and analyze a sample of STEM classrooms (see Table 5).
The third question focuses on assessing the fidelity of implementation (FOI) needed to explain how the instructional changes are developed and implemented over time (Buxton, Allexsaht-Snider, Kayumova, Aghasaleh, Choi, and Cohen, 2015; Lee, Penfield, and MaertenRivera, 2009). Research on the FOI of instructional practices serves multiple purposes: 1) monitoring the strength of the intervention across courses and faculty, 2) understanding the exposure of the pedagogical intervention on Hispanic undergraduates, and 3) adjusting instructional practices over multiple implementations of a project intervention (“feedback loop”). Instructors will be observed three times during each semester, including during lesson study activities. FOI observations of university faculty will focus on qualitatively describing and quantitatively evaluating (Scale 0-3) how instruction adheres to planned lesson activities (e.g. activities/topics, course readings, curriculum materials, review/attention to course assignments) and to integrating language and STEM content areas (i.e., academic language in physical sciences). Adherence here refers to how consistent each course addresses intervention areas and to what extent participants are exposed to anchor lessons, target activities and materials, and pedagogical practices.
Data on specific indicators of student success will be collected and analyzed to assess if students are benefiting and improving due to the pedagogical changes. Performance indicators regarding the success of the program will be linked to retention and graduation rates; the research opportunities open to students; and the development of critical thinking and progress/success in the students’ professional development. The following questions and metrics (Table 6) will be used to assess the effectiveness of the program in reaching its objectives in regard to students’ success.
The data will be collected from our Office of Institutional Research and we will utilize the EAB Student Success Collaborative Campus software to allow faculty the ability to track their students’ performance for addressing Evaluation Question 3.
The PIs and instructors will use the data collected and evaluation assessment to create a feedback loop to the program activities yearly in the Spring semester. The purpose of this process is to encourage feedback, discussion, and modifications to project activities so that project objectives are met. This evaluation will determine which activities are most effective and which activities need improvement. If improvements are needed, the PIs will make modifications to the project implementation based on input from the evaluation to ensure that the program objectives are met.
3. Dissemination Plan and Sustainability
We will develop a website that will feature all materials, tools, and practices developed for this program. Therefore, a large part of our communication strategy will focus on disseminating what we have learned so that data can be compared across institutions. We will do this by attending conferences, such as the American Society for Engineering Education, and relevant professional organization conferences, such as SACNAS, HACU, the American Chemical Society, American Physical Society, and others. In addition, we will publish our findings in peer-reviewed journals and through other publications and sites that are relevant to Hispanic Serving Institutions. As relevant, we will write op-ed pieces for Inside Higher-Ed or the Chronicle of Higher Education.
Since the proposed project is led by the University College, this will ensure sustainability after its completion. The university has a major strategic initiative on student success and the lessons learned from this project will be incorporated into this initiative. Also, the STEM academic colleges will be highly encouraged to continue these efforts of curriculum redesign since they directly contribute to student success for which they are held accountable. In addition, having a representative from our local community college on the advisory board ensures communications between the two higher education institutions on how best to serve transfer students in the future and incorporate academic literacy into courses at both institutions.
4. Intellectual Merit
The project team will improve student learning and self-efficacy by (1) developing new methods of instruction and (2) implementing associated curricular changes focused on the intersection between academic literacy and content competencies through evidence-based teaching approaches and STEM faculty coaching. These proposed activities focus on the critical transition between lower and upper-level undergraduate courses, which is key to student retention and graduation in these areas. In addition, innovative cross-sector partnerships will be established between collaborating STEM and Education faculty to create interdisciplinary professional development lesson study groups of instructors that draw explicit attention to the role of academic literacy in the development of strong university-level STEM teaching and learning. Expected outcomes from the proposed work are that (1) faculty will have access to innovative, evidence-based best practice teaching approaches that enhance student learning; and (2) provide students with marketable skills to be successful in the professional world.
5. Broader Impacts
The proposed plan builds on UTSA’s and the NSF’s dedication to increasing diversity in the STEM workforce. The proposed program is an integrated approach to increasing the number of undergraduate underrepresented minorities who graduate from the UTSA Colleges of Engineering and Sciences by improving retention and graduation rates. It is anticipated that this approach will not only attract but also retain, students in STEM fields and give them the skills necessary to succeed as STEM professionals and/or graduate students. In addition, this program will increase the diversity of the STEM workforce specifically in South-Central Texas. It is expected that the program will benefit STEM undergraduates at UTSA by increasing their retention rates, critical thinking skills, professional knowledge, and self-efficacy. Taken together, this will support the timely completion of undergraduate degrees and will increase the marketability and job placement of UTSA graduates.