After-School All-Stars
PSA is currently conducting a mixed-methods evaluation of the 21st Century Community Learning Center (CCLC) program (funded by the U.S. Department of Education) for After-School All-Stars (ASAS) Cleveland. ASAS is a nonprofit organization supporting a national network of chapters that provide comprehensive school-based afterschool programs to at-risk, primarily middle-grades students. The ASAS Cleveland evaluation uses academic data, stakeholder surveys, program attendance data, focus groups, interviews, and program observations to examine the outcomes and quality of implementation of a 21CCLC afterschool program operating remotely during COVID-19. PSA works in collaboration with ASAS Cleveland to leverage evaluation findings to advance continuous improvement and to report on the program’s efficacy in meeting its federal grant requirements.
Previously, PSA conducted 21CCLC evaluations for ASAS in Philadelphia and Newark, and engaged in a formative evaluation of an afterschool site in the District of Columbia. Additionally, PSA partnered with the ASAS national office to develop an implementation rubric aligned with the theory of change for the ASAS program model, and to support ASAS in developing a performance management strategy and approach to evaluating the impact of its national network.
New York City Department of Youth and Community Development
Since 2005, the Out-of-School Time (OST) Programs for Youth initiative (now COMPASS) of the New York City Department of Youth and Community Development (DYCD) has provided access to OST services through one of the largest afterschool systems in the country. For the first three years of the initiative (2005-06 through 2007-08), PSA’s evaluation examined the implementation and operations of these OST programs, and analyzed participants' educational and social development in light of the duration and intensity of their OST participation and the features of their OST programs. In later phases, PSA evaluated DYCD's efforts to improve OST program quality, including:
In-depth studies of strategies to support program quality in OST programs serving elementary- and middle-grades youth, including studies of the implementation of skill-based programming, social-emotional learning, literacy and STEM activities, and of the role of Education Specialists in continuous program improvement.
Evaluation of the Transition to High School programs, designed to facilitate the successful transition into high school for students entering the ninth grade.
The development of a program-quality measurement tool to be used by DYCD staff in monitoring the quality of its programs, and training of agency staff on the use of this tool for coaching and assessing programs.
Evaluation of a pilot initiative to build the capacity of OST provider organizations through intensive technical assistance focused on fiscal management, human resources management, and program quality.
A 10 year retrospective analysis of participation
Strategies for Social-Emotional Learning in COMPASS Programs
Longitudinal Analyses of the DYCD Transition to High School Program: Cohorts 1 and 2
Promising Practices in Planning and Implementing COMPASS Literacy Programming
Retrospective Analysis of COMPASS Participation, 2004-2015
Role of the Education Specialist in Supporting OST Program Quality
Lessons from the Field: DYCD OST Summer Programming
Supporting STEM Activities in Out-of-School Time Programs
Principal Perspectives on DYCD Out-of-School Time Programs
Building the Capacity of Nonprofits to Support High-Quality Youth Programs
Approaches for Integrating Skill-Based Activities in Out-of-School Time Programs
Achieving Program Quality in the DYCD Out-of-School Time Initiative: Strategies from 15 Programs
Evidence of Program Quality and Youth Outcomes in the DYCD Out-of-School Time Initiative
PSA OST Observation Instrument
PSA designed the Out-of-School Time (OST) Observation Instrument to collect consistent and objective observation data about the quality of afterschool activities. Grounded in a youth development framework, the PSA OST Observation Instrument is based on two core assumptions about the features of high-quality afterschool programs: (1) certain structural and institutional features (adequate qualified staff, adequate space and resources, and the availability of varied opportunities for academic and interpersonal skill-building) support the implementation of high-quality programs; and (2) instructional activities promote positive youth outcomes if they offer varied content, mastery-oriented instruction, and positive youth-to-youth and youth-to-staff relationships. PSA collected data on the psychometric properties of the observation instrument from studies in which the instrument has been used. Evidence from several of these studies has been reported in a report of the instrument's reliability and validity. The PSA OST Observation Instrument is available for use by researchers and practitioners. Please contact Christina Russell at crussell@policystudies.com or 202-939-5340 with any questions.
OST Observation Instrument: Report of the Validation Study
OST Observation Instrument-Procedures
OST Observation Instrument
4-H Science, Engineering and Technology Initiative: Youth Engagement, Attitudes, and Knowledge Study
PSA conducted a multi-part evaluation of the implementation of the 4-H Science Initiative, which engaged youth in high-quality informal science educational opportunities built around hands-on inquiry to increase the number pursuing post-secondary education in scientific fields. Through interviews and surveys of state- and county-level staff, and program site visits, PSA examined state- and county-level implementation, delivery of science programming, youth engagement and promising practices and developed a “Priming the Pipeline: Lessons from Promising 4-H Science Programs” case study, a practice-oriented document that the National 4-H Council shared nationally.
Boys & Girls Clubs of America
PSA evaluated multiple initiatives of the Boys & Girls Clubs of America. For the Be Great 9th Grade Cohort Project, supported by AT&T, a mentoring-based program to support students in their transition to high school, PSA analyzed: (1) how Clubs implemented and adapted the 9th Grade Cohort Project; (2) strategies Clubs used to collaborate with schools in support of 9th Grade Cohort participants; (3) the relationship between program design, implementation, and youth outcomes; (4) strategies Clubs used to support productive, stable relationships between youth and mentors; (5) factors that appeared to encourage or discourage Club members’ participation in 9th Grade Cohort activities; (6) possible improvements to the design and delivery of the 9th Grade Cohort; (7) specific model components that posed implementation challenges; and (8) issues related to sustainability and scale-up. For Project Learn, a BGCA partnership with the David and Lucile Packard and Orrin Ingram Foundations to develop learning centers in 18 Clubs in eight Club organizations in Northern California and in Middle Tennessee, PSA assessed the implementation of high-yield learning activities, relationships with schools and parents, training and technical assistance, and sustainability. PSA also conducted a four year evaluation of CareerPrep that explored the strategies for and barriers to implementing each of the program’s key components: targeted recruitment, pre-employment training, case management, job placement, and follow up).
YMCA of the USA After-School Upgrade
PSA conducted a formative evaluation of the YMCA of the USA’s (Y-USA) Afterschool Upgrade pilot initiative, a program upgrade designed to enhance the delivery of afterschool program services in YMCA Associations across the country. Using qualitative methods, the evaluation explored the implementation and effectiveness of the Afterschool Upgrade and provided recommendations for the supports, guidance, and refinements for Y-USA to consider at a national level and at the Y Association level as it considers taking the Afterschool Upgrade to scale across the Y movement. The evaluation also explored the conditions and factors that facilitated or hindered the adoption of the Afterschool Upgrade, including elements of Y business models conducive to change. PSA provided formative feedback to Y-USA staff throughout the evaluation process, presented a briefing of preliminary findings to inform the delivery of technical assistance to pilot participants, and produced a final report with case study examples that the organization used to inform the implementation of best practices at scale.
New Hampshire Department of Education 21st CCLC
PSA worked with the New Hampshire Department of Education to design and launch an ongoing performance-management and evaluation strategy for the state’s 21st Century Community Learning Centers program. This was a collaborative effort with NHDOE and stakeholders to build a strategy and system that NHDOE could continue to administer internally to generate consistent information about the ways in which programs are implemented and contribute to desired academic and life skills outcomes for students. The evaluation team helped the NHDOE identify promising practices in the 21st CCLC system, learn more about the relationship between the implementation of specific program elements and the achievement of outcomes, define quality benchmarks, and offer targeted supports and resources for improving program quality. In addition, the evaluation system developed generated center-specific data that can be used by 21st CCLC programs in their own evaluation and reporting efforts.