Identifying quality in afterschool programs

By Fan Kong - June 2011

This study is a summary of the review of the research literature of afterschool quality frameworks. It presents the debates on program effectiveness to help organizations, policymakers, funders, and evaluators make decisions about afterschool programming. This review is of help to ISE educators and program directors in understanding the current trends in outcomes-based programming.

The authors summarized eight quality frameworks of program effectiveness, ranging from qualitative to quantitative reviews—including three meta-analyses—to define program quality. In conducting a review of this kind, they acknowledged that program effectiveness varies greatly depending on the context, type, and goals of the program. Therefore, the authors focused on understanding the dimensions of quality rather than the variability of programs.

Based on the quality frameworks reviewed and considering their common themes, the authors identified six common domains of program quality:

– safe, supportive, and positive relationships – focused, intentional programming – strong partnerships with families, schools, and community organizations – youth engagement in program – a healthy, safe environment – management practices that support program sustainability and continuous program improvement.

To complement this picture further, the authors also took into account the current stage of research in the afterschool field. They observed that understandings of afterschool program quality are primarily based on a mix of expert opinion, research from related fields (such as youth development), correlational studies, and occasionally a few quantitatively rigorous studies. To answer the question how or why programs work, authors had, until recently, turned to correlational studies. However, these studies could not establish causal links to determine which program practices led to desirable outcomes. Process data, on the other hand (as opposed to outcomes data), help give context to the afterschool program with information such as the characteristics of participants, the broader program context, and what actually takes place in the program, and thus offer a picture of program quality from the field.

According to the authors’ literature review, these six domains represent a common understanding of program quality in the afterschool field. For practitioners in the ISE field, these six domains can help guide decisions on staff training and implementing program improvements. Moreover, as informal settings often face limited resources, program directors may necessarily prioritize certain dimensions over others, and a common framework can help determine the degree to which a program is likely to produce positive youth outcomes.

Quality frameworks reviewed include:

Beckett, M., Hawken, A., & Jacknowitz, A. (2001). Accountability for afterschool care: Devising standards and measuring the adherence to them. Santa Monica, CA: RAND Corporation. Retrieved March 28, 2008, from http://www.rand.org/pubs/monograph_reports/2007/MR1411.pdf.

Bodilly, S., & Beckett, M. K. (2005). Making out-of-school time matter: Evidence for an action agenda. Santa Monica, CA: RAND Corporation. Retrieved May 20, 2008, from www.rand.org/pubs/monographs/2005/RAND_MG242.pdf.

The Colorado Trust Afterschool Initiative. (2002). Cultural competency: The role of afterschool programs in supporting diverse youth. Denver: The Colorado Foundation for Children and Families. Retrieved July 30, 2008, from http://www.coloradofoundation.org/pdf/Cultural_Competency_Best_Practices_Booklet_Jill.pdf.

Durlak, J. A., & Weissberg, R. P. (2007). The impact of afterschool programs that promote personal and social skills. Chicago: Collaborative for Academic, Social, and Emotional Learning. Retrieved March 28, 2008, from http://www.casel.org/downloads/ASP-Full.pdf.

Harvard Family Research Project (HFRP). (2006). Afterschool program quality assessment: Categories of standards. Cambridge, MA: Author. Retrieved March 28, 2008, from http://www.gse.harvard.edu/hfrp/ content/projects/afterschool/conference/summit-2005-standards.pdf.

Little, P. M. D., Wimer, C., & Weiss, H. B. (2008). Afterschool programs in the 21st century: Their potential and what it takes to achieve it. Issues and Opportunities in Out-of-School Time Evaluation, 10, 1–12. Retrieved February 25, 2008, from http://www.hfrp.org/content/download/2916/84011/file/OSTissuebrief10.pdf.

Metz, R. A., Goldsmith, J., & Arbreton, A. J. A. (2008). Putting it all together: Guiding principles for quality afterschool programs serving preteens. Philadelphia: Public/Private Ventures. Retrieved July 20, 2008, from http://www.ppv.org/ppv/publications/assets/234_ publication.pdf.

Yohalem, N., & Wilson-Ahlstrom, A., with Fisher, S., & Shinn, M. (2009). Measuring youth program quality: A guide to assessment tools (2nd ed.). Washington, DC: The Forum for Youth Investment, Impact Strategies, Inc. Retrieved January, 14, 2009, from http://www.policys- tudies.com/studies/youth/Promising%20Programs%20Fi nal%20Report%20FINAL%2010-23-07.pdf.