PME 802 - Module 2B (Steps 3 & 4)

Step 3 - Program Theory

The program theory for this evaluation follows the "if/then" format outlined and recommended by Wilder Research.  Dr. Sharpe noted that "A program theory consists of a set of statements that describe a particular program, why, how and, under what conditions the program effects occur, predict the outcomes of the program, and specify the requirements necessary to bring about the desired program effects." (Sharpe, 72).  Furthermore, Wilder Research reminded me that "clarity and plain language are essential" since "a clear program theory help[s] others see the sense of your program." (Wilder Research, 1).  "Appendix B: Program Theory Development" in the Wilder Research article is also quite helpful for organizing my thinking and ensuring all the necessary activities, results, and evidence are included.  Keeping all of this in mind my program theory for Beyond the Bell is as follows:

If children attend the Beyond the Bell program, then they will receive academic support in the form of tutoring, as well as opportunities to experience new activities that will support their overall development and wellness.  If the children receive academic support in the form of tutoring and experiences that support their development and wellness, then they will show growth in their academic achievements at school and have a higher self-esteem and a more positive self-image as evidenced by increased grades, improved report cards, and anecdotal observations from teachers, school staff, program staff, family members, and self-reflection.

Logic Model

"A logic model is a representation - generally a picture - of how your initiative or programme is supposed to work.  It explains why what you are going to do is a good solution to the issue you are addressing...It focuses on explaining the logic behind the process." (Social Policy and Evaluation Research Unit, 15).  For my logic model I followed the template provided by SUPERU in "Making Sense for Evaluation: A Handbook for Everyone."  The following screenshot from the Handbook shows the graphic and descriptions/definitions that kept me on track while creating my own logic model.

Retrieved from: "Making Sense for Evaluation: A Handbook for Everyone." by Social Policy Evaluation and Research Unit.  Available: https://dpmc.govt.nz/sites/default/files/2018-03/Evaluation%20Handbook%20Dec%202017.pdf


Here is my Logic Model for the YMCA Beyond the Bell program.


You can also click HERE to view a larger version of this document.


Step 4 - Evaluation Method

Beyond the Bell's goal is to close the achievement gap for children in lower income families.  Simon Hearn talked about the importance of clearly defining terminology and expectations to avoid identification gaps and misunderstandings later on.  Keeping that in mind it makes the most sense to define achievement as academic achievement specifically in numeracy and literacy.

After watching the "Approaches to Evaluation" video by Education at Illinois it was obvious that "Evaluation for learning and use" best fit the needs of the Beyond the Bell program.  The purpose of this type of evaluation is to contribute to the quality of the program and maximize the usefulness of the program in an effort to determine how well the program is sticking to its theory of change and addressing the needs of a diverse group of program participants.  Given that the goal of Beyond the Bell is to work with each child to address and improve their individual literacy and numeracy needs this approach seems like the best fit.

Furthermore, multiple entry evaluation, as described by Chen, would be more effective than a single entry evaluation so that data could be collected at the beginning of a program session, throughout the program, and then upon completion of the program as well.

That being said this evaluation will focus on process evaluation and impact evaluation (as described in the Types of Evaluation article by the CDC).  Process evaluation will be beneficial to implement as soon as the program begins in the form of a pre-survey (as discussed in the logic model).  The impact evaluation allows for evaluation during AND at the end of the program.  By collecting data throughout the program you are able to pivot as necessary to improve the experience and outcomes for the participants.  Collecting data at the end (post-survey) allows the evaluators to determine if the Beyond the Bell program had a lasting impact on students' achievement and growth.  If the answer is yes, funding would be easier to secure for the following year and few changes would need to be made to the program.  If the answer is no, it allows the YMCA to make necessary changes before the next cohort of kids begins the next session.


Evaluation Questions (Updated)

The purpose of the evaluation is to determine if the Beyond the Bell program is effectively closing the achievement gap in schools for lower income children.  As a result of this goal my updated evaluation questions are:

1.  Does the program address, support, and align itself with the individual needs (academic, behavioural, special needs, etc.) of each student to promote overall growth?

2.  Does the program track and gauge numeracy and literacy needs and improvements of individual students based on the Ontario curriculum?  How is this done?

3.  Does the program take individual student interests into consideration when planning lessons and special activities?

4.  How does the program collect, analyze, and track data for internal purposes (program development and improvement) and external purposes (continued funding and program promotion/advertisement)?


Sources:

CDC. (n.d.). Types of Evaluation.  Available: https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf

Chen, H.-T. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage Publications 

Donaldson, S. I., & Lipsey, M.W. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In I. Shaw, J. Greene, & M. Mark (Eds.), The Handbook of Evaluation: Policies, Programs, and Practices (pp. 56-75). London: Sage. 

Greene, Dr. Jennifer. (2017) "Approaches to Evaluation."  Available: https://www.youtube.com/watch?v=zpwB_nSv6HM&feature=emb_imp_woyt

Hearn, Simon. (2013).  "Define What is to Be Evaluated." Available: https://www.betterevaluation.org/en/events/coffee_break_webinars_2013

Sharpe, Dr. Glynn. (2011).  "A Review of Program Theory and Theory-Based Evaluations." in American International Journal of Contemporary Research. vol. 1, no. 3.  Available: http://www.aijcrnet.com/journals/Vol_1_No_3_November_2011/10.pdf

Social Policy and Evaluation Research Unit. (2017).  "Making Sense of Evaluation: A Handbook for Everyone."  Available: https://dpmc.govt.nz/sites/default/files/2018-03/Evaluation%20Handbook%20Dec%202017.pdf

Wilder Research. (2009).  "Program Theory and Logic Models".  Available: http://www.evaluatod.org/assets/resources/evaluation-guides/logicmodel-8-09.pdf

YMCA SWO. (2020) "Beyond the Bell".  Available: https://www.ymcaswo.ca/beyond-bell



Comments

Post a Comment

Popular posts from this blog

PME 802 - Module 3B (Steps 5, 6, & 7)

Step 5 - Data Collection & Analysis Methods The data collection for this evaluation is a combination of qualitative and quantitative approaches.  A variety of data collection methods is most beneficial because "different methods reveal different aspects of the program." (Taylor-Powell & Steel, 6).   Furthermore, the CDC writes that "e mploying multiple methods (sometimes called “triangulation”) helps increase the accuracy of the measurement and the certainty of your conclusions when the various methods yield similar results." (CDC.gov).   Literacy & Numeracy Checklist:  " Checklists are used to encourage or verify that a number of specific lines of inquiry, steps, or actions are being taken, or have been taken, by a researcher." (Given, 2008).  One of the Beyond the Bell program goals is to increase literacy and numeracy skills.  An important element would be ensuring the skills taught in the program align with the Ontario curriculum s...
Read more

PME 802 - Program Evaluation Design (Steps 1 & 2)

Program Evaluation Design - Beyond the Bell Step 1 General Overview: I will be evaluating the Beyond the Bell Tutoring Program.  This program is run by the YMCA with the goal of " break[ing] the cycle of poverty by closing the achievement gap experienced by low-income children. This program takes a holistic approach to the well-being of every child." (YMCA SWO).  It runs throughout the school year from September to June and supports elementary aged students in grades 1-6 with homework assistance, individual learning needs, social skills, health and wellness, and other fun and creative activities.  Students attend for approximately two hours after school, 2 or 3 days a week.  There is no cost to families as it is funded by a number of community sponsors such as the school board, community foundations, individual donors, and the Ontario Trillium Foundation. The program currently runs at four locations in my area.  Two schools are in the public school board, one s...
Read more