A theory and process evaluation of the Learner Engagement Programme (LEP) implemented by Just Grace NPC - Kwenda, Geraldine (Department of Statistical Sciences, 2023)
Background The Learner Engagement Programme (LEP) is an after school dropout prevention programme that operates in Langa, a township located in Cape Town, South Africa. Langa is an impoverished township characterised by socio-economic challenges, such as high unemployment, violence, few economic opportunities and poor school infrastructure. The LEP aims to address learner disengagement of at-risk high school learners. It operates within the only five high schools in Langa: Langa High, Khulani High, Isimela High, Zimasa High, and Ikamva High. The programme is implemented by Just Grace, a non-profit organisation whose goal is to uplift the youth and community of Langa through educational, community and youth development programmes. Just Grace is funded by Trusts and Companies such as DGMT, Mineral Loy (Pty) Ltd, Swiss Philanthropic Foundation, Capfin (Pty) Ltd, Enigma Electrical (Pty) Ltd, Lot Emphangeni (Pty) Ltd, Mergon Foundation, Dairycap CC. Aims of the evaluation The evaluation aimed to determine: (a) the extent to which the programme design can realistically bring about the desired outcomes and (b) the extent to which the programme's planned activities are implemented with fidelity. A programme theory evaluation and process evaluation were carried out to address the following evaluation questions: Programme theory evaluation questions 1) What is the theory and logic underlying the LEP? 2) Is the programme theory and logic plausible? Process evaluation questions 1) Is the programme consistently servicing the planned target population? a. To what extent are learners appropriately identified as at risk? b. Which support services are used the most by learners? c. Are the programme services relevant to meet the learners' needs? 2) Are initial home visits being delivered according to planned programme procedures? 3) Are the programme staff adequately trained and equipped to work with at-risk learners and implement the programme's different components? Methodology The choice of methods for this evaluation was informed by the evaluation questions as well as by the practical opportunities and constraints associated with Level 3 lockdown in South Africa in response to the COVID-19 pandemic. Access to programme beneficiaries was restricted during this time and the risks and concerns associated with face- to- face contact compelled the evaluator to capitalise on available secondary data sources and data gathered from programme staff (through a focus group) to address the process evaluation questions The programme theory evaluation was guided by Donaldson's (2007) systematic five-step framework in conjunction with Brouselle and Champagnes (2011) steps for a logic analysis. An initial LEP programme theory was developed using data obtained through a structured engagement with a purposive sample of four programme staff and a review of relevant programme documents. The plausibility of the programme theory was then examined in line with the best practice literature. Brousselle and Champagne's (2011) steps for a logic analysis were applied to guide process, which culminated into a reconstructed programme theory. To answer the process evaluation questions 1-3, programme documents were systematically analysed. A focus group, which gathered programme staff's experiences of and insights into the current programme infrastructure, challenges, and organisational support, was also conducted. The focus group data was analysed using Krueger's (1994) framework for thematic analysis. Key Findings The programme theory evaluation confirmed that the LEP initial programme theory and logic was plausible: the programme does incorporate a multi-level approach to tackling learner disengagement, targeting the individual, family and community. It addresses the psychosocial aspects that contribute to learner disengagement through the provision of one on one counselling, a life skills programme and parental support groups and training. The programme also has elements of an effective after school programme with qualified staff, adequate resources and efficient programme practices. A few shortcomings were identified through the evaluation: the programme lacks academic support, early warning systems, specialised external partners, and behavioural outcome measures, which are crucial in preventing school dropout. While the evaluator cannot conclusively determine whether the initial home visits are being delivered according to planned procedures (given the limitation of the data at hand), the process evaluation confirmed that the programme the criteria used to identify at risk learners are in line with the best practice literature. The process evaluation also revealed factors that compromised the effective implementation of the programme, including lack of commitment from partner schools, lack of trust in programme methods from parents/caregivers and a lack of staff safety when conducting home visits. Recommendations Key recommendations discussed in this evaluation include the following: • Development of an early warning system in tandem with partner schools as data on atrisk learners needs to be collected earlier in their school career and consistently to ensure the learner receives the necessary assistance timeously and suitable interventions are developed. • Provision of an academic component as learners who are provided with academic support in addition to psychosocial support have a higher chance of school completion. • Development of a behavioural monitoring system as effective programmes utilise behavioural outcome measures to assess programme effects on learners' behaviours • Forging partnerships with external agencies to assist the programme in specialised areas as the programme would benefit by being embedded in a broader network of community-based organisations, NGOs, civil organisations, and government agencies trained to provide specialised support and assistance to their beneficiaries.