IC End-Term Evaluation for the Global Fund Supported Adolescent Girls and Young Women (AGYW) Program 2018-2020
The Global Fund is an innovative financing mechanism that has been supporting programs in Zimbabwe, since 2003, for prevention, treatment, and care for people with HIV/AIDS, Tuberculosis (TB), and Malaria. The primary goal of the HIV grant (2018-2020) which supported the Adolescent Girls and Young Women (AGYW) program in Zimbabwe is to reduce the rate of HIV incidence i.e. to prevent new infections and reduce HIV related morbidity and mortality in Zimbabwe. The AGYW program which started under the NMF2 grant (2018-2020) is being scaled up under the NFM3 HIV grant (2021-2023). The GF HIV resources are being used to improve access to HIV counselling and testing services; support HIV prevention through behaviour change communication and comprehensive sexuality education; expand PMTCT services; strengthen provision of community-based HIV services (including GBV prevention); and achieve universal access to HIV treatment. UNDP is the Principal Recipient of the HIV Grant in Zimbabwe.
The AGYW Programme 2018 – 2020:
Prior to the grant period 2018-2020, there was an estimated 1.3 million people living with HIV (PLHIV) in Zimbabwe (Zimbabwe DHS 2015). Adult HIV prevalence had shown a steady declining trend from 18.1% in 2005 to 13.8% in 2015 (Zimbabwe DHS 2015). HIV prevalence among children (0-14) was estimated to be at 1.6%. While the epidemic had declined among both men and women (15-49), women continued to bear disproportionate burden with prevalence levels of 16.7% compared to 10.5% among men in 2015. The same gender disparity was true for new infections, where women had an HIV incidence of 0.67%, compared to 0.28% among men aged 15-49 (ZIMPHIA Fact Sheet 2016).
Zimbabwe’s modes of transmission study of 2017 showed that the greatest number of new infections – more than 16,000 a year – occurred among never married women. Adolescent girls and young women (AGYW) in particular, experienced dramatically disproportionate burden and risk factors. Young women (20-24), for instance, had an HIV prevalence 2.78 times greater than their male peers. In addition, 17.1% of women aged 15-19 who had sex in the last year did so with a partner that was ten or more years older - up from 15.2% in 2010 and 7.5% in 2005 (WHO Epi Review 2016). In addition, 41% of girls reported sexual debut before 18 years as unwanted (ZIMSTAT) and rates of transactional sex were high and increasing (Epidemiological Review and Modes of Transmission Study 2017). The HIV prevalence among young women (18-24) with two or more transactional sex partners in the last six months was estimated at 32%, compared to 10% among those who had never had transactional sex (Zimbabwe National Sex Program – UNICEF).
Upon this background, the AGYW program was designed as part of the critical interventions to address the social and structural factors that fuel intergenerational, forced and/or transactional sex - particularly gender inequality and sexual and gender-based violence (GBV) - for preventing HIV among AGYW. Given the fact that less than half of young people in Zimbabwe were observed to be knowledgeable about HIV prevention methods (46% among women and 47% among men in 2015), improved Comprehensive Sexuality Education (CSE) was also prescribed as a critical intervention.
The AGYW Programme started in 2018 under the 3-year NFM2 HIV grant (2018-2020). It is managed, coordinated and implemented under National AIDS Commission (NAC) working closely with Plan International, and the Zimbabwe Association of Church related Hospitals (ZACH) – the Grant Sub-Recipients.
The AGYW programme focuses largely on four delivery approaches:
- The “modified DREAMS” approach, which is a layered and holistic package of services to vulnerable AGYW aged 10-24 years. The emphasis is mainly on both in-school and out-of-school AGYW, targeted for comprehensive sexuality education (CSE), social protection (keeping girls in school), provision of health care services (including HIV Testing Services, Family Planning, STI Screening, Condoms and PrEP), GBV prevention and post-violence care.
- The Sista2Sista peer group model covers 30 districts and specifically targets vulnerable girls (school drop-outs, orphans and teenage mothers) reached by comprehensive sexuality education and/or life skills–based HIV education through the Sister to Sister approach (using sister-to-sister mentors). The S2S package provides messaging and skills for HIV prevention, sexual and reproductive health (including contraception), child marriage, GBV and substance abuse. The program also provides dignity packs with sanitary pads to reduce the risk of transactional sex.
- The Start Awareness Support and Action (SASA!) model is a community mobilization intervention that seeks to prevent violence against women and reduce HIV risk behaviours. SASA! is implemented in 6 districts and is uniquely designed to address aspects of GBV and HIV/AIDS.
- The One Stop Centres (OSC) are primarily designed to provide post-violence care to victims of gender-based violence. The OSC model is implemented in 4 districts, providing increased access to holistic services including health, counselling, legal, psycho-social support and police services to survivors of GBV.
The AGYW program is delivered in partnership with existing community structures, such as youth groups and activity clubs, to ensure that the adolescent girls and young women are able to obtain and generate information on their sexual and reproductive health rights, freely and effectively express themselves, strengthen their life skills, as well as have unlimited access to legal aid and advice on their rights.
National AIDS Council: NAC is an organization enacted through the Act of Parliament of 1999 to coordinate and facilitate the national multi-sectorial response to HIV and AIDS. Under the GF Program, NAC is the Sub recipient (SR) responsible for managing and coordinating the (i) Adolescent Girls and Young Women (AGYW) and (ii) Key Populations (KP) programs.There are 4 Sub-Sub Recipients (SSRs) under NAC which include: Plan International and ZACH (AGYW); and CeSHHAR and UNFPA (KP).
The Evaluation Context
The overall objective of the End-Term Evaluation is to establish and document the effectiveness and programmatic reach of the AGYW program including activities, outputs and results since project inception in 2018 to completion in 2020. The evaluation will be guided by the 6 core Official Development Assistance (ODA) evaluation criteria: relevance, coherence, effectiveness, efficiency, impacts, and sustainability (but specifically the four OECD-DAC that are emphasized in UNDP guidelines – relevance, effectiveness, efficiency and sustainability). The AGYW program was rolled-out in 2018 under the 3-year GF HIV grant (2018-2020); and was followed by a baseline assessment in 2019. It should be noted that the baseline had some design gaps particularly in terms of reach whereby data were not adequately collected on some key indicators. The consultant will propose strategies to address these data gaps since the end-term evaluation compares progress made in achieving the underlying objectives of the project against the baseline results. The findings from the evaluation will be used by NAC and key stakeholders to inform the next cycle of grant implementation, but also to draw relevant lessons and appropriate recommendations for AGYW programs in Zimbabwe.
It is important to note that the evaluation context takes into consideration the potential impact of the Covid-19 pandemic which may have caused disruptions to programme continuation resulting into slow implementation in the primary and secondary education system, thus attendance and keeping the girls in school.
The evaluation strategy that has been designed and adopted to guide the implementation of the program since its inception includes: a) baseline assessment which was conducted in 2019; b) monthly collection and reporting of routine data by NAC through the National HIV & AIDS Activity Report Form (NARF) reporting c) quarterly reports submitted to UNDP as the PR; d) program review meetings, including SR quarterly and semester review meetings e) documentation of best practices and lessons learnt; and f) the end-term evaluation scheduled for 2021 to measure the overall impact of implementing program interventions. The evaluation design as prescribed by the potential consultant(s) must comply with the OEDC DAC Evaluation Quality Standards and the UNDP Evaluation Guidelines. The evaluation principles that will guide the end-term evaluation and to which potential consultant(s) must adhere include: independence, transparency, participatory and gender responsiveness.
The end-term evaluation will cover the period Jan 2018 – Dec 2020 and will take into consideration the project geographical areas and the relevant implementing partners. The evaluation will target (and be guided by) the various thematic/result areas of the AGYW project as stipulated by the NAC Strategic Documents including the ZNASP III and the National M&E Plan for the period. These key areas include: (i) Sista2Sista and Male Mobilization interventions; (ii) DREAMS programme (iii) SASA! and; (iv) One Stop Centres (OSC).
The end-term evaluation will assess the effectiveness of the implementation strategy and the results. This will include the implementation modalities, roles and responsibilities, coordination, partnership arrangements, institutional strengthening, beneficiary participation, replication, and sustainability of the programme. The evaluation will address how the interventions under the AGYW programme sought to strengthen the application of the rights-based approach and mainstream gender in development efforts. In addition, the evaluation will also be used to collect data on any unintended outcomes of the AGYW program during the course of the 3-year implementation period. This will be discussed during the inception meeting when the consultant is selected.
The following key questions are proposed to guide the programme evaluation:
- Relevance – (Design and focus of the programme)
- To what extent was the project in line with the national development priorities, the country programme’s outputs and outcomes, the UNDP Strategic Plan and the SDGs?
- To what extent were perspectives of those who could affect the outcomes, and those who could contribute information or other resources to the attainment of stated results, taken into account during the project design processes?
- To what extent did the programme achieve its overall objectives?
- What and how much progress has been made towards achieving the overall outputs and outcomes of the programme?
- To what extent does the project contribute to gender equality, the empowerment of women and the human rights-based approach?
- To what extent have the poor, indigenous and physically challenged, women and other disadvantaged and marginalized groups benefited from the UNDP/GF supported programs in the country?
- Were the inputs and strategies identified, and were they realistic, appropriate and adequate to achieve the results?
- Was the programme relevant to the underlying needs of the AGYW?
?2. Effectiveness – (Describe the management processes and their appropriateness in supporting delivery)
- Was the programme effective in delivering desired/planned results?
- To what extent did the programme’s M&E mechanism contribute in meeting programme results?
- How effective were the strategies and tools used in the implementation of the programme?
- To what extent were the results (outcomes and outputs) achieved?
- How effective has the programme been in responding to the needs of the beneficiaries, and what results were achieved?
- What are the future intervention strategies and issues?
3. Efficiency – (of Programme Implementation)
- Was the process of achieving results efficient? Specifically did the actual or expected results (outputs and outcomes) justify the costs incurred? Were the resources effectively utilized? - What factors contributed to implementation efficiency?
- Did programme activities overlap and duplicate other similar interventions (funded nationally and /or by other donors? Are there more efficient ways and means of delivering more and better results (outputs and outcomes) with the available inputs? - Could a different approach have produced better results?
- To what extent were partnership modalities conducive to the delivery of project outputs?
- How efficient were the management and accountability structures of the programme?
- How did the project financial management processes and procedures affect programme implementation?
- What are the strengths, weaknesses, opportunities and threats of the programme implementation process
4. Coherence – (Measuring the extent to which the intervention is compatible with other interventions within the country, the sector or institution). This involves assessing internal and external coherence to determine how other interventions, policies and programs support or undermine the AGYW program and vice versa.
- Establishing synergies and inter-linkages
- Determining consistency of the AGYW program with relevant global norms and standards
- Assessing aspects of complementarity, harmonization and value addition without duplication of efforts
5. Impact Measurement
- This involves positive and negative effects as a result of implementing project activities. The consultant (or team of consultants) will design an appropriate evaluation model taking into consideration the findings at baseline. The evaluation will include the active involvement of beneficiaries in terms of feedback (and probably as appropriate using control groups).
- Impact measurement to involve the specific details regarding changes brought about by the project, comparing the actual situation before and after the project
- The effects could be primary, medium-term or long-term
6. Sustainability and Ownership
- To what extent are the benefits of the programme likely to be sustained after the completion of this programme?
- What is the likelihood of continuation and sustainability of programme outcomes and benefits after completion of the programme?
- How effective were the exit strategies, and approaches to phase out assistance provided by the programme including contributing factors and constraints?
- What are the key factors that will require attention in order to improve prospects of sustainability of programme outcomes and the potential for replication of the approach?
- How were capacities strengthened at the individual and organizational level (including contributing factors and constraints)?
- Describe the main lessons that have emerged.
- What are the recommendations for similar support in future? (NB: The recommendations should provide comprehensive proposals for future interventions based on the current evaluation findings).
A combination of both quantitative and qualitative methods will be used for the end-term evaluation with various targeted groups / evaluation participants to obtain primary data on the key evaluation questions. A participatory approach will be adopted to appropriately gather the various perspectives of key stakeholders. Overall, the AGYW program is implemented in 30 districts (for S2S), in 6 districts for SASA! and 4 districts for DREAMS and OSC. The S2S program targets to reach 15,000 AGYWs annually. The selected consultant (or team of consultants) will be required to propose the details in terms of suitable methodology for the evaluation, and this should be well articulated in their technical proposal. Once selected and in consultation with UNDP, NAC and key stakeholders, the consultant(s) will revise the methodology and reflect this (including interview schedule, field visits and data to be used in the evaluation) in the Inception Report. After the methodology is agreed upon, the consultant(s) will develop relevant instruments to capture and analyse the data during the evaluation. The proposed methodology, tools and schedules should be gender and target group sensitive.
The end-term evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The consultant is required to safeguard the rights and confidentiality of information providers, interviewees and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The consultant must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses with the express authorization of UNDP and partners.
Duties and Responsibilities
Description of Responsibilities
Specifically, during the process of carrying out the End-Term Evaluation the consultant (or team of consultants) shall:
- Conduct the End-Term Evaluation to obtain the outcomes and impact of the project based on planned interventions. The evaluation will specifically focus on project beneficiaries, implementers, project managers, community leaders and any other relevant AGYW stakeholders.
- Undertake a desk review on available literature and project documents to identify relevant information for the final project evaluation
- Design the End-Term Evaluation methodology and tools focusing and ensuring that samples are representative of the target population groups
- Recruit and train enumerators for data collection and this should be costed and submitted as part of the proposed budget for the assignment.
- Conduct in depth interviews with potential target groups and stakeholders including project beneficiaries, implementers, project managers, community leaders and any other relevant AGYW stakeholders.
- Conduct key-informant interviews with project implementers and partners, including Adolescent Girls and Young Women, Community-Based Organizations (CBOs), Civil Society Organizations (CSOs) and Ministry of Women Affairs among others
- Conduct data analysis using appropriate methodologies and will clearly describe the methodology of data analysis in their proposal / inception report
- Write a comprehensive report based on the findings from the evaluation. The draft will be shared with a technical team before presenting to the stakeholders.
- Prepare and present the findings in a dissemination workshop organized by NAC for stakeholders.
- Use comments from the stakeholders’ workshop to revise the report and submit a final acceptable report to UNDP
Timeframe and deliverables
Time frame 69 working days over a period of 04 months
- Conceptualization of the evaluation design, literature review, methodology and detailed work plan. Inception Meeting Initial briefing & a one-day stakeholder consultation session. Submission of a Comprehensive Inception Report Deliverable - Inception Report 12 days
- Recruit and train enumerators for data collection - 12 days. Documents review and stakeholder consultations. Drafting of Data Collection Tools / analysis plan. Field Visits for data collection, evaluation Data analysis, debriefing and presentation of draft - Deliverable Draft Evaluation Report - 35 days
- Validation Workshop and Validation – Deliverable - Workshop Report, 5 days
- Finalization of evaluation report incorporating additions and comments provided by all stakeholders and Submission to UNDP through National AIDS Council NAC should receive and endorse the report – Deliverable - Final Evaluation Report - 5 Days
The following deliverables are expected. These should include:
(a) evaluation inception report including a workplan and evaluation schedule;
(b) draft evaluation report for comments;
(c) an audit trail detailing how comments, questions and clarifications have been addressed;
(d) final report (addressing comments, questions and clarifications); and
(e) presentations and other knowledge products – including relevant de-briefings.
[a] Inception report
The Consultant (or team of consultants) will prepare an inception report which will show how the assignment will be accomplished. This is to ensure that the potential consultant(s) and the key stakeholders (the ZACH, Plan International, UN women, Ministry of Primary Education, Police, the UN Joint Team on AIDS and NAC) have a shared understanding of the assignment. The inception report will include the evaluation matrix summarizing the evaluation design, methodology, evaluation questions, data sources and collection analysis tool for each data source and the measure by which each question will be evaluated. In addition, the report will include the scope of work, work plan with clear timelines, proposed schedule of tasks; activities and deliverables, with clear responsibilities for each task or product. The inception report will provide details in terms of evaluation questions that the evaluation will answer, data sources, data collection, analysis tools or methods appropriate for each data source, and the standard or measure by which each question will be evaluated. It will also provide resource requirements tied to the evaluation activities and deliverables detailed in the workplan; as well as the evaluation matrix and an Outline of the draft/final report as per UNDP guidelines – as provided in the link below.
The inception report will be submitted 12 days after starting the assignment (inclusive of the inception meeting). The consultant(s) will thereafter develop data collection tools. The tools will be reviewed and validated by a technical group to ensure that the tools reflect the data that need to be collected/questions to be answered.
[b] Draft report
The Consultant (or team of consultants) will prepare a draft report, in the appropriate format to be agreed with NAC and UNDP. The draft report will be shared with NAC and UNDP for review and comments. NAC will arrange for the consultant(s) to present the draft findings at a AGYW stakeholder workshop where inputs would be sought to inform the revision of the report, and to ensure that the evaluation meets the required quality criteria. It is anticipated that there will be three drafts (the zero draft, first draft and Final Report). The consultant(s) will then consolidate the input within 5 working days after submission of the Draft Evaluation Report. The report will be produced in English.
[c] Final Report
The comprehensive final End Term Evaluation Report should be submitted to UNDP through NAC after addressing all comments from stakeholders emanating from the draft report within the last 5 days of the consultancy. NAC may request for the report to be improved on or accept the report with changes at its discretion. The content and the structure of the final analytical report with findings, recommendations and lessons learnt covering the scope of the evaluation should meet the requirements of the UNDP Monitoring and Evaluation Policy Guidelines http://web.undp.org/evaluation/guideline/ and should include the following:
• Executive summary (1-2 pages)
• Introduction (1 page)
• Description of the evaluation methodology (6 pages)
• Situational analysis about the outcome, outputs and partnership strategy
• Analysis of opportunities to provide guidance for future programming
• Key findings, including best practices and lessons learned (4-5 pages)
• Conclusion and recommendations (4-5 pages)
• Appendices: charts, terms of reference, tools and questionnaires, people interviewed, documents reviewed
Note: The above number of pages are just indicative.
The duty station of the work is Harare, Zimbabwe.
UNDP will contract the consultant (or team of consultants) on behalf of NAC to undertake the End-term Evaluation. UNDP will hire and manage the contract to ensure that the assignment is delivered in line with UN values, whilst NAC, through the M&E Director’s office shall provide overall supervision in line with the core objective of the assignment. All contractual issues will be handled by UNDP.
- Ability to research and write at a high level; and
- Ability to produce high quality outputs in a timely manner while understanding and anticipating the evolving client needs;
- Strong organizational skills;
- Ability to work independently, produce high quality outputs;
- Sound judgment, strategic thinking and the ability to manage competing priorities;
- Demonstrates integrity by modelling universal rights, values and ethical standards;
- Displays cultural, gender, religion, race, nationality and age sensitivity and adaptability.
Required Skills and Experience
Key Qualifications (Lead Consultant):
• Advanced University degree (Masters and Above) in Public Health, Epidemiology, Development Studies, or any of the relevant Social Sciences.
- At least 7-10 years’ experience in conducting programme evaluations of HIV prevention / treatment programmes aimed at Key Vulnerable Populations (KVPs), specifically young people (experience with AGYW strongly desirable).
- Experience in Health Promotion, social mapping and Reproductive health programming as well as evaluations.
- Evidence of experience in terms of (resumes, work samples, references) will be expected to support claims of knowledge, skills and experience.
- Extensive knowledge of the Zimbabwe national HIV sector;
- Experience in the usage of computers and office software packages (MS Word,
- Excel, etc.) and advanced knowledge of spreadsheets, databases and statistical packages
Please note that the end-term evaluation will require the evaluators’ independence from any organization(s) that have been involved in designing, executing or advising any aspect of the intervention that is the subject of the evaluation.
- Excellent written and spoken English;
- Understanding of Shona and Ndebele will be an added advantage.
Key Qualifications (Other Technical Expert – Statistician):
- Advanced University degree (Masters and Above) in Statistics, Mathematics, Public Health, Epidemiology, or Demography.
- Experience of at least 5 -7 years in progressively responsible work experience in designing, collection, compilation analysis and dissemination of statistical data.
- Experience in designing programme evaluations, modelling and/or measuring impact for health related programmes is desirable.
- Advanced knowledge in managing statistical databases using contemporary statistical programs and software packages such as R, STATA, SPSS, EPIINFO, ODK and others.
- Extensive knowledge of the Zimbabwe national health sector;
- Excellent written and spoken English;
- Understanding of Shona and Ndebele will be an added advantage.
Other team members may include specialists in Gender, Education and Community Development.
How to Apply:
Scope of Price Proposal and Schedule of Payments
The consultant (or team of consultants) shall be paid the consultancy fee upon completion of the following milestones.
- 20% after adoption of the inception report
- 30% after presentation and adoption of the draft report
- 50% after the approval of the final report
Recommended Presentation of Offer Interested bidders must submit the following:
- Duly accomplished Letter of Confirmation of Interest and Availability using the template provided by UNDP;
- Personal CV or P11, indicating all experience from similar projects, as well as the contact details (email and telephone number) of the Candidate and at least three (3) professional references;
- Brief description, separately or as part of the CV, of why the individual considers him/herself as the most suitable for the assignment, highlighting specialized background advantages for the scope of work and requirement.
- Technical Proposal that details the evaluation design, methodology, data analysis and presentation
- Financial Proposal that indicates the all-inclusive fixed total contract price, supported by a breakdown of costs, as per template provided.
Criteria for Selection of the Best Offer
Submissions will be evaluated in consideration of the Evaluation Criteria as stated below:
The offer will be evaluated by using the Best value for money approach (combined scoring method).
Technical proposal will be evaluated on 70% whereas the financial one will be evaluated on 30%. Below is the breakdown of technical proposal on 100% which will be brought to 70%:
Criteria for Selection:
- Presentation of evaluation design and methodology 40%
- Advanced University degree in Public Health, or any relevant Social Sciences 3 %
- Experience in Health Promotion, social mapping and development of baseline studies for adolescent girls and young women 3 %
- Experience of conducting program evaluations within the health sector focusing on Key Vulnerable Populations (KVPs), specifically young people (experience with AGYW strongly desirable). 10%
- Experience conducting advocacy on Sexual and Reproductive Health Rights. 5%
- Knowledge of the Zimbabwe national Health / HIV sector; 5%
- Experience in the usage of computers and office software packages (MS Word, Excel, etc.) and advance knowledge of spreadsheet and database packages 2%
- Fluency in both written and oral English 2%
Financial proposal 30%
All relevant strategic documents and reference materials will be provided to the consultant(s) in soft copy upon signing of the contract. These will include, but not limited to:
(a) Zimbabwe National Strategic Plan for HIV/AIDS – ZNASP III & IV
(b) The National M&E Plan and the Results Framework 2015 – 2020
(c) ZIMPHIA Fact Sheet 2016
(d) Zimbabwe Demographic and Health Survey 2015
(e) Epidemiological Review and Modes of Transmission Study 2017
(f) National Baseline Survey on the Life Experiences of Adolescents (NBSLEA – ZimStat)
(g) National AIDS Council Mini-M&E Plan 2018 – 2020
(h) The AGYW and KP M&E Plan 2021 – 2023
(i) Key stakeholders and partners as referenced in NAC Mini Plan and ZNASP III
(j) Documents to be reviewed and consulted
(k) Evaluation Matrix Template
(l) Outline of the evaluation report format
(m) Code of conduct forms