| Role objectives |
The independent evaluation will focus on the 2nd phase of the Stabilization Facility project which started 1 January 2019 and cover the period from 1 January 2019 to 31 December 2020. The evaluation will be conducted over a duration of 68 days with pre-identified number of working days for each evaluator under “ duration of work section page 7 & 8. . The evaluation must address two main issues: the extent to which the Facility was able to adapt to changes in operating context brought on by the fighting in Tripoli in early April 2019 which persisted for more than one year and on the impact of the Covid-19 pandemic. The SFL went through an in-depth review in 2018 which led to an approach for the 2nd phase of the Facility (1 January 2019 – 31 December 2021). The evaluation should address a) the shift in the project outcome in the new phase to the promotion of stronger legitimate and internationally recognized state authorities and national unity, b) how the introduction of local stabilization goals to the Theory of Change has made a difference in the project implementation; and c) how far the changes in project structure and methodology set out in the 2nd phase project document have been reflected in practice and what reasons underlay that outcome.In assessing the Project, the evaluation will take into consideration the following evaluative dimensions and questions:
- The validity of the Design and Relevance:
The extent to which the project strategy, proposed activities and expected outputs and outcomes are justified and remain relevant to beneficiaries’ assessed needs, country’s policies and donor’s priorities. More specifically, the relevance of the project should be assessed through the following questions:
- To what extent is the intervention strategically relevant to fulfil its political objective as stated in the Project Document?
- To what extent are the revised objectives for the 2nd phase of the Facility still relevant given the continued political instability, Libya’s financial crisis, and the impact of COVID-19 pandemic?
- Is there evaluation evidence which indicates that SFL’s work on locally based stabilization (though an instrument like the SFL) is relevant to support political settlements?
- Has the project put in place a mechanism to understand and design specific interventions to address the issues of women, girls and boys?
- Is the SFL’s Theory of Change – national and local – valid? What changes, further assumptions or additional risk management might be required to give it greater purchase?
- Has the SFL provided a relevant response to the COVID crisis?
- Efficiency of resource used
The extent to which the project resources (funds, expertise/human resources, time, etc.) are optimally used and converted into intended outputs. More specifically, the efficiency of the project should be assessed through the following guiding questions:
- To what extent has the SFL adopted the changes in structure, processes and methodology set out in the project document for the 2nd phase?
- To what extent has SFL adapted to the COVID emergency to ensure efficiency in implementation?
- To what extent have SFL’s communications efforts contributed to efficient implementation? Are the resources for communication (with donors, with Government, with Libyan citizens, and with donor taxpayers) appropriate? Has the project communicated the perspective of women in their role in peacebuilding? Has the project been able to communicate benefits for women and girls?
- Is SFL’s risk matrix fit for purpose? Is the risk management approach appropriate?
Does the risk matrix include or distinguish issues as relates to women in their role in peacebuilding and the specific risks faced by women and young girls?
- Effectiveness
Effectiveness the extent to which the project’s expected outputs and outcomes are being achieved or are expected to be achieved. Factors contributing to or detracting from the achievement of the project desired results and objectives should also be included in the assessment. More specifically, the effectiveness of the project should be assessed through the following guiding questions:
- Is the sequence of Output 3-Output 2-Output 1 being reflected in practice? Is it working effectively? Are there trade-offs (e.g. with speed of visible delivery) which might merit revisiting the concept or its implementation? Has the SFL’s revised approach to Output 3 been sound in principle? Has it worked? What factors have made it more or less effective?
- What factors have influenced the ability of SFL to deliver its results? Has the impact of the Covid-19 pandemic affected implementation of project activities? To what extent has SFL adapted and responded to any factors affecting its implementation? Has the project put in place a mechanism to understand and design specific interventions to address the issues of women, girls and boys?
- To what extent has the SFL provided a relevant response to the COVID crisis? In what ways has the pandemic affected overall implementation?
Has the project responded effectively to the specific needs of women, girls and boys, especially with regards to the identification of stabilization goals and the identification of priority investments in support of stabilization goals (Output 1)?
- Impact and Sustainability of the Project.
In assessing the impact and sustainability of the Project, the evaluation will look at the positive and negative changes produced by the Project’s development interventions, directly or indirectly, intended or unintended. It will also include the positive and negative impact of external factors, such as changes in terms of peacebuilding and reconciliation conditions.On sustainability, the evaluation will measure the likeliness of project’s results continuity after donor funding has been withdrawn. Some of the key questions will include:
- To what extent did the Project contribute to the advance on reconciliation and dialogue among the community leaders in Libya?
- To what extent was sustainability considered in the planning and execution of the Project’s activities? To what extent is there evidence of sustainability of results?
- To what extent are the SFL’s approach and results likely to be replicated and scaled up by national partners?
- To what extent has the Project modified its approach to respond to the needs of the municipalities to address the Covid-19 pandemic?
- Have the resources of the SFL been sufficient to contribute to meeting the needs of the municipalities regarding Covid-19?
- Has the project been able to efficiently deliver interventions to address Covid-19 pandemic?
- Proposed Methodology
Based on UNDP guidelines for evaluations (UNDP Handbook on Monitoring and Evaluating for Results and the UNDP Guidelines for Outcome Evaluators), and in consultation with UNDP Libya CO, the evaluation will be inclusive and participatory, involving all principal stakeholders into the analysis. The evaluation will consider the social, political, security and economic context which affects the overall performance of the outcome achievements. During this evaluative exercise, the evaluation team is expected to apply the following approaches for data collection and analysis.The evaluation will be conducted by the evaluation team composed of an International Evaluation Consultant/Team Leader and National Evaluation Consultant (team member)The National Evaluation Consultant will bear responsibility for providing support to the International Evaluation Consultant in conducting the final project evaluation. This entails: providing support in designing the evaluation process according to this terms of reference; rigorous desk review; gathering data from different sources of information; analysing, organizing and triangulating the collected information; responding to comments and factual corrections from stakeholders and incorporating them, as appropriate, in subsequent versions; and making briefs and presentations ensuring the evaluation findings, conclusions and recommendations are communicated in a coherent, clear and understandable manner.The proposed evaluation methodology consists of a preliminary desk review of Project materials and deliverables and review of existing information relevant to the Project context; preparation of an Inception report followed by field visits; preparation of a draft Evaluation Report, debriefing with stakeholders; and Final Evaluation Report write-up. An integral part of the Final Project Evaluation will also be a concept note and a set of recommendations on directions of the next phase of the Project.The National Evaluation Consultant is required to assist the International Evaluation Consultant in evaluating the Project Document, progress, annual and mid-year Project reports, UNDP Quality Assurance assessments, key Project deliverables and other relevant documents. As needed, the National Evaluation Consultant will accompany the International Evaluation Consultant at the meetings with representatives of the main project stakeholders for an initial briefing as well as for the debriefing at the end of the assignment.Furthermore, the National Evaluation Consultant is expected to support International Evaluation Consultant in interviewing the Project team, partners as well as other stakeholders as needed. To assess Project performance, approach and modalities, the Consultants will meet with key Project partners, namely the Ministry of planning and selected officials from the local governments (Mayors and municipality council), in addition to other local partners and beneficiaries so as to obtain critical feedback and information on the Project activities and results and assess the Project performance and its approach and modalities. Each visit will allow for meetings with 2 – 3 partners/stakeholders in the respective region. The Consultants will also meet with representatives of other agencies active in the field of stabilization and local community development to assess their cooperation and level of synergies with the SFL, if needed. During these meetings, it would be important to record and accumulate inputs necessary not only for the Project evaluation, but also to inform potential design of a next project phase.The National Evaluation Consultant is encouraged to propose additional data collecting and analysis methods to ensure better insight into and review of project achievements and may use additional approaches such as simple questionnaire-based data collection. .The Final Evaluation Report will capture the feedback by key Project partners and stakeholders as well as observations and conclusions by the Consultants. The Final Evaluation Report will particularly highlight the recommendations.
- Data Collection, Data Review and Analysis:
The evaluation methodology will adhere to the United Nations Evaluation Group (UNEG) Norms & Standards. The evaluation will be carried out by an independent evaluation team. The evaluation team should adopt an integrated approach involving a combination of data collection and analysis tools to generate concrete evidence to substantiate all findings. Evidence obtained and used to assess the results of UNDP support should be triangulated from a variety of sources, including verifiable data on indicator achievement, existing reports, evaluations and technical papers, stakeholder interviews, focus group discussions (FGD), surveys and site visits where/when possible. It is expected that the evaluation methodology will comprise of the following elements:
- Review documents (Desk Review);
- Interviews with project staff, present and past;
- Interviews with key stakeholders including government line ministries, development partners, civil society and other relevant partners through a participatory and transparent process;
- Consultations with beneficiaries through interviews and/ or focus group discussions;
- Survey and/ or questionnaires where appropriate;
- Triangulation of information collected from different sources/methods to enhance the validity of the findings.
The evaluation is expected to use a variety of data sources, primary, secondary, qualitative, quantitative, etc. to be extracted through surveys, storytelling, focus group discussions, face to face interviews, participatory methods, desk reviews, etc. conducted with a variety of partners. A transparent and participatory multi-stakeholder approach should be followed for data collection from government partners, community members, private sector, UN agencies, multilateral organizations, etc.Evidence will be provided for every claim generated by the evaluation and data will be triangulated to ensure validity. An evaluation matrix or other methods can be used to map the data and triangulate the available evidence.In line with the UNDP’s gender mainstreaming strategy, gender disaggregation of data is a key element of all UNDP’s interventions and data collected for the evaluation will be disaggregated by gender, to the extent possible, and assessed against the programme outputs/outcomes.Due to travel restrictions imposed globally and internally by Covid-19 pandemic, the majority of the work will be done remotely using different mediums (Zoom, WhatsApp, Microsoft teams, etc.) to conduct the evaluation. FGDs in Libya will be limited in number in order to conform with country restrictions on public meetings and gatherings. Special Note: As of 11 March 2020, the World Health Organization (WHO) declared COVID-19 a global pandemic as the new coronavirus rapidly spread to all regions of the world. Travel to the country has been restricted since 2 March 2020 and travel in the country is also restricted. If it is not possible to travel to or within the country for the evaluation then the evaluation team should develop a methodology that takes this into account the conduct of the evaluation virtually and remotely, including the use of remote interview methods and extended desk reviews, data analysis, surveys and evaluation questionnaires. This should be detailed in the Inception report and agreed with the Evaluation Manager. If all or part of the evaluation is to be carried out virtually. Consideration should be taken for stakeholder availability, ability or willingness to be interviewed remotely. In addition, their accessibility to the internet/ computer may be an issue as many government and national counterparts may be working from home. These limitations must be reflected in the evaluation report. Since a data collection/field mission is not possible for the international consultant, then remote interviews must be undertaken through telephone or online (skype, zoom etc.). International consultant will work remotely with national evaluator support in the field if it is safe for him/her to operate and travel. No stakeholders, consultants or UNDP staff should be put in harm’s way and safety is the key priority.One qualified and independent national consultants will be hired to contribute to the evaluation and interviews in country as long as it is safe to do so.
- Composition and qualification of the evaluation team
Members of the mission must be independent to the project’s formulation, implementation or monitoring phases. It is proposed that the evaluation team be composed of a senior team leader (international), assisted by a national consultant. The national consultant will perform the following tasks;Task 1 – Provide support in desk reviewFollowing the initial meeting, the National Evaluation Consultant will support the International Evaluation Consultant in conducting a detailed review of all relevant Project documents produced during its implementation. Documentation includes but is not limited to: Project document; Project annual work plans; Project reports; monitoring records, conceptual and methodological papers and analyses; empirical studies, data on implementation of CSOs/NGOs financed from the Project grant funds. During the desk review, the focus will be on evaluating the Project baseline, indicators and targets, as well as the relevance, quality and adequacy of Project approach versus its objectives and intended outputs., the National Evaluation Consultant may also provide some assistance in preparation of an Inception report.Task 2 – Provide support in evaluation process. Upon the approval of the Inception report and the evaluation work plan by the UNDP, the National Evaluation Consultant will assist the International Evaluation Consultant in conducting the evaluation of the Project, via direct interviews with key stakeholders and beneficiaries, in person if security permits or via telephone. UNDP will provide support in organization of meetings and interviews. Once the interviews are completed, the National Evaluation Consultant will help in data analysis and collection of information (qualitative and quantitative), as inputs for the International Evaluation Consultant draft of the evaluation report , including main findings and a concept for a possible follow-up direction. This evaluation report should include a contextual analysis of the environment in which the Project is working and its relevance in fulfilling a role in that environment . The report shall seek to assess Project progress, efficiency and adequacy; process and level of success of partnership building and ownership over Project products and results; the quality of Project deliverables. The report should include the data, inputs and analysis, as well as success indicators used, and an overview of the effectiveness of the Project from the perspective of various stakeholders. The evaluation will also capture the efficiency of Project organisation and management. The draft report will contain the positive or negative, intended or unintended, changes brought about by the Project and identify factors which facilitated or impeded the realization of intended objectives.The national consultant and international consultants will create a google drive for the inception and draft reports to be reviewed by the UNDP team and Evaluation Reference Group members.. The minimum structure of the evaluation report (to be written in English language) is the following:
- Executive summary;
- Introduction;
- Methodological approach;
- Evaluation findings against the main evaluation criteria;
- Lessons learned
- Main conclusions and recommendations;
- Concept note outlining key priorities to be addressed in a possible project future phase;
- Annexes.
A debriefing session will be organised with representatives of UNDP and relevant stakeholders, so as to present the preliminary findings and recommendations of the Final Evaluation Report.Task 3 – Provide support in consolidation of the Final Project Evaluation Report Following the briefing session, the National Evaluation Consultant the on the draft evaluation report, received from project stakeholders. Also, any observations that may arise from the evaluation will be incorporated into the final report. The International Evaluation Consultant will prepare a final Project Evaluation Report, capturing findings and recommendations on both the Project approach and performance.Duration of WorkThe detailed evaluation workplan will be agreed upon between the UNDP and the evaluation team. The Project evaluation will require 20 working days for the national consultant. The national Consultant is expected to commence the assignment on 1st April 2021 (subject to restrictions and conditions imposed by the Covid-19 pandemic). The assignment and final deliverable are expected to be completed by the evaluation team, no later than 31 May 2021, with the detail as described in the below table signifying the tasks of the team leader and the national consultant.
| Activity |
National consultant |
Inception report on proposed evaluation methodology, work plan and proposed structure of the report.
|
|
| Briefing to UNDP on inception report for agreeing methodology |
|
| Desk review of existing documents, interviews, and preparation of guidance for national consultant |
Provide support in desk review
5 workdays
|
| Data collection and interviews in the country |
Provide support in evaluation
10 workdays
|
| Draft evaluation report |
Provide support in consolidation of the Final Project Evaluation Report
3 workdays
|
| Debriefing with UNDP |
|
| Elaboration of the final report (incorporating comments received on first drafts) and the set of recommendations |
1 workday |
| Presentation to SFL Board |
1 workday |
| |
20 |
|
|
