End of Program Evaluation - DFID INTAJ Economic Development & Social Stability

Summary of the Position:

Financed by: United Kingdom’s Department for International Development (DFID) Conflict, Security and Stability Fund (CSSF)

Type of Position: Consultancy

Main Location: Beirut, with potential travel to programme locations across the Bekaa, Mount Lebanon, Nabatiyeh, North and South Governorates

Duration: Up to 40 days

Dates Anticipated: February - March 2018

1. Introduction

Working in Lebanon since 1993, Mercy Corps has been running community development programs focused on promoting economic development and increased opportunity for disadvantaged communities, while responding to emergencies as they have emerged. The program portfolio has expanded in response to the massive refugee influx from Syria. To date, more than one and a half million refugees have fled to Lebanon. As numbers have continued to increase, humanitarian conditions are deteriorating. The burden on host communities and the financial and emotional stress on refugees and their hosts are tremendous. Mercy Corps is responding to these needs by addressing urgent basic needs, child protection as well as the psychosocial impact of the conflict and displacement. At the same time, Mercy Corps’ programmatic focus is on mitigating social tensions between hosts and refugees, particularly over economic resources, by addressing both livelihood opportunities, conflict mitigation capacity and local government capacity. The Mercy Corps Lebanon team consists of over 180 staff working across five offices throughout Mount Lebanon, South Lebanon, Tripoli and the Bekaa Valley.

With funding from the United Kingdom’s Department for International Development (DFID) Conflict, Security and Stability Fund (CSSF), Mercy Corps launched a two-year programme, following a successful eight-month pilot (August 2015 - March 2016), to implement economic development strategies that are potentially scalable and transferrable. The program, Improved Networks, Training and Jobs (INTAJ) in Lebanon, aims at improving the stability and resiliency in target communities in the Beirut, Bekaa, Mount Lebanon, Nabatiyeh, North and South Governorates with the following core activities:

• Support demand-driven, practical skills training and workforce-skills development for individuals.
• Offer small and medium enterprise (SMEs) specialized technical assistance, limited in-kind grants and linkages to existing business support services to lower risks and strengthen the ability of companies to grow their workforce.
• Improve efficiency and quality of solid waste management and recycling (SWM/R) value chain by building new market-driven relationships and incentivizing further investments in upgrading the value chain.

The current two-year programme was initiated in April 2016 and runs through the end of March 2018. Mercy Corps aims to complete an evaluation of the programme before potentially extending to continue to provide support to potential skills trainees, local SMEs and solid waste management and recycling market actors in future years. The aim of this evaluation is to assess key lessons learned, effectiveness in achieving outcomes (and, where possible, impact), and an assessment of the project’s theory of change towards ultimately achieving its intended impact.

2. Background

The INTAJ programme has three main components:

● Improving the employability of individuals: INTAJ engages private sector companies and/or non-governmental organizations (NGOs) to provide demand-driven, technical and professional skills for jobseekers, across a variety of sectors, to increase their employability and link them with employment opportunities.

● Supporting existing businesses to improve operations and increase their workforce: INTAJ supports small and medium enterprises (SMEs) - particularly those offering labour- intensive products in high demand - with potential for growth by providing a mix of specialized technical assistance, limited in-kind grants and linkages to existing business support services to strengthen the ability of companies to improve their operations and grow their workforce. In addition, we are supporting a small number of local Business Service Providers to increase their capacity to support local business in the North and Bekaa regions.

● Improving the value chain for solid waste management and recycling (SWM/R) in Lebanon to improve efficiency and quality, and create ‘green’ jobs: INTAJ supports the development of the SWM/R sector and encourages job growth by working with private sector companies and/or NGOs and municipalities on waste sorting at the household level, as well as waste collection and processing by supporting businesses that process recyclables. INTAJ implements multiple interventions along the SWMR/ value chain, such as: provision of technical assistance to SWM/R SMEs; strengthening UoMs and Municipalities by supporting them to strengthen their infrastructure and increasing ability of municipalities to address the needs of the community related to SWM/R; and supporting partnerships between local governmental and non-governmental entities to implement awareness and sorting at source activities. In addition, INTAJ has launched community events through small funds provided to local entities to implement events within the communities in support of enhancing solid waste management. These events have included clean-up of specific areas, concerts and zero landfill days. The main goal of these events is to raise awareness and to bring different groups to work together for a common goal.

Through the project:

● 2400 individuals will receive market-driven skills training and linkages to employment. To date, around 2100 individuals have either been trained or are in the process of completing a training. Of those who have fully completed their training, over 1,500 have reported an increase in confidence in economic future.

● 955 individuals will gain full-time, part-time, seasonal jobs or internships/apprenticeships. To date, 548 (259 female, 289 male) individuals have been placed in jobs under Skills, 154 (76 female, 78 male) individuals have been employed under SMEs and 116 individuals have been employed under SWM/R. Jobs created under SWM/R include, sorters, truck drivers and collectors with minimal managerial and administrative positions (11 female, 105 male).

● 6-8 business service providers will increase their capacity to support local businesses.

● 120 SMEs will receive support to improve their operations and expand their workforce. To date, around 100 SMEs have either completed or are in the process of receiving support.

● 76 businesses will have increased economic confidence. To date, 58/70 endlined SMEs have reported increased confidence in economic future. In addition, 42/70 endlined SMEs have reported an increase in revenue and/or decrease in cost. Under SWM/R, 22 out of 24 endlined actors have reported an increase in confidence in economic future.

● 22 SWMR value chain actors will expand their commercial networks. To date, SWM/R interventions have established over 60 documented partnerships among private sector, civil society and government market actors. 47 municipalities have been engaged and around 6 UoMs as well as 8 CBOs.

● 40,000 households will have increased access to solid waste management and recycling. To date, around 28,000 households have received services through the SWM/R interventions.

The programme’s Theories of Change are:

• If job creation and placement occur alongside building transferable, market-driven technical skills and market linkages, then individuals are more likely to secure steady, productive employment;
• If Lebanese feel confident about their economic future, then they will be less likely to have negative perceptions regarding other communities, and be less likely to engage in hostile acts;
• If individuals, municipalities, civil society and local businesses work together to address a common crisis or problem, like solid waste management, then perceptions of the other and local government will improve.

The INTAJ team has strived to take a rigorous approach to analysis and application of learning throughout implementation through the development of a series of ‘evidence papers’. These are analytical papers that aim to critically assess progress/ achievements against intended goals and drive incorporation of learnings into continued implementation.

3. Evaluation Objectives

The evaluation has two broad objectives:

1. Assess the degree to which INTAJ has achieved its intended outcomes and, where possible, impact
2. Provide recommendations on possible improvements to programme design, approach and delivery for the next phase of programme implementation (FY 2018-19).

These objectives are outlined in greater detail below.

1. Assess the degree to which INTAJ has achieved its intended outcomes and, where possible, impact
   1. Assess the degree to which impact has not yet been achieved or cannot yet be determined, is the programme on track to achieving impact;
   2. Assess the degree to which the observed changes (both in terms of development and stability outcomes and impact) be attributed to the INTAJ programme.
   3. Assess the extent to which a gender lens was applied within programme design and subsequent implementation of programme activities
   4. Analyse programme against key quality and performance standards (see definitions below for economy, efficiency, effectiveness, cost-effectiveness, relevance, impact, sustainability, adaptiveness, and influence)
2. Provide recommendations on possible improvements to programme design, approach and delivery for the next phase of programme implementation (FY 2018-19)

The intended audience for this evaluation is Mercy Corps’ Lebanon and Global teams, as well as DFID, the Social Stability and Livelihoods Sectors, and programme stakeholders where appropriate/ relevant.

4. Evaluation Design

A. Evaluation Questions: The following questions are oriented primarily around DFID’s 3Es Framework, complemented by several other criteria important to Mercy Corps. This list provides an indicative framework for evaluation questions to guide the design of the Final Evaluation. It is expected that the Consultant comment on these in their proposal, and that a final set of evaluation questions be developed in coordination with Mercy Corps during the evaluation inception phase.

1. Has the problem that the programme aims to address been correctly and adequately defined?
2. Are the inputs appropriate towards producing the intended outputs?
3. Are the inputs sensitive to local needs and nuances?
4. To what degree has conflict analysis, gender sensitivity, and general context awareness been incorporated into project design and implementation?
5. Have the outputs been appropriately identified to addressing the problem?
6. To what extent has the project ensured buy-in or political will for the uptake and use of project outputs?
7. Do the assumptions that underlay the project theory of change hold true in each implementation context and amongst the project target beneficiary group?
8. To what extent has the project achieved its intended outcomes (both in terms of development and stability)?
9. Are the anticipated outcomes appropriately defined and measurable?
10. To what extent has the project achieved its intended impact?
11. And to what extent is the achievement of impact (or outcomes) attributable to programme support versus contribution or influence from alternative actors or explanations?
12. What ‘unintended’, whether positive or negative, impacts can be identified?
13. To what extent can programmatic outcomes be sustained beyond the life of the programme? (With consideration to institutionalization, shifts in enabling environment)
14. How effective was the programme’s beneficiary selection/targeting?
15. How robust was the programme’s monitoring, evaluation and learning approach?

Stemming from analysis of the final evaluation questions, the Consultant shall provide a summative analysis of the following evaluation criteria:

• Economy (Are we or our agents buying inputs of the appropriate quality at the right price? (Inputs are things such as staff, consultants, raw materials and capital that are used to produce outputs)
• *Efficiency (*the strategic and efficient use of resources to achieve the program results, how well coordinated the program was with local government, other NGOs and communities themselves)
• Effectiveness (whether the program achieved the results anticipated – in line with the logframe, indicators and targets)
• Cost-effectiveness (How much impact on poverty reduction does an intervention achieve relative to the inputs that we or our agents invest in it?)
• Relevance (do the objectives of the intervention correspond to the needs of beneficiaries and stakeholders, how well the program ensured local stakeholders feel ownership of activities and feel engagement in the program design and implementation)
• Sustainability (the continuation of program benefits after the end of funding)
• *Impact (*the positive, negative, primary and secondary, direct and indirect impacts of the program, measuring the program against a do no harm framework, taking into account the impact of the program on women, men, girls and boys of different nationalities)
• *Adaptiveness (*how well did the program team demonstrate a capacity to adaptively manage the program and respond to changes in context, feedback from stakeholders and beneficiaries, and evidence on effectiveness and impact of program interventions)
• Influence (have the project interventions influenced other actors in the domain to change their way of operating)

B. Scope: The evaluation would be expected to include a review of relevant programme documents, as well as appropriate primary data collection as per the methodological design of the evaluation. The consultant will have access to the following materials:

• Proposal narrative
• Logical framework
• Programme organizational chart
• Programme datasets (organized by component)
• Selection criteria (organized by component)
• Reports submitted to donor
• INTAJ Evidence Papers
• Notes from INTAJ Year 1 Lessons Learned
• Notes from INTAJ Complex Program review
• DFID Stability Framework
• Guidelines for writing Evaluations (Mercy Corps’ internal document)
• Adaptive Management Guidelines and ADAPT Scan methodology (Mercy Corps internal documents)

As needed following review of existing programme documents, the evaluation may include additional data collection from programme stakeholders, including, but not limited to, the following levels:

• Programme team
• Training providers
• Trainees
• Service providers
• SME owners
• SWM/R market actors (including SME owners, CBO managers, volunteers, municipalities, Unions of municipalities etc.)

C. Methodology: The prospective supplier should propose a detailed methodological framework with substantiating explanation for the choice of methods for conducting the evaluation. The approach should incorporate mixed methods of data collection to ensure robust results while being mindful of the challenges of data collection in the given context. The proposed methodological framework should therefore anticipate challenge to data collection while ensuring rigour for robust results. Particular emphasis should be provided on how the proposed approach will attribute results to programme support, or determine the contribution of the programme to observable changes.

5. Deliverables: The deliverables expected in the undertaking of this evaluation include, but are not limited to, the following:

• Inception report, detailing design (methodology), approach, and work plan
• Set of research instruments (data collection tools, implementation plan, ethical guidance framework, participation request protocol, etc.)
• Draft report, with findings and analysis,
• Presentation of findings as reflected in draft report;
• Final report, with edits based on inputs provided in draft review.

6. Timeframe: It is anticipated that the evaluation be conducted over a total period of two (2) months, to begin from February, with the submission of the draft report no later than mid-April.

7. Team Composition, Qualifications, Roles and Responsibilities: Mercy Corps is looking for a consultant or team of consultants to work on the final evaluation. Final division of roles and responsibilities will be dependent on the team proposed. Where one consultant has all skills required, as well as thematic experience in Economic Development; Value Chain Development/ Market Facilitation approaches; and/or Governance & Social Stability programming, their application will also be considered if applying without other team members.

Required expertise includes extensive experience in conducting development evaluations, with particular expertise in Economic Development and Governance & Social Stability programmes, especially in fragile, complex, and conflict-affected environments. In particular, the following specific areas of experience are essential:

• Experience leading and implementing mixed methods evaluations for multi-year projects;
• Experience in conducting evaluations with resulting in clear recommendations on project design, implementation, and scope improvements;
• Understanding of and ability to use a range of evaluation approaches in line with accepted best practice, with demonstrable experience of implementing evaluations of varying design, methods and approaches;
• Thematic expertise in Economic Development; Value Chain Development/ Market Facilitation approaches; Governance & Social Stability;
• Demonstrated report writing skills and experience;
• Excellent spoken and written English language skills.
• Highly desirable experience includes the following
• Deep contextual knowledge of Lebanon;
• Experience conducting evaluations in Lebanon and/ or the Middle East region;
• Experience working with an HMG framework, and providing project recommendations to cross-Whitehall stakeholders;
• Strong spoken Arabic language skills preferred.

8. Supervision: The Consultant(s) will report to the INTAJ Programme Manager. The Consultant(s) will work closely with:

• INTAJ Team
• Technical Advisors within Mercy Corps’ Global Technical Support Unit
• Monitoring, Evaluation & Learning Advisor
• Performance and Quality Manager
• Director of Programs

Send i) a technical and financial proposal; ii) CV(s) of proposed consultant(s); iii) an example of previous written work / evaluation carried out; and iv) three references with the subject line “INTAJ Final Evaluation Application: your name(s)” to gfox@mercycorps.org by 26 January