Terms of Reference for Final Evaluation

1. Project Summary

2. Project Background 

The project is implemented in Sindh (District Matiari, Larkana, Shaheed Benazir and and Khairpur), Punjab (Layyah, Chakwal and  Rahim Yar Khan). The goal of the project is to provide access to improved and community and school inclusive eye health services. This is to be achieved by strengthening ophthalmic services at the primary and secondary health level, by capacity building of local health staff, by strengthening organizations of persons with disabilities (OPD’s) and engaging in systematic advocacy with health authorities.

CBM Peek Project Chakwal & Layyah are implemented in Punjab (District Chakwal & Layyah). The goal of the projects is to strengthen the existing health care systems of by providing access to improved and inclusive eye health services through the inclusion of Peek technology. This is to be achieved by strengthening Ophthalmic Services at the primary and secondary health level, by capacity building of local health staff and engaging in systematic advocacy with health authorities.

Before these projects, access to primary-level eye health services in the district was very much limited with significant barriers for the general population, especially individuals with disabilities. There was limited outreach and availability of essential services to meet the rising demand for vision care. The District Headquarter Hospital (DHQ-Chakwal & Layyah) and Tehsil Headquarter Hospital (THQ-Talagang & Thal) in the district were the only two providers of eye care services in the region. However, the facilities lacked adequate infrastructure, specialized equipment and trained staff, leading to significant delays in diagnosing and treating eye conditions.

These shortcomings contributed to a growing backlog of untreated cases. A critical gap was due to the absence of effective coordination among stakeholders such as Lady Health Workers (LHWs), Lady Health Supervisors (LHSs), Lady health visitors (LHVs), School Health & Nutrition Supervisors (SH&NSs), Teachers and the ophthalmic sector within Chakwal & Layyah. This lack of collaboration hindered efforts to improve service delivery and optimize the allocation of resources.

In addition, the data gap hindered planning efforts and the design of targeted interventions to meet the needs of this population. The government faced challenges in providing high-quality and comprehensive eye care services at both primary and secondary healthcare levels. The limitations in technical expertise, staffing, and capacity-building efforts resulted in compromised eye care for the local population.

The program's objectives are expected to address these systemic challenges and improve the overall delivery of eye care services in the region. 

Specific Objective:People with threatened or existing visual impairment benefit from newly established, accessible, and inclusive structures for prevention, diagnosis, and treatment of visual impairments in Sindh and Punjab.

Project location:The project covers the as follows 

Sindh:

Total Schools in Matiari  963

Total Schools Covered594

Total Students enrolled 123593

Schools covered in Khairpur and Benazirbabd 

BHURHCTHQ/ DHQTertiary

District Matiari25040301

District Larkana04020100

District Khairpur13060001

District Shaheed Benazirabad 11030100

Punjab:

BHURHCTHQ/ DHQTertiary

District Chakwal67130401

District Layyah 42060701

Project time frame:01.01.2018 - 31.12.2024

Target group:

• Sindh Project Workflow

3. Project Summary and Activities

The project aims at reaching the following set of results by the end of the project:

1. Access to high quality inclusive and comprehensive eye health services is provided by trained eye health staff in BHUs, RHC and THQ

2. Person with disabilities and their families are organized and trained to advocate for awareness about disability, their rights and access to inclusive eye health services

3. A committee comprising stakeholders in the ophthalmic field in Sindh and Punjab has been established. Its members collaborate on a strategy with shared objectives to deliver inclusive and high-quality eye health services.

Major activities 

• Screening of the clients through peek capture by using android devices by LHWS in the communities and through LHVS at health facilities 

• Establishment of optometry clinics by deploying optometrist and provision of necessary equipment like slit lamp, auto refractometer, refraction sets 

• Provision of equipment of at secondary level

• Training of screeners in peek capture

• Making health facilities accessible 

• Coordination and laison with social welfare department 

• Developing cooperation between IEH and CBID programme in district Layyah

• Establishment of district eye health committee in district Layyah.

• Strong social mobilization at all districts through IEC material developed within the project.

• Providing low vision services in the districts.

• Implementing CSOM in district Layyah 

• Peek core team certified to use peek capture 

• Provision of state of the art equipment to strengthen eye health institutes through peek project 

4. Evaluation Objective, Scope and Intended Use

The final evaluation will review, analyse and assess the overall project implementation according to the project plan from the project start on 01.01.2018 and ends by 31.12.2024

1. Scope of the Evaluation 

The scope of the evaluation will cover the following:

1)   Review the Implementation of Project Activities percentages.

2)   Review the impact of using Peek Technology into the existing IEH services & it’s referral system / patient work flow

3)   To review the Impact of establishing the Triage levels using Peek Technology.

4)   Review planned and achieved project objective, results, activities, and indicators specified in the approved impact matrix (log frame) and adapted project plan (original proposal and change request).

5)   Impact of Peek powered screening on School Eye Health through School Health & Nutrition Supervisors.

6)   To review planned and achieved project objective, results, activities, and indicators specified in the approved impact matrix (log frame) and adapted project plan.

7)   To review the quality of improved inclusive(eye) health services with regard to infrastructure (project hospitals and medical equipment) and staff capacities.

8)   To look at achievements of the awareness raising / sensitization activities as well as capacity building activities of the project

9)   To review utilisation of funds vs. project activities and its effectiveness.

10)       Determine the project's effectiveness/impact in raising awareness about the rights of persons with disabilities and increasing access to eye health services, particularly for women, children, and individuals with disabilities.

11)       Document effective strategies and approaches for delivering inclusive eye health services and promoting disability inclusion and accessibility.

12)       Identify areas for improvement, lessons learned, successes, and challenges to improve future projects, ensuring high-quality eye health services for all.

13)       To recommend actions for risk management ensuring effectiveness, efficiency, and overall feasibility of projects with a similar objective/agenda.

 Coverage of the evaluation 

The geographical coverage of the evaluation includes people /beneficiaries/ stakeholders reached through project related activities in the Tehsil Head Quarters , Rural Health Centres, Basic Health Units, schools, and communities as mentioned above (Project Location)  in Sindh and Punjab.

Target Audience 

• Adults and children with visual impairments, blindness, and eye conditions.

• People with disabilities in the catchment area of the project.

• Families of the beneficiaries.

• Various types of project-related stakeholders (primary health care staff, school health and nutrition supervisors, OPDs representatives established under the project, BHU static and outreach staff/paramedics)

• District Level Health Authorities/Officials, IEHC members, MS and Consultant Ophthalmologists

• Community members

• Teachers/school personnel, and school children with visual impairments, blindness and low vision.

• Elderly people with vision and health problems.

• Women with and without disabilities. 

2. Intended Use

. The final evaluation shall be designed as to act as a ready, accessible and easily understandable reference that provides a review of project interventions and achievements and offers recommendations for improvement in in future endeavours of the partners and beyond. For implementing organizations, it can be used as an instrument and go-to learning document when it comes to confirm, revise, reassess or make adjustments in order to facilitate a successful project implementation and target as well as impact achievement in projects with a similar objective/agenda and to secure sustainability of project results/outcomes.

For the donors, the evaluation provides a stock-taking of the project’s achievements and impact, and documents adherence to funding guidelines and project plan- not the least for the sake of accountability. In addition, for SIOVS-COAVS-HANDS as the private executing agency of the co-funded CBM project, the evaluation helps in discussing necessary adjustments to future project design and implementation with partners / implementing organizations and informs future project monitoring. 

For the target group, including persons with disabilities keen to avail inclusive health and education services and new opportunities to generate income, the evaluation will elicit crucial feedback and participatory input which may help steering future projects of a similar nature to a closer alignment with the target groups’ demands and expectations.

The evaluation will also be useful to the various stakeholders involved in project activities or addressed through the project interventions as it will assess the programmatic work of the partner/ implementing organization and address any shortcomings with respect to providing access to inclusive health and education services and new opportunities to generate income.

5. Limitations

• The scope of the evaluation is limited by a budget and requires close coordination between the consultant/evaluation team and the project partner to determine duration, margins of geographical coverage as well as sample size of target group to be surveyed. 

• The evaluation might be affected by the unavailability of some target beneficiaries. To address this challenge, the partner/contracting organization will make prior arrangements with all beneficiaries whom the evaluator/evaluation team will sample from to ensure they spare some time to attend this important exercise. Furthermore, the political situation is instable on the district level, resulting in a frequent change of district leadership positions. This might affect access to some stakeholders. Additionally, the following circumstances might affect the evaluation:

• All Government key staff might not be available for interviews.

• Site visits might be challenging due to unforeseeable weather conditions.

• Language barrier between consultant and community people or respondent.

• Flood / heavy raining and cold dry weather can influence the pace of the study.

6. Evaluation Questions

As per project requirements the guiding questions for the evaluation are based on the OECD (Organization for Economic Cooperation and Development) DAC (Development Assessment Criteria) evaluation criteria of relevance, coherence, effectiveness, efficiency, impact, and sustainability. Gender equality and disability inclusion and Child Safeguarding are mandatory areas of enquiry for each evaluation.

The evaluation shall measure progress and assess relevance, effectiveness, efficiency, coherence, impact and sustainability of the project(interventions) in line with project outcome and output deliverables. In doing so, the evaluation shall refer to timelines, scope, budget and quality of implementation against targets set in the project plan. In line with disability inclusive development (DID) standards, the evaluation will also assess the engagement of persons with disabilities within different phases of the project cycle. Other areas the evaluation needs to consider are gender equity, disability inclusion and child and vulnerable adults safeguarding. The evaluation will identify and analyse good practices and success stories for sharing and learning. It will set recommendations that will help improving future projects with a similar objective/agenda

6.1Relevance, Quality of Design and Appropriateness 

1. How well did the project address the identified gaps and needs in eye health services and culturally acceptable in SINDH and PUNJAB?

2. To what extent were the project objectives aligned with the priorities and policies of the local health authorities and stakeholders?

3. How relevant were the interventions to the socio-economic and demographic characteristics of the target population (e.g., rural communities, persons with disabilities)?

4. How comprehensive and coherent was the project design in achieving its stated objectives (e.g., improving access to inclusive eye health services, raising awareness about disability rights)?

5. How well did the project adapt to local contexts and challenges (e.g., infrastructure limitations, socio-cultural beliefs) in delivering inclusive eye health services?

6. To which extent inclusion is addressed in the project intervention area

6.2Effectiveness

1. To what extent were the targets set for improving access to inclusive eye health services met? (e.g., coverage of screenings through peek capture , treatment rates)

2. What evidence exists of improvements in eye health outcomes among the targeted population (reduction in visual impairments, increased access to treatments)?

3. What changes were observed in the awareness and behaviour of stakeholders (e.g., community members, health staff, local authorities) towards inclusive eye health practices and disability rights?

4. To what extent did the project contribute to enhancing the capacity and skills of local health staff in delivering inclusive eye health services.

5. What lessons learned from the project could be applied to similar interventions or scaled up in other regions?

6.3Efficiency or cost-effectiveness (of planning and of implementation)

1. Do you think the available staff was sufficient to work on the project?

2. How efficiently were project resources (financial, human, and technical) utilized in relation to the outcomes achieved.

3. Does the project team have sufficient and appropriate understanding, attitude and skills required to do their job? Is there any record of training/support provided to these staff?

4. Were the resources allocated (financial, human, and technical) adequate and effectively utilized to achieve the project's goals and objectives?

5. Can you please share what procedures were in place to ensure efficient use of the project funds? Have the procedures been fully understood and followed by the project staff in the implementation of the project and reasons for insufficiency if any?

6. Were there any areas where resource allocation could have been optimized to improve effectiveness? 

7. How did the project ensure ‘value for money’ in delivering eye health services?

6.4Impact

1. How did you ensure the project has contributed to strengthen the health system to deliver qualitative eye and ear care?

2. Did the project contribute to decrease blindness in the district; what are the proxy evidence?

3. How did the project ensure that persons with disabilities received the same services and benefits as their peers without disabilities (e.g. provision of reasonable accommodation)? 

4. What do you think is the most significant impact of this project? How do you measure that?

6.5Sustainability

1. What were the major factors that influenced the achievement or non-achievement of sustainability of the project? How could this have been improved? Had there been a sustainability plan in place?

2. What steps local district authorities have taken with reference to sustainability of the project

3. What are the specific results of the project that will remain and continue even after the project closure? 

4. What could be done to ensure that community people receive sustainable eye health services even after the project is closed? Please explain. 

5. Will you be able to continue or even replicate the project somewhere else in their operational areas, after the cessation of funding from CBM?

6. What measures were put in place to ensure the sustainability of project outcomes beyond the project's duration? (e.g., institutional capacity building, policy advocacy, community ownership) 

6.6Gender

1. Were women, men, girls, and boys with and without a disability equally able to access the full range of services offered? What difficulties did each of those groups have in accessing the services, for example relating to physical accessibility and costs of services? What are commendations for addressing the barriers in the future?

2. Did partner ensure gender equality in project staffing?

6.7Child Safeguarding

1. Does partner have their own Child Safeguarding policy?

2. Do partner hospitals have their own Child Safeguarding policy?

3. Do you know whether staff of partner hospitals understand child safeguarding issues in general?

4. How has a safe environment for children been established and maintained at the projects?

5. How did the project ensure safeguarding in outreach camps?

6. What is the mechanism of complain/feedback and reporting?

7. How did the project ensure safeguarding while working with children at school and in communities?

6.8Disability Inclusion

1. Has the awareness and understanding of disability increased amongst program staff? 

2. Have organisational policies, procedures and project decisions been reviewed and updated to be disability inclusive?

3. How did the project ensure that persons with disabilities are equally benefited from project activities?

4. How did the project ensure the participation of person with disabilities, organizations of person with disabilities and self-help groups during project implementation and how did they benefit from project activities?

7. Evaluation Criteria 

Proposed method/methodology for evaluation.

Timetable/work plan: realistic to meet evaluation need.

Financial proposal: realistic and feasible

Knowledge and experience of evaluation team

7.1 Evaluation Team and Management Responsibilities 

Final evaluation assignment will be commissioned under the responsibility of the CBM Pakistan Country Office. The procurement process will include CBM International and partners. Partners will support/facilitate consultant during field work and other related to activities, 

8. Evaluation Team

The Lead Evaluator is expected to have the following qualifications

• Master’s degree or higher academic degree related to Social Sciences, International Development, public health, economic and social development related studies, Organisational Development or similar (at least the Team Manager should be an ophthalmologist and eye health expert)

• Proven consultancy experience of at least 10-15 years with a record of providing high quality, creative and analytical report and including at least with one international community development organization.

• Proven experience in the evaluation of projects/programmes by specific public donors 

• Experience in evaluation of similar programmes and sound knowledge of Project Cycle Management and evaluation criteria

• Experience in participatory approach is a must as well as facilitation skills.

• Good analytical skills, Proven working experience in the field of disability and inclusive at national and international organizations.

• Knowledge about local culture, policies, and laws to understand the situation for persons with disabilities in Sindh and Punjab districts plus international experiences.

• Excellent writing and presenting skills in English and local languages.

• Ability to work on time bound situation and deliver expected outcomes on agreed time frame.

• Any similar activity already conducted in Sindh and Punjab Districts as mentioned or any other province.

• The evaluation team is expected to be medically ensured during the time of service under this consultancy. The contracting agencies will not take responsibility for any damage, injury, or loss in the framework of this evaluation.

• As a condition of entering into a consultancy agreement the evaluators must sign the organizational Child Safeguarding Policy and abide by the terms and conditions thereof. 

9. Management of the evaluation and logistics 

The consultant / firm will be responsible for all logistics and boarding’s , travels etc 

Approve all reports and make payment to evaluation firm. 

Project Partner has responsibility for:

Ensuring that project report, record and other documents are available for evaluation team.

Providing access and managing time to discuss/interview with beneficiaries, project staff and other project stakeholders as per evaluation TORS

Organising a brief meeting with relevant stakeholders to share the evaluation findings.

10. Outputs and Deliverables 

This evaluation expects following output:

Working language for this evaluation process will be Sindhi/ Urdu/Punjabi/ Saraiki at field but reporting language is English.

Comprehensive evaluation report according to the ToR/report template.

Evaluation report should be in English, report generated from field like case story/beneficiary interview can be attached as annex in Sindhi/ Urdu/Punjabi/Saraiki.

Evaluation team will get feedback on draft report from NC office  and is responsible for finalization within due date.

Evaluation team will organize evaluation finding debriefing meetings with concerned stakeholders.

First draft report is expected to be provided by 1st Week of February while Final report by 3rd week of February 2025. 

11. Duration and Phasing

12. Costs and Payments

Cost of project final evaluation have been allocated by project and planned based on budget availability. Consultant payments, logistics and other expenditures for the project final evaluation will be NC office.

Proposed payment schedule is as below: 

• 30% payment at time of signing contract

• 40% payment on submission of draft report.

• 30% payment on approval of final report.

13. Application Process

Interested and qualified consultants are expected to submit a technical and financial proposal including: 

• A description/profile of the consultant(s)/consultancy firm, 

• CV of suggested team lead and members, 

• an outline of the understanding of these ToRs and suggested methodology, and a detailed work plan for the entire assignment. 

• A detailed budget for the expected assignment shall include all costs expected to conduct a disability inclusive and participatory study, and taxes according to the rules and regulations of the consultants’ local tax authorities.