Consultancy for the Independent Evaluation of the FarajaMH Generative AI Screening Model Before and After Fine-Tuning

  • Deadline: December 10, 2026
  • Kenya

Introduction & Summary

1. Background

The African Population and Health Research Center (APHRC) is a premier research-to-policy institution, generating evidence, strengthening research and related capacity in the African research and development ecosystem, and engaging policy to inform action on health and development. The Center is Africa-based and African-led, with its headquarters in Nairobi, Kenya, and a West Africa Regional Office (WARO), in Dakar, Senegal. APHRC seeks to drive change by developing strong African research leadership and promoting evidence-informed decision-making (EIDM) across sub-Saharan Africa.

The Faraja Mental Health (FarajaMH) project, a Wellcome Trust-funded consortium led by APHRC, is building a bilingual (English and Swahili) generative AI model to help screen for depression, anxiety and psychosis in community and primary care settings. In Work Package three, an open-weight, audio-capable base model will be chosen and fine-tuned using federated LoRA (bf16), combining federated supervised fine-tuning with centralised Direct Preference Optimisation (DPO)

Before the model is frozen (Track A) and passed to Work Package four for validation in a randomised controlled trial, it needs an independent, reproducible evaluation, both as released by its developer and after each fine-tuning stage. This evaluation will determine which base model is used, show whether fine-tuning improves screening accuracy, safety, language understanding and cultural fit without losing prior capabilities, and give the tri-committee governance structure the evidence it needs to approve the freeze. Because Kenya’s Data Protection Act (2019), Tanzania’s Personal Data Protection Act (2022) and the project’s data sovereignty principles restrict cloud storage of personal clinical records, all evaluation platforms must run on APHRC premises. The consultancy supplies specialist evaluation capacity to work alongside the existing WP3 team.

2.Purpose and Objectives

To design, deploy on APHRC premises and run an independent, reproducible evaluation of the FarajaMH GenAI screening model before fine-tuning (base-model baseline and selection) and after fine-tuning (SFT, DPO and safety-layered versions), producing the evidence required for the Track A model freeze and handover to WP4 by 31 August 2027.

The objectives are to:

  • Establish a locked, hash-pinned evaluation protocol and onboard on an existing on-premises evaluation platform at APHRC.
  • Produce a pre-fine-tuning baseline report for all candidate open-weight models.
  • Measure the effect of fine-tuning on screening accuracy, clinical reasoning, Swahili and code-mixed language understanding, idioms of distress, audio understanding, empathy, cultural fit, safety and fairness, against the pre-fine-tuning baseline.
  • Involve clinicians, people with lived experience and caregivers as blinded evaluators of model(s) behaviour.
  • Assess readiness against the Track A freeze criteria and hand over a reusable evaluation harness to the APHRC team.

Duties / Responsibilities

3.Scope of work

3.1 Inception and Evaluation Protocol

  • Review the work package three (WP3) Technical Methodology, the work package (WP3
  • Model Evaluation Strategy, the FarajaMH study protocols and the work package four (WP4) design and agree the evaluation plan with the principal investigator (PI)and Work package three (WP3) team.
  • Finalise and lock the evaluation protocol before any model is run. The protocol is hash-pinned (SHA-256) and not changed once results are seen.
  • Map each evaluation measure to the governance decision it informs base-model selection, the per-round PROMOTE, RETRAIN or ROLLBACK gate, and the Track A freeze trigger.

3.2. On-Premises Evaluation Platform

  • Onboard on the existing evaluation platform or install, configure and document their evaluation platform on APHRC servers.

3.3. Evaluation Data Banks and Test Security

  • Work with clinicians, the WP2 team, linguists and Lived Experience Advisory Committee (LEAC)members to complete, quality-check and freeze the Testing Set v1.0: core cases (B1), language (B2), idioms of distress (B3), safety (B4), audio (B5), the FarajaMH MCQ set and the public benchmarks.
  • Validate the Golden Set (500 to 1,000 clinician-adjudicated examples) for coverage of all conditions, languages, sites and WP4 demographic strata, and confirm that it never enters training.
  • Calibrate the judge models against clinician ratings for each language variety (accepted at Krippendorf’s alpha ≥ 0.67; otherwise, all replies in that variety go to human raters).

3.4. Pre-Finetuning (Baseline) Evaluation

  • Evaluate every candidate model, as released by its developer, under identical conditions: hard gates (native audio, Swahili smoke test, LoRA dry run on one node, no critical safety failure) and the nine test areas (Swahili depth, idioms of distress, screening accuracy, clinical reasoning, empathy and cultural fit, safety, audio understanding, knowledge, practicality).
  • Score and rank the candidates with 95% bootstrap confidence intervals, DeLong tests for paired AUROC differences and a rank-stability analysis; shortlist the finalists.
  • Re-test the finalists after the APHRC team's short LoRA fine-tuning probe and prepare the base-model decision pack for tri-committee sign-off.

3.5. Post-Fine-Tuning Evaluation

  • Implement and validate the automated per-round Golden Set gate (per-condition AUROC, sensitivity, specificity, subgroup AUROC and safety refusal rate), with every PROMOTE, RETRAIN or ROLLBACK decision written to the audit trail.
  • At the end of each data wave within the contract period, re-run the full locked evaluation on the merged SFT adapter and on the DPO-aligned adapter, and report the change from the pre-fine-tuning baseline for every area.
  • Report classification performance (accuracy, precision, recall, F1-score, AUROC, calibration, sensitivity at fixed specificity) by condition, and generative quality (BLEU, ROUGE, BERTScore against clinician reference responses), by site and by country, including the held-out external test set.
  • Check for catastrophic forgetting across waves, registers (CLINICAL_NOTE), (CONVERSATION), (LONGITUDINAL) and modalities (text, audio, paired).
  • Evaluate the safety layer: single-turn and multi-turn red-teaming (including VERA-MH style crisis conversations), HarmBench and XSTest in English and Swahili, refusal scripts, content filters, the suicide-risk escalation pathway and over-refusal.
  • Audit fairness across language variety, country, site, sex and age band, and flag any subgroup AUROCgap above 0.05.

3.6. Human-Centred Evaluation

  • Design and run blinded, randomised rating sessions, before and after fine-tuning, with psychiatrists and clinical psychologists (reasoning and safety), people with lived experience, caregivers and Community Health Promoters (empathy, cultural fit, stigma and acceptability), and native speakers from each country (language and register).
  • Use two raters per reply with a third to resolve disagreements and report inter-rater agreement.
  • Run sessions in a trauma-informed way, in the participants' preferred language, with safeguarding and support arrangements agreed with the LEAC and the PI.
  • Report findings back to the team: idioms the model misses go to WP2, and failure patterns inform DPO preference data.

3.7. Track A Freeze Readiness and Handover

  • Assess the fine-tuned model against the four freeze criteria: Golden Set AUROC ≥ 0.78 for depression, anxiety and psychosis; no subgroup AUROC gap above 0.05; adversarial safety refusal rate ≥ 95%; and documented committee sign-off.
  • Prepare the evidence dossier for the Ethics and AI Evaluation Committee, the Data Governance and Protection Committee and the LEAC chairs and record the hash-pinned audit entry for the frozen artefact.
  • Hand over the evaluation harness, documentation and model card to APHRC, and train the APHRC team to run it for Track B and post-WP4 evaluation.

4. Collaboration and Rationale

The consultant will work closely with the following people and groups, under the supervision of the PI: Partner Role in the evaluation
PI, FarajaMH (APHRC) Supervision, approval of the protocol and acceptance of deliverables
APHRC AI engineers and data scientists Model training, fine-tuning probe, federated rounds, platform access, data pipelines
CODATA Knowledge Graphs, Benchmarks, red-teaming and statistical review
Clinicians in Kenya and Tanzania (psychiatrists, clinical psychologists) Case writing, answer keys, Golden Set adjudication, rating of reasoning and safety
People with lived experience, caregivers and Community Health Promoters (LEACs, BNBR Rating of empathy, cultural fit and acceptability; review of test items
Mental Health Planet (MHP) Lead on human rating and safety evaluation
CUHAS site team (Mwanza) Deployment and runs on the Tanzania node
Tri-committee and Steering Committee Sign-off of the base-model decision and the Track A freeze

5. Oversight

The consultant will report directly to the Principal Investigator (FarajaMH Project) and submit bi-weekly progress updates as agreed in the work plan. Coordination meetings with the WP3 team will be held bi-weekly to review progress, address challenges and ensure timely delivery.

6.Deliverables and Timelines

No Scope Area Deliverable Due Date Fee (USD), proposed by consultan t
D1 I Inception report and locked evaluation protocol: detailed work plan, final evaluation protocol (areas, weights, gates, metrics, thresholds, decision margin) hash-pinned before any locked test run, and data protection plan. 29 January 2027
D2 II On-premises evaluation platform commissioning report:pinned container, judge models and rating tool installed at APHRC, reproducibility test and speech smoke test . 26 February 2027
D3 III Frozen Testing Set v1.0 and Golden Set validation report:quality-checked, split and hash-pinned data banks, Golden Set coverage report, and judge calibration report per language variety. 19 March 2027
D4 IV Pre-fine-tuning baseline evaluation report : gate pass or fail table, nine-area scorecard with 95% confidence intervals, rank-stability analysis, language and idiom results, safety panel, and blinded human-rating results with agreement statistics. 16 April 2027
D5 V Base-model decision pack:fine-tuning probe re-test of the finalists, final scores, recommendation, risks and audit entry, presented for tri-committee sign-off. 30 April 2027
D6 VI Automated per-round Golden Set gate: evaluation workflow live on the APHRC platform, validation test log and gate decision log (PROMOTE, RETRAIN, ROLLBACK) ready for the first federated round . 21 May 2027
D7 VII Wave 1 post-fine-tuning evaluation report: SFT and DPO adapters compared with the pre-fine-tuning baseline on every area; classification and generative metrics by condition, site and country; catastrophic forgetting checks . 30 June 2027
D8 VIII Safety, fairness and human-centred evaluation report: red-teaming and safety-layer results, escalation pathway tests, over-refusal, subgroup fairness audit, and post-fine-tuning ratings by clinicians, people with lived experience and caregivers . 30 July 2027
D9 IX Track A freeze readiness dossier: assessment against the four freeze criteria, evidence pack for the committees and hash-pinned audit entry, or, if criteria are not met, a gap analysis with a remediation plan. 13 August 2027
D10 X Final evaluation report and handover: consolidated before-and-after report, model card, reproducibility package (code, configurations, hashes, logs), harness documentation and hands-on training of the APHRC team 31 August 2027
Total

End-of-wave evaluations for any further data wave completed before 31 August 2027 are included in D9 and D10. The consultant will also provide bi-weekly progress updates, which are not separately priced.

7.Timelines

  • Pre-Fine-Tuning Phase (baseline and base-model selection): January to April 2027 (D1 to D5).
  • Post-Fine-Tuning Phase (federated fine-tuning, DPO and safety layer): May to August 2027 (D6 to D10).
  • If the Scientific Advisory Board extends WP3 because the freeze trigger has not been met, the contract may be extended by up to two months by written agreement.

8.Budget and Payment Schedule

APHRC invites the consultant to propose the fee for each deliverable. The financial proposal must:

  • Complete the fee column of the table in Section six (6), in USD, for each of D1 to D10, with the total at the end.
  • Show, for each deliverable, the number of days of effort, the daily rate and the team members involved.
  • APHRC will provide the GPU servers, storage, network access, evaluation platform, meeting venues, clinicians, LEAC members, caregivers and other raters in line with APHRC guidelines.
  • These costs should not be included in the consultant's fees. No cloud computing, external API or software subscription costs will be accepted.
  • Payment is made per deliverable, within 30 days of the PI's written acceptance of that deliverable and receipt of an invoice. A deliverable that does not meet the agreed specification will be returned with comments and paid once accepted

9.Intellectual property, ethics, confidentiality, data protection and Data Residency requirements

All data, test items, model artefacts, code and project-related materials remain the property of APHRC. The consultant shall not disclose, publish or share any portion of the data, test items or findings without prior written consent from APHRC. Intellectual property generated from this consultancy, including the evaluation harness, will be vested in APHRC. Authorship of any resulting publication follows APHRC's authorship policy.

All evaluation platforms, models, data and results must remain on APHRC-controlled infrastructure. The consultant must ensure that: –

  • All evaluation software, models, judge models and rating tools are installed and run on APHRC servers in Nairobi (and, for Tanzanian data, on the CUHAS node in Mwanza).
  • No public cloud service, rented GPU, external API, external judge model or personal device is used to store or process project data, test items or model outputs, including synthetic and public benchmark runs, so that all results come from the same pinned environment.
  • Access is only through APHRC-managed accounts, on site or through APHRC-approved secure remote access, with no data copied off APHRC systems.
  • Data handling complies with Kenya's Data Protection Act (2019), Tanzania's Personal Data Protection Act (2022), APHRC's Data Protection Policy, Ethical Data Management Framework and Safeguarding Policy, and the conditions of the study's ethics approvals.
  • Test items are never posted online or pasted into external tools, and canary strings are kept in every test file.
  • All consultant team members sign APHRC's Data Confidentiality Agreement and complete accredited online human subjects research ethics training before accessing data.
  • Rating sessions with people with lived experience and caregivers follow the approved protocol, informed consent in the participant's preferred language, and the LEAC's safeguarding and emotional safety arrangements.
  • Any potential critical safety failure found during evaluation is confirmed by a clinician and reported to the PI and the Ethics and AI Evaluation Committee within 48 hours.
  • A summary of data protection and ethical compliance is submitted with D10.

Qualifications, Skills and Experience

10.Required Qualifications and Experience

The consultancy is open to an individual or a small team. The lead consultant should have:

  • A PhD, or a master's degree with substantial equivalent experience, in machine learning, computer science, biostatistics, health data science or a closely related field. •
  • At least five years' experience in machine learning, with a demonstrated record of evaluating large language or speech models (for example with lm-evaluation-harness, vLLM or Hugging Face tooling).
  • Experience evaluating AI for health, preferably mental health screening or clinical decision support, including safety and red-teaming.
  • Strong statistical skills: AUROC and calibration, bootstrap confidence intervals, inter-rater agreement and subgroup fairness analysis.
  • Experience deploying containerized GPU workloads on on-premises or institutional servers, with no reliance on cloud services.
  • Experience designing human evaluation with clinicians and community members, ideally including people with lived experience.
  • Knowledge of data protection law and research ethics in Kenya or Tanzania.
  • Working knowledge of Swahili; experience with speech or audio model evaluation; experience with federated learning; and peer-reviewed publications in AI evaluation are an added advantage.

Additional Content

Footer Details

Application Requirements

Interested consultants should submit:

  • A technical proposal (maximum 10 pages): understanding of the assignment, evaluation approach, on-premises deployment plan, work plan against the dates in Section 6, and team composition.
  • A financial proposal priced per deliverable (Section 6).
  • CVs of the lead consultant and team members.
  • At least one example of a previous model evaluation report or publication, and two professional references.

Preference will be given to a consultant with experience evaluating clinical or mental health AI in low-resource and multilingual settings, and of involving people with lived experience in research. This consultancy contributes to the project's wider goal of building ethical AI evaluation capacity in East Africa, and the harness it leaves behind will be reused for Track B and post-WP4 evaluation.

Applications should be submitted via email to consultancies@aphrc.org, copying procurement@aphrc.org , with the email subject " FarajaMH Generative AI — Consultancy". The deadline for application submission is December 10th, 2026.

Special Notice

APHRC is an equal opportunity employer that is committed to creating a diverse and inclusive workplace. All employment decisions are made based on qualifications and organizational needs. Reasonable accommodation may be provided to applicants with disabilities upon request, to support their participation in the recruitment process.

As part of our safeguarding and protection framework, successful candidate(s) will be subject to robust reference checks, including verification of past conduct related to safeguarding and workplace behavior.


At Impactpool we do our best to provide you the most accurate info, but closing dates may be wrong on our site. Please check on the recruiting organization's page for the exact info. Candidates are responsible for complying with deadlines and are encouraged to submit applications well ahead.
Before applying, please make sure that you have read the requirements for the position and that you qualify. Applications from non-qualifying applicants will most likely be discarded by the recruiting manager.