This job is no longer available
This job expired on 11/10/2026. It no longer accepts applications.
LLM Evaluator – Model Response Analyst (Remote)
Odixcity Consulting
Job description
About the role
We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to assess and improve the performance of large language models. Working remotely, you will analyze AI‑generated text for factual accuracy, coherence, safety, bias and alignment with our guidelines.
Key responsibilities
- Evaluate and rank model‑generated responses using complex rubrics covering factuality, coherence, safety, instruction‑following and creativity.
- Compare multiple outputs for the same prompt, select the preferred one and provide clear justifications.
- Provide concise feedback to modeling and training teams on recurring failure patterns.
- Design adversarial prompts to expose biased, harmful or insecure model behavior and help patch safety gaps.
- Collaborate with QA to refine evaluation guidelines and participate in cross‑checking sessions to ensure inter‑rater reliability.
- Investigate under‑performing outputs, hypothesize root causes and flag novel model behaviors for research.
- Assist in designing experiments and A/B tests to verify improvements in new model versions.
Required profile
- Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP QA or related fields.
- Bachelor’s degree in Computer Science or a related discipline.
- Proven ability to craft prompts that test model limits and to explain why a response is good or bad.
- Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
- Familiarity with dataset sourcing, cleaning, annotation and understanding of data distribution impacts.
Required skills
- Prompt engineering
- RLHF data collection
- Dataset annotation and cleaning
- A/B testing for AI models
- Experiment design and evaluation
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Tunisia.
Salaries by job title
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Odixcity Consulting
Related job offers
-
Software Engineer Intern (Pre-Employment)
Astrolab Sousse -
SAP Country Lead - Tunisia
DXC Technology Gouvernorat Tunis -
Quality Assurance Consultant - Customer Care & CRM
Akkodis Gouvernorat Tunis -
SAP Business One Support Senior Consultant (French-speaking)
Be One Solutions -
Corporate SAP S/4HANA Specialist (PP/QM/EPPDS)
COFICAB Group Gouvernorat Tunis