Telecommut
About The Role
You will translate product and research goals into labeling instructions, refine guidelines for edge cases, adjudicate ambiguity, and drive measurable quality improvements through audits, calibration, and error analysis. Work includes RLHF preference ranking, prompt evaluation, named entity recognition, computer vision annotation (bounding boxes/segmentation), and content safety labeling.
What You Will Do
- Create, iterate, and maintain annotation guidelines and SOPs for complex datasets
- Perform and audit data labeling with measurable annotation guidelines compliance
- Execute RLHF comparison tasks and curate preference data with consistent rubrics
- Run QA evaluation programs: sampling plans, golden sets, inter-annotator agreement, defect taxonomy tracking
- Connect annotation outcomes to model performance improvement and large language model evaluation outcomes
- Document dataset lineage, versioning, and evaluation criteria
- Senior-level experience in AI data annotation / data labeling and rubric-based review
- Ability to interpret ambiguous examples, adjudicate edge cases, and write clear instructions
- Understanding of how labeled datasets impact RLHF outcomes and downstream model behavior
- Strong communication and cross-functional collaboration (Engineering, Product, Research)
- Hands-on RLHF workflows and prompt evaluation for conversational AI
- Content safety labeling and policy-driven evaluation experience
- Multimodal computer vision annotation exposure (detection/segmentation)
- Experience scaling annotation programs with vendors or distributed teams
Apply via Rex.zone with a resume highlighting AI data annotation scope, QA evaluation experience, and any RLHF or prompt evaluation work. Include examples of guideline writing, calibration, adjudication, and training data quality improvements.
