AI Agent Evaluation Engineer
💰 $80,000 – $130,000/yrMarket estimate · not provided by the employer
At a glance
- Who can apply
- United States, Canada
- Work arrangement
- Remote · Canada (Remote); United States (Remote)
- Employment type
- Full-time
- Salary
- $80,000 – $130,000/yr · market estimate
- ✓ Open to early-career candidates
Checked against the employer's source on November 22, 2026. Details come from the employer's posting; confirm them on the application page.
Advertisement
Put your brand in front of 300,000+ monthly visitors worldwide
Job Description
Cresta builds an AI platform combining conversational AI agents, real-time agent augmentation, and conversation intelligence for customer experience teams. This role owns defining and measuring the quality of its AI Receptionist agent across production conversations.
- Define and maintain the evaluation rubric covering adherence, outcome, and conversation experience for the agent
- Write annotation guidelines and a failure taxonomy, and run calibration sessions measuring inter-rater agreement
- Classify and prioritize failure modes by severity and frequency so engineering and product focus on what matters
- Review production conversations by hand weekly, directing QA partners, and cluster failures into automatable evaluations
- Trace each failure to its likely cause and route it to the right owner with supporting evidence
- Build and operate an evaluation pipeline scoring every production conversation with automated checks, LLM judges, and human review
- Calibrate every automated judge against human labels and design stratified sampling so reported quality changes are real
- Maintain the golden set and regression suite that prompt, model, knowledge, and flow changes must pass before release
- Maintain the failure-mode coverage map and design evaluations for handoffs, misuse handling, and upstream failure resilience
- Report quality trends by language, accent, audio quality, and intent, stating uncertainty clearly to stakeholders
Summary of Cresta's posting by JobsRemoteList. Read the full details and apply on the employer's site.
Requirements
- At least 3 years of professional experience in software engineering, applied ML, or data-intensive research roles
- At least 1 to 2 years working hands-on with LLM-based systems in production
- Bachelor's or Master's in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or related field, or equivalent practical experience
- Has built and operated an end-to-end evaluation system for a production LLM product, including rubric, human labels, judge, and regression suite
- Strong Python programming skills and experience with LLM APIs, able to ship evaluation tooling independently
- Understands LLM agent mechanics including prompts, tool calls, context handling, and multi-turn state
- Professional fluency in at least one language other than English for evaluating multilingual agent behavior
- Strong written communication skills, since the role authors guidelines others follow
- Fine-tuning and ML framework experience is not required
Benefits
- Comprehensive medical, dental, and vision coverage for employees and families
- Flexible PTO
- Paid parental leave for all new parents
- Retirement savings plan
- Remote work setup budget
- Monthly wellness and communication stipend
- Equity as part of total compensation
- Base pay is location-based and competitive; the posted range is not included in the posting text provided
Advertisement
Put your brand in front of 300,000+ monthly visitors worldwide
Related Jobs
Find more than a single listing
Discover more jobs
Join now and find your dream job or a higher-paying opportunity with better employer visibility, stronger internal discovery and easier access to worldwide hiring signals.
Discover hidden jobs
Surface roles you may miss on noisier boards and move through cleaner employer links and better intent pages.
Head start against the competition
See fresh roles, related pages, salary context and company signals faster so you can apply earlier and better.
10,000+ more jobs than narrower job boards
Explore remote, hybrid and location-based opportunities across worldwide markets instead of relying on one narrow inventory.
Worldwide opportunity coverage
Use region pages, country-eligibility pages and skill pages built for global job seekers, not just one market.
Social proof
Trusted by professionals
“There are good vacancies, but availability varies depending on the region.”
Giulia Romano
Recruiter · 🇮🇹 Italy
“I found an international opportunity that closely matched my experience.”
Tunde Adeyemi
Data Engineer · 🇳🇬 Nigeria
“I found several relevant roles and the vacancy pages were easy to understand.”
Nur Aisyah
MEAL Coordinator · 🇲🇾 Malaysia