JobsRemoteList
Cresta logo

AI Agent Evaluation Engineer

Cresta·November 12, 2026·
🌍 Remote · Canada (Remote); United States (Remote)Open to: United States, CanadaFull-timeArtificial Intelligence

💰 $80,000 – $130,000/yrMarket estimate · not provided by the employer

At a glance

Who can apply
United States, Canada
Work arrangement
Remote · Canada (Remote); United States (Remote)
Employment type
Full-time
Salary
$80,000 – $130,000/yr · market estimate
  • ✓ Open to early-career candidates

Checked against the employer's source on November 22, 2026. Details come from the employer's posting; confirm them on the application page.

Advertisement

Put your brand in front of 300,000+ monthly visitors worldwide

Advertise here →

Job Description

Cresta builds an AI platform combining conversational AI agents, real-time agent augmentation, and conversation intelligence for customer experience teams. This role owns defining and measuring the quality of its AI Receptionist agent across production conversations.

  • Define and maintain the evaluation rubric covering adherence, outcome, and conversation experience for the agent
  • Write annotation guidelines and a failure taxonomy, and run calibration sessions measuring inter-rater agreement
  • Classify and prioritize failure modes by severity and frequency so engineering and product focus on what matters
  • Review production conversations by hand weekly, directing QA partners, and cluster failures into automatable evaluations
  • Trace each failure to its likely cause and route it to the right owner with supporting evidence
  • Build and operate an evaluation pipeline scoring every production conversation with automated checks, LLM judges, and human review
  • Calibrate every automated judge against human labels and design stratified sampling so reported quality changes are real
  • Maintain the golden set and regression suite that prompt, model, knowledge, and flow changes must pass before release
  • Maintain the failure-mode coverage map and design evaluations for handoffs, misuse handling, and upstream failure resilience
  • Report quality trends by language, accent, audio quality, and intent, stating uncertainty clearly to stakeholders

Summary of Cresta's posting by JobsRemoteList. Read the full details and apply on the employer's site.

Requirements

  • At least 3 years of professional experience in software engineering, applied ML, or data-intensive research roles
  • At least 1 to 2 years working hands-on with LLM-based systems in production
  • Bachelor's or Master's in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or related field, or equivalent practical experience
  • Has built and operated an end-to-end evaluation system for a production LLM product, including rubric, human labels, judge, and regression suite
  • Strong Python programming skills and experience with LLM APIs, able to ship evaluation tooling independently
  • Understands LLM agent mechanics including prompts, tool calls, context handling, and multi-turn state
  • Professional fluency in at least one language other than English for evaluating multilingual agent behavior
  • Strong written communication skills, since the role authors guidelines others follow
  • Fine-tuning and ML framework experience is not required

Benefits

  • Comprehensive medical, dental, and vision coverage for employees and families
  • Flexible PTO
  • Paid parental leave for all new parents
  • Retirement savings plan
  • Remote work setup budget
  • Monthly wellness and communication stipend
  • Equity as part of total compensation
  • Base pay is location-based and competitive; the posted range is not included in the posting text provided

Advertisement

Put your brand in front of 300,000+ monthly visitors worldwide

Advertise here →

Find more than a single listing

Discover more jobs

Join now and find your dream job or a higher-paying opportunity with better employer visibility, stronger internal discovery and easier access to worldwide hiring signals.

🔎

Discover hidden jobs

Surface roles you may miss on noisier boards and move through cleaner employer links and better intent pages.

🚀

Head start against the competition

See fresh roles, related pages, salary context and company signals faster so you can apply earlier and better.

📈

10,000+ more jobs than narrower job boards

Explore remote, hybrid and location-based opportunities across worldwide markets instead of relying on one narrow inventory.

🌍

Worldwide opportunity coverage

Use region pages, country-eligibility pages and skill pages built for global job seekers, not just one market.

Social proof

Trusted by professionals

Read more reviews →
★★★★★

“There are good vacancies, but availability varies depending on the region.”

Giulia Romano

Recruiter · 🇮🇹 Italy

★★★★★

“I found an international opportunity that closely matched my experience.”

Tunde Adeyemi

Data Engineer · 🇳🇬 Nigeria

★★★★★

“I found several relevant roles and the vacancy pages were easy to understand.”

Nur Aisyah

MEAL Coordinator · 🇲🇾 Malaysia