Ai Evaluation Engineer

About FirstIgniteFirstIgnite is an AI‐powered business development platform that helps university technology transfer offices turn research breakthroughs into partnerships, licenses, and companies.
We combine LLM‐driven workflows with human relationships to deliver expert discovery, grants search, and AI‐driven outreach.Role OverviewWe are hiring an AI Evaluation Engineer to own the quality bar for every LLM‐powered feature we ship.
You will design, build, and scale the infrastructure that provides evidence on whether changes to prompts, models, or agents improve or degrade performance.
This is a high‐leverage role that interacts closely with engineering, product, and applied AI teams.What You'll DoBuild evaluation infrastructure: design and maintain evaluation suites using Promptfoo, LLM‐as‐judge methods, and custom harnesses for features such as expert search, grants search, and AI SDR agents.Define what good means: translate fuzzy customer outcomes into precise, measu...