AI Engineer – AI Observability and Quality
Location
toronto, on
Job Type
Full-time
Category
Other-General
Posted
June 01, 2026
About the Role & Team
IDC is building the next generation of AI-powered intelligence platforms that transform how technology decisions get made. Our platform re-imagines the way decision-makers discover and interact with trusted research and data, and is foundational to IDC's future.
What You’ll Do
- Design and build the evaluation infrastructure that ensures the platform's AI systems produce accurate, well-sourced, high-quality responses
- Build automated test suites that validate answer quality across agent pipeline changes
- Develop regression detection systems that catch quality degradation before it reaches users
- Create evaluation frameworks that measure response accuracy, citation correctness, and source quality
- Work closely with the product team to translate quality criteria into measurable, automatable test scenarios
- Build cost and latency monitoring that tracks the operational efficiency of AI ...