AI Engineer – AI Observability and Quality

About the Role & Team

IDC is building the next generation of AI-powered intelligence platforms that transform how technology decisions get made. Our platform re-imagines the way decision-makers discover and interact with trusted research and data, and is foundational to IDC's future.

What You’ll Do

• Design and build the evaluation infrastructure that ensures the platform's AI systems produce accurate, well-sourced, high-quality responses
• Build automated test suites that validate answer quality across agent pipeline changes
• Develop regression detection systems that catch quality degradation before it reaches users
• Create evaluation frameworks that measure response accuracy, citation correctness, and source quality
• Work closely with the product team to translate quality criteria into measurable, automatable test scenarios
• Build cost and latency monitoring that tracks the operational efficiency of AI ...