Senior Machine Learning Engineer, Agentic Evals and Self-Improvement (Nova)

About this role

Lead evaluation and automatic optimization for Iterable Nova’s models and AI agents. Build systems that identify failures and turn evaluation insights into measured improvements to prompts, context, tools, and agent harnesses.

About the company

Iterable is a cross-channel marketing platform for email, push, SMS, and in-app messaging. It helps growth and lifecycle teams orchestrate personalized customer journeys.

The team

Nova, Iterable’s AI intelligence layer.

Stack

What you'll do

• Build model and agent evaluation systems, including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
• Create a workflow to diagnose failures, implement fixes, and measure improvements.
• Automate optimization of prompts, context, tools, rubrics, and agent harnesses.
• Build synthetic data pipelines for evaluation and post-training.
• Apply research in LLM-as-judge, reward modeling, test-time search, and post-training to production.
• Mentor engineers and help set Nova’s technical direction.

What we're looking for

• 5+ years of hands-on machine learning engineering experience shipping complex ML systems at scale.
• Master’s or PhD in computer science, machine learning, AI, or a related field.
• Strong Python skills and experience with modern ML frameworks.
• Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning.
• Experience building automated evaluation systems such as LLM judges, reward models, or benchmark infrastructure.
• Experience with prompt or context optimization, post-training, preference optimization, or reinforcement learning.
• Strong communication and cross-team collaboration.

Nice to have

• Experience with LLM APIs from multiple providers, including tool calling and prompt caching.
• Experience with automatic prompt optimization and error analysis.

Benefits

• Equity
• Medical, dental, and vision coverage
• Financial benefits