170K–170K USD / year
Lead Engineer, AI Platform
AIRemote — worldwide
Published on 2026-10-30
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Lead the engineering team building infrastructure to evaluate, monitor, and improve Circle’s production AI agents. This hands-on player-coach role focuses on agent quality, reliability, latency, and cost rather than model research or training.
About the company
Circle is a technology company that provides a platform for building community-driven spaces, focusing on enabling creators and teams to engage and collaborate effectively. They offer tools for marketplace solutions, enhancing user experience through design and advanced technology.
The team
AI Quality engineering team, partnering closely with AI Core engineering.
Stack
Ruby on RailsPythonBraintrustLangSmith
What you'll do
- Build evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for AI agents.
- Diagnose quality failures across agent planning, tool selection, and execution.
- Create annotation workflows and AI-generated or simulated evaluation conversations.
- Run experiments on prompts, models, and agent harnesses to improve quality, cost, and latency.
- Lead the AI Quality engineering team while remaining hands-on in the code.
- Partner with AI Core engineering to assess the impact of product changes.
What we're looking for
- 7+ years building and shipping production software.
- Experience with production LLM agents that use tools, planning, orchestration, or sub-agents.
- Experience building evaluation or observability infrastructure for AI/ML systems.
- Experience designing datasets, annotation workflows, or labeling pipelines for AI evaluation.
- Comfort with Ruby on Rails and/or Python, and ability to learn new technologies quickly.
- Proficient English at CEFR C2 / ILR Level 5.
Nice to have
- Python proficiency, especially for data and evaluation work.
- Experience with evaluation frameworks such as Braintrust or LangSmith.
Benefits
- Company equity with ongoing refresh grants.
- 35 days of paid time off per year.
- Two company retreats per year.