204K–352K USD / year
Research Engineer, Benchmarks & Evaluation
AIHybrid — New York, NY, San Francisco, CA
Published on 2026-11-03
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Own benchmarking and evaluation for Ramp Labs’ AI models and agents working on finance tasks. Build benchmarks, graders, and RL environments from real workflows, and use results to guide model and product decisions.
About the company
Ramp is a financial automation platform that provides companies with tools to manage expenses, optimize procurement, and improve financial workflows. It serves businesses looking to streamline their financial processes and enhance visibility into spending.
The team
Ramp Labs, Ramp’s AI research group.
Stack
PythonSQL
What you'll do
- Build versioned benchmarks from real finance workflows, agent traces, edge cases, and expert labels.
- Run benchmark suites across models and agent versions, and report results for decision-making.
- Design and calibrate rubric-based, programmatic, and LLM-as-judge graders.
- Turn evaluations into reward functions and RL environments; detect reward hacking and data contamination.
- Publish external benchmarks and write-ups on model performance in finance work.
What we're looking for
- Hands-on experience building LLM benchmarks or evaluations used for real decisions.
- Strong Python and software engineering skills; write reproducible, fast evaluation code.
- Statistical judgment, including confidence intervals, sample size, inter-rater agreement, and distinguishing signal from noise.
- Working knowledge of post-training, including SFT and RL with verifiable rewards.
- Comfort working with SQL and large production datasets.
- Good judgment about what is worth measuring.
Nice to have
- Industry experience on an evaluation, benchmarking, or post-training team.
- Experience building RL environments or verifiers for agentic tasks.
- Published benchmark or evaluation research.
- Contributions to evaluation frameworks such as verifiers, lm-evaluation-harness, or HELM.
- Familiarity with accounting, finance, or other expert-domain work.
- Public benchmark, evaluation suite, or failure-analysis work.
Benefits
- Equity.
- Flexible PTO.
- Medical, dental, and vision coverage for employees, with partial dependent coverage.
- 401(k) with employer match.
- Parental leave of up to 16 weeks at 100% pay for birthing and bonding, or 8 weeks for bonding only.
- Relocation expense coverage to NYC or San Francisco, if needed.
- Home-office equipment, health and wellness stipend, and in-office meals and refreshments.