240K–290K USD / year
Member of Technical Staff, ML Platform (Evals)
Machine learningRemote — unknown
Published on 2026-11-12
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Own and build an end-to-end evaluation platform for Runway’s machine-learning models. The platform will generate and score model outputs, support human review, track results across checkpoints and releases, and help research teams make reliable model decisions.
About the company
Runway is an innovative platform that specializes in machine learning and artificial intelligence applications, providing tools for creative professionals and enterprises to harness the power of AI in their workflows. The company focuses on developing advanced research and deployment solutions in this rapidly evolving field.
The team
ML Platform
Stack
PythonPyTorchKubernetes
What you'll do
- Build systems for generating, annotating, reviewing, and adapting evaluations at scale.
- Develop the platform’s CLIs, APIs, GUIs, and storage layers.
- Partner with research teams across video, image, audio, agents, and robotics to build and generalize evaluation tools.
- Set standards for reproducibility, metrics, reporting, and trustworthy results.
- Drive adoption and integrate the platform into research, training, and model serving.
What we're looking for
- 5+ years building production ML infrastructure or data platforms, including work on evaluations, experimentation, or benchmarking.
- Strong Python and PyTorch skills, with experience running large batch GPU workloads on Kubernetes.
- Experience with media or model-output data pipelines, versioning, and reproducibility.
- Familiarity with experimental statistics, including paired comparisons, confidence intervals, multiple comparisons, and inter-rater agreement.
- Ability to build internal tools from command line to browser and lead a broad technical area.
- Understanding of the model development lifecycle and production reliability.
Nice to have
- Experience building evaluation suites for generative models.
- Hands-on experience with LLM- or VLM-as-judge pipelines.
- Experience with online experimentation and connecting offline metrics to product outcomes.
- Experience evaluating agents or robotics policies.
