240K–270K USD / year
Engineering Manager - Inference Performance
Engineering managerHybrid — San Francisco, New York
Published on 2026-10-07
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Lead a team building performance improvements for GPU-based AI inference workloads. This hands-on management role sets the technical roadmap, supports engineering execution, and develops the team as it grows.
About the company
Baseten is an inference cloud that helps companies run and scale AI models in production with flexible infrastructure and developer tooling.
The team
Inference Performance team within the Inference Runtime organization.
Stack
GPUsPyTorchTensorRTTensorRT-LLMvLLMSGLangCUDATriton
What you'll do
- Lead, mentor, hire, and develop inference performance engineers.
- Own the runtime performance roadmap and balance customer priorities with platform investments.
- Guide profiling, design reviews, and GPU optimization work.
- Productionize techniques such as quantization, speculative decoding, KV-cache reuse, and chunked prefill.
- Measure improvements in throughput, utilization, latency, and cost.
- Coordinate model and hardware bring-up with partner teams.
What we're looking for
- Experience managing engineers, including hiring, mentoring, feedback, and performance reviews.
- Experience leading or closely supporting GPU optimization teams.
- Strong understanding of GPU architecture and performance tradeoffs.
- Familiarity with ML libraries such as PyTorch, TensorRT, or TensorRT-LLM.
- Track record of driving roadmaps and shipping complex technical projects.
- Bachelor's degree or higher in a relevant field.
Nice to have
- Familiarity with inference engines such as vLLM, SGLang, or TensorRT-LLM.
- Production experience with LLM optimization techniques.
- GPU kernel experience with CUDA, Triton, CUTLASS, or similar.
- Experience scaling a team at a fast-growing startup.
- Prior hands-on performance or systems engineering experience.
Benefits
- Meaningful equity.
- 100% medical, dental, and vision insurance coverage for U.S. employees and dependents.
- Flexible PTO and company-wide Winter Break.
- Paid parental leave.
- Fertility and family-building stipend.
- Company-facilitated 401(k) for U.S. employees.