180K–360K USD / year
Software Engineer - Inference Performance
Machine learningHybrid — San Francisco, Toronto, New York, Montreal, Seattle
Published on 2026-10-05
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Optimize production LLM inference across runtimes, GPU kernels, scheduling, serving, and routing. The role focuses on improving model latency, throughput, and GPU efficiency, and bringing new model architectures onto new hardware.
About the company
Baseten is an inference cloud that helps companies run and scale AI models in production with flexible infrastructure and developer tooling.
The team
EPD, Engineering, Inference
Stack
PythonC++PyTorchTensorRTTensorRT-LLMvLLMSGLangCUDA
What you'll do
- Implement and productionize inference techniques such as quantization, speculative decoding, KV-cache reuse, and chunked prefill.
- Profile and optimize inference across kernels, memory layout, scheduling, batching, routing, and prefill/decode disaggregation.
- Improve GPU utilization and tokens per GPU-hour while evaluating latency, throughput, and cost tradeoffs.
- Bring up and tune new model architectures on new hardware.
- Build benchmarks across models, sequence lengths, batch sizes, and hardware configurations.
- Contribute to open-source inference engines and collaborate with model, infrastructure, and customer-facing teams.
What we're looking for
- Degree in computer science, engineering, mathematics, or a related field.
- Experience with a general-purpose programming language such as Python or C++.
- Familiarity with LLM optimization techniques, including quantization, speculative decoding, or continuous batching.
- Strong familiarity with ML libraries such as PyTorch, TensorRT, or TensorRT-LLM.
- Demonstrated interest and experience in LLMs.
- Deep understanding of GPU architecture.
Nice to have
- Performance optimization experience for software systems, particularly LLMs.
- Contributions to vLLM, SGLang, TensorRT-LLM, or another inference engine.
- Experience with large-scale distributed serving, autoscaling, load balancing, or multi-region infrastructure.
- GPU kernel development or optimization using CUDA, Triton, CUTLASS, or similar.
- Production experience with quantization or speculative decoding.
- Track record developing and deploying AI/ML inference solutions.
Benefits
- Meaningful equity.
- Medical, dental, and vision coverage for employees and dependents in the U.S.
- Flexible PTO and company-wide winter break.
- Paid parental leave.
- Fertility and family-building stipend.
- Company-facilitated 401(k) in the U.S.