180K–360K USD / year
Software Engineer - Inference Platform
AIHybrid — San Francisco
Published on 2026-10-05
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.
About this role
Build and operate the distributed runtime and platform for large-scale LLM inference. The role spans developer-facing APIs, Kubernetes orchestration, traffic routing, and production reliability, with a focus on making model serving fast, scalable, and cost-efficient.
About the company
Baseten is an inference cloud that helps companies run and scale AI models in production with flexible infrastructure and developer tooling.
The team
EPD, Engineering, Inference
Stack
KubernetesvLLMSGLangTensorRT-LLMTGINVIDIA Dynamo
What you'll do
- Build systems for deploying and running distributed LLM inference, including routing, autoscaling, scheduling, and runtime management.
- Develop and operate Model APIs with structured outputs, tool calling, and multimodal serving.
- Implement API versioning, validation, usage metering, quotas, and authentication.
- Build observability and benchmarks for performance, reliability, and quality.
- Debug and harden production systems across Kubernetes, networking, distributed runtimes, and GPU workloads.
- Own projects end to end, from architecture and deployment to monitoring and iteration.
What we're looking for
- 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.
- Experience owning low-latency, reliable backend services, including rate limiting, authentication, quotas, metering, and migrations.
- Skills in profiling, tracing, capacity planning, and SLO management.
- Ability to debug performance and reliability across application, runtime, and infrastructure layers.
- Strong developer-experience instincts and written communication skills.
Nice to have
- Experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or Dynamo.
- Deep Kubernetes experience, including operators and custom resources.
- Experience with distributed scheduling, autoscaling, service orchestration, or production GPU workloads.
- Background in developer-facing infrastructure or APIs, or open-source infrastructure and ML systems.
- Familiarity with observability, CI/CD, or release automation.
Benefits
- Meaningful equity.
- Medical, dental, and vision insurance fully covered for U.S. employees and dependents.
- Flexible PTO and company-wide Winter Break.
- Paid parental leave.
- Fertility and family-building stipend.
- Company-facilitated 401(k) for U.S. employees.