200K–400K USD / year

Software Engineer - Inference Product

BackendHybrid — San Francisco, New York
Published on 2026-11-19
These details were extracted automatically from the original listing. They may not be complete or fully up to date — it's worth checking the original job post.

About this role

Build and operate the API layer behind Baseten’s hosted Model APIs. The role focuses on reliable, low-latency services for authentication, routing, rate limiting, usage metering, and billing, with direct impact on developer experience and AI inference products.

About the company

Baseten is an inference cloud that helps companies run and scale AI models in production with flexible infrastructure and developer tooling.

The team

Inference Platform team, partnering closely with Model APIs, inference, and product teams.

What you'll do

  • Build and operate highly available, low-latency APIs for request handling, authentication, rate limiting, and routing.
  • Develop real-time usage metering and billing infrastructure.
  • Build rate-limiting and capacity systems that balance customer throughput with GPU fleet utilization.
  • Ship API-layer products and developer-facing features.
  • Improve usage visibility, self-serve controls, and predictable API behavior.
  • Instrument services with observability and alerting.
  • Own projects from design through deployment, on-call, and iteration.

What we're looking for

  • 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.
  • Experience owning reliable, low-latency backend services, including rate limiting, authentication, quotas, metering, or migrations.
  • Knowledge of profiling, tracing, capacity planning, and SLO management.
  • Ability to debug performance and reliability across application, runtime, and infrastructure layers.
  • Strong developer-experience instincts and written communication skills.
  • Interest in inference engineering; prior ML or LLM experience is not required.

Nice to have

  • Deep Kubernetes experience, including operators and custom resources.
  • Familiarity with service meshes or API gateways.
  • Experience with distributed scheduling, autoscaling, or service orchestration.
  • Background in developer-facing infrastructure or APIs, or open-source infrastructure or ML contributions.
  • Familiarity with observability, CI/CD, or release automation.

Benefits

  • Meaningful equity.
  • 100% medical, dental, and vision coverage for U.S. employees and dependents.
  • Flexible PTO and company-wide Winter Break.
  • Paid parental leave.
  • Fertility and family-building stipend.
  • Company-facilitated 401(k) for U.S. employees.
View original job post