Senior Engineer 2 – Inference Data Plane

Auto Import

Job Description:

  • Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models
  • Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards
  • Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing
  • Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs
  • Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement
  • Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health

Requirements:

  • Strong experience with microservices, messaging systems, databases, and infrastructure as code
  • Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or Modular
  • Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve
  • Understanding of GPU-level optimization and experience with interconnect technologies like NVlink, XGMI, or RoCE
  • Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization)
  • Expert-level proficiency in GoLang or Python
  • Familiarity with gRPC
  • Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean
  • Experience integrating and building with open-source software

Benefits:

  • Competitive salary
  • Employee Assistance Program
  • Local Employee Meetups
  • Flexible time off policy
  • Reimbursement for relevant conferences, training, and education
  • Access to LinkedIn Learning's 10,000+ courses
  • Equity compensation including grants upon hire and Employee Stock Purchase Program
Back to blog