Principal Software Engineer

Role Overview

We are seeking a Principal Software Engineer for the AI Inference Platform at Rafay. This role sits at the intersection of deep distributed systems and cutting-edge LLM serving technologies, building the platform that turns GPU capacity into consumable revenue.

Key Responsibilities

• Architecture: Own the architecture of core platform services including the multi-tenant control plane for scheduling and lifecycle management across heterogeneous fleets.
• Serving Path: Design inference systems that maintain low latency under heavy load while maximizing hardware utilization.
• Multitenancy: Implement commercial-grade isolation, quotas, fairness policies, and noisy-neighbor mitigation strategies.
• Leadership: Set technical direction for major platform areas, conduct code reviews, mentor engineers, and define the engineering bar across teams.

Requirements

• 12+ years of experience building large-scale production systems with deep distributed systems fundamentals (consistency, availability, scalability).
• Expert-level proficiency in Go or Python for microservices design.
• Strong understanding of containerization and Kubernetes.
• Familiarity with GPU infrastructure, modern LLM serving stacks, NVLink topology, and MIG/time-slicing techniques.