SDE 3

Responsibilities:

• Design, build, and own production-grade LLM agents integrated into real healthcare workflows.
• Develop backend services, APIs, and tools that these agents utilize for operations.
• Create agent workflows encompassing tool calling, orchestration, context management, memory systems, guardrails, and failure handling protocols.
• Construct rigorous evaluation frameworks using golden datasets, deterministic checks, LLM-as-judge methods, human review processes, and trajectory evaluations.
• Establish and track quality metrics including precision/recall/F1 scores, task success rates, tool-call accuracy, and regression analysis.
• Implement eval-gated release pipelines featuring automated regression suites, shadow/canary deployments, and rollback mechanisms.
• Instrument agents in production environments to monitor quality, reliability, latency, and cost efficiency.
• Troubleshoot failures across prompts, models, tools, and underlying backend infrastructure.
• Make strategic decisions on model selection balancing accuracy, cost, and latency requirements.
• Determine appropriate solutions by distinguishing when AI agents are optimal versus deterministic software approaches.

Requirements:

• Possess 6+ years of total software engineering experience with a proven track record owning production backend systems.
• Showcase 1+ year of hands-on experience building LLM agents in production environments serving real users.
• Demonstrate ownership of at least one agent end-to-end, including design, prompts, tools, evaluations, deployment, and post-launch optimization.
• Maintain strong fundamentals in backend engineering and distributed systems architecture.
• Have practical experience evaluating AI systems against real datasets with clearly defined metrics.
• Demonstrate robust understanding of APIs, production architecture patterns, reliability principles, and observability practices.
• Exhibit ability to reason about model choice, prompting strategies, tool design, context utilization, and failure modes.
• Show strong ownership mindset and capability for cross-functional collaboration across engineering, product, and domain teams.