Remote Python Engineer

About The Projects

We are building large language model (LLM) evaluation and training datasets focused on realistic software engineering problems. Our approach involves creating verifiable Software Engineering (SWE) tasks based on public repository histories using a synthetic method with human-in-the-loop validation, expanding dataset coverage across various programming languages, difficulty levels, and task types.

About The Role

We are seeking experienced software engineers at the tech lead level who possess deep familiarity with high-quality public GitHub repositories. This role involves hands-on engineering work including:

• Development environment automation setup (Dockerization).
• Triage of issues across trending open-source libraries.
• Evaluation of unit test coverage and quality metrics.
• Modifying codebases locally to assess LLM performance in bug-fixing scenarios.

This is a unique opportunity to blend practical software engineering with AI research, influencing the future of AI-assisted development. You may also lead junior engineers on collaborative projects.

Daily Responsibilities

• Analyze and prioritize GitHub issues across trending open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and quality assurance protocols.
• Modify and run complex codebases locally to assess LLM performance in bug-fixing scenarios.
• Collaborate with researchers to design strategies for identifying challenging repositories and issues for LLMs.