AI Engineer – Python | Remote

About the Job

Position: Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

Type: Contractor Assignment (3 months)

Compensation: $19-$20/hr

Location: Remote

Commitment: 40 hrs/week with some overlap with PST

Role Responsibilities

• Evaluate and analyze GitHub issues across widely used open-source repositories.
• Set up and configure development environments for repositories, including Docker setup.
• Modify and run codebases locally to evaluate LLM performance in debugging and bug-fixing scenarios.
• Evaluate unit test coverage and assess the quality of existing tests.
• Work with real-world Python repositories to identify engineering tasks suitable for LLM evaluation.
• Collaborate with researchers to expand datasets used for training and evaluating AI models.
• Identify repositories and issues that present challenging real-world scenarios for LLMs.
• Support the creation of realistic software engineering tasks based on repository histories.

Requirements

• Strong experience in software development.
• Strong experience with Python programming.
• Proficiency in Git, Docker, and development environment setup.
• Experience working with well-maintained public GitHub repositories.
• Ability to understand, modify, and test complex codebases.
• Strong analytical thinking and problem-solving skills.
• Excellent written English and ability to document findings clearly.
• Ability to work independently in a remote environment.
• Desktop or laptop with a stable internet connection.

Preferred Qualifications

• Experience contributing to open-source projects.
• Exposure to LLM evaluation, AI research, or machine learning projects.
• Experience with developer tools, automation agents, or CI/CD pipelines.

Application Process

1. Application Form
2. ICF + Profile Review
3. Assessment
4. Submit

Desired Skills and Experience

• Analysis
• Python
• GitHub
• TECHNICAL
• LLM