AI Engineer – Python | Remote

Position: Senior Software Engineer – Python (LLM Evaluation & Repository Validation)
Type: Contractor Assignment (3 months)
Compensation: $19-$20/hr
Commitment: 40 hrs/week with some overlap with PST
Role Responsibilities:
Evaluate and analyze GitHub issues across widely used open-source repositories.
Set up and configure development environments for repositories, including Docker setup.
Modify and run codebases locally to evaluate LLM performance in debugging and bug-fixing scenarios.
Evaluate unit test coverage and assess the quality of existing tests.
Work with real-world Python repositories to identify engineering tasks suitable for LLM evaluation.
Collaborate with researchers to expand datasets used for training and evaluating AI models.
Identify repositories and issues that present challenging real-world scenarios for LLMs.
Support the creation of realistic software engineering tasks based on repository histories.
Requirements:
Strong experience in software development.
Strong experience with Python programming.
Proficiency in Git, Docker, and development environment setup.
Experience working with well-maintained public GitHub repositories.
Ability to understand, modify, and test complex codebases.
Strong analytical thinking and problem-solving skills.
Excellent written English and ability to document findings clearly.
Ability to work independently in a remote environment.
Desktop or laptop with a stable internet connection.