Software Engineer – AI Code Evaluat...

About the Role

We are conducting a paid study on the effectiveness of coding evaluation harnesses used to test AI agents. In this AI-moderated session, you will walk through a series of proposed coding tasks and environments, reviewing their structure, difficulty, and realism. You will provide technical feedback on the evaluation harnesses and suggest improvements to the test suites.

Key Responsibilities

• Review proposed coding tasks and programming environments for realism.
• Provide technical feedback on the structure and difficulty of evaluation harnesses.
• Walk us through your approach to verifying complex code challenges.
• Suggest improvements to make the test suites more robust.

Key Requirements

• Professional software engineer based in South Asia (backend developers, test automation engineers, and full-stack engineers welcome).
• Experience building or reviewing realistic programming tasks.
• Familiarity with coding evaluation harnesses or test automation.