Podcaster

Position Overview

We are seeking an AI Voice Evaluation Specialist to compare the conversational performance of two AI models through real-time, voice-based interactions. The role involves recording conversations, assessing responses against defined criteria, identifying errors, and documenting a rationale for the preferred model.

Key Responsibilities:

• Review an assigned scenario and roleplay a natural conversation with two different AI models
• Conduct comparable conversations with Model A and Model B, keeping the scenario, conversational approach, and overall interaction as consistent as possible
• Maintain approximately the same number of conversational turns with each model to support a fair comparison
• Record your voice during each interaction and pay close attention to the quality and behavior of the model's responses
• Evaluate each model across five defined evaluation dimensions
• Identify and categorize relevant error clusters that may occur in the audio or conversation
• Compare the performance of Model A and Model B based on the evaluation criteria and your observations
• Select the model that demonstrated the stronger overall performance according to the defined evaluation framework
• Write a detailed rationale explaining your final preference, referencing specific examples and observations from both conversations
• Apply evaluation guidelines consistently across different scenarios and model interactions