Senior AI Test Engineer - Bengaluru, India

EY is seeking a skilled Senior AI Test Engineer to join our dynamic team in Bengaluru, India. This role is crucial for developing and executing comprehensive test plans and test cases for AI models and systems, ensuring their performance, reliability, and accuracy. You will collaborate closely with data scientists, software engineers, and product managers to define testing requirements and deliver high-quality AI products.

Key Responsibilities

Test Planning and Design

• Collaborate with cross-functional teams to define testing strategies for AI applications.
• Develop comprehensive test plans and test cases based on functional and non-functional requirements.
• Demonstrate a strong understanding of the various testing touchpoints within a model development lifecycle.
• Define test cases to validate input data, model functional performance, and model response.

Test Execution

• Execute manual and automated tests for AI models and systems.
• Monitor and document test results, identifying defects and areas for improvement.
• Utilize AI-specific testing methods such as Pairwise testing, Metamorphic testing, Back-to-Back testing, Bias Testing, and Drift testing.
• Apply knowledge of responsible AI testing principles.
• Leverage knowledge of explainability testing tools (e.g., LIME, SHAP).

Model Validation

• Possess a good understanding of performance benchmarks across different ML models.
• Validate AI models against performance benchmarks and metrics (e.g., accuracy, precision, recall).
• Conduct exploratory testing to uncover edge cases and potential failure modes.
• Gain exposure to testing LLM models.

Data Management

• Work with data engineers to ensure the quality and consistency of training and validation datasets.
• Implement data validation checks to assess the integrity of input data used for AI models.
• Demonstrate prior experience in performing data validation activities including data collection/generation, data augmentation, exploratory data analysis, and addressing data bias and privacy concerns.

Automation Development

• Develop and maintain automated testing frameworks and scripts for AI applications.
• Utilize tools and libraries such as TensorFlow and PyTorch for testing AI models.

Defect Tracking and Reporting

• Track and manage defects using issue-tracking tools like JIRA or Bugzilla.
• Prepare detailed reports on test results, defect status, and the overall quality of AI systems.

Continuous Improvement

• Participate in post-mortem analyses of testing processes and results to identify areas for improvement.
• Stay current with industry best practices in AI testing and testing methodologies.

Collaboration and Communication

• Communicate effectively with technical and non-technical stakeholders regarding testing progress and outcomes.
• Provide training and support to team members on AI testing tools and techniques.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field. A Master’s degree is preferred.
• 5-9 years of experience in software testing or development, with exposure to Python.
• 1-2 years of experience specifically in testing AI/ML applications.

Technical Skills

• Strong understanding of AI/ML concepts, algorithms, and frameworks (e.g., TensorFlow, PyTorch).
• Knowledge of ML/LLM frameworks and libraries.
• Proficiency in programming languages commonly used in AI testing, such as Python.
• Experience with test automation tools (e.g., Selenium, TestNG) and frameworks.
• Working knowledge of AI testing platforms and tools (e.g., Functionize, Applitools, Testim).
• Familiarity with data validation, model evaluation, and statistical analysis techniques.
• Proficiency in programming languages such as Python, Java, or C++.
• Experience with testing frameworks and tools (e.g., Selenium, PyTest, unittest).
• Familiarity with version control systems (e.g., Git) and continuous integration/continuous deployment (CI/CD) pipelines.
• Knowledge of data preprocessing, feature engineering, and model evaluation techniques.
• Excellent analytical and problem-solving skills.
• Strong attention to detail and a commitment to quality.

Preferred Qualifications

• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and their AI services.
• Familiarity with big data technologies (e.g., Hadoop, Spark).
• Knowledge of Agile methodologies and experience working in Agile teams.

EY is committed to building a better working world by creating new value for clients, people, society, and the planet, while building trust in capital markets. Enabled by data, AI, and advanced technology, EY teams help clients shape the future with confidence and develop answers to today's and tomorrow's most pressing issues. EY teams offer services across assurance, consulting, tax, strategy, and transactions, providing expertise in over 150 countries and territories.