Databricks QA / Tester at Prodapt Solutions - Chennai, Guindy

Prodapt Solutions is seeking a skilled Databricks QA/Tester with 2-4 years of experience to join their dynamic data engineering and analytics team in Chennai, Guindy. This role is crucial for ensuring the quality and reliability of data processing workflows.

About the Role

We are looking for a Databricks QA/Tester with 2-4 years of hands-on experience in Python, PySpark, and the Databricks platform. The primary focus will be on testing, validation, and quality assurance of ETL pipelines and distributed data processing workflows. You will play a key part in maintaining high standards for our data engineering and analytics solutions.

Key Responsibilities

• Design and implement comprehensive test plans, test cases, and automation scripts for ETL workflows built on Databricks.
• Validate data ingestion, transformation, and integration pipelines developed using PySpark and Python.
• Conduct functional, regression, and performance testing for large-scale distributed data processing jobs.
• Ensure data quality, consistency, and accuracy across all layers of the medallion architecture (Bronze, Silver, Gold).
• Collaborate effectively with developers, data engineers, and business stakeholders to define and execute QA strategies.
• Implement and maintain automated testing frameworks for Databricks notebooks and workflows.
• Identify, troubleshoot, and report defects, working towards timely resolution and continuous process improvement.
• Document all QA processes, test results, and best practices for knowledge sharing and future reference.

Required Skills & Qualifications

• 2-4 years of dedicated experience in QA/testing for data engineering or ETL pipelines.
• Strong proficiency in Python for scripting and automation.
• Expertise in PySpark for distributed data validation and testing.
• Hands-on experience with the Databricks platform, including notebooks, jobs, clusters, and Unity Catalog.
• Solid understanding of data warehousing concepts, SQL, and relational databases.
• Familiarity with modern data processing technologies such as Structured Streaming, Auto Loader, and Delta Lake.
• Excellent analytical and problem-solving abilities.
• Proven ability to collaborate effectively within agile team environments.

Preferred Qualifications

• Previous exposure to major cloud platforms like Azure, AWS, or GCP.
• Experience with CI/CD pipelines and version control systems such as Git.
• A foundational understanding of data governance, security, and compliance principles.
• Knowledge of test automation frameworks specifically designed for big data environments.

Join Prodapt Solutions and contribute to cutting-edge data solutions!