Databricks / Spark Data Engineer

Role Overview

We are seeking a skilled Data Engineer with robust Apache Spark fundamentals to build scalable data pipelines within cloud-based environments. You will leverage modern development tools and AI-assisted coding (e.g., GitHub Copilot) to enhance productivity.

Key Responsibilities

• Design, build, and maintain high-performance ingestion and processing pipelines using Apache Spark and Databricks.
• Optimize distributed data workloads for speed and efficiency.
• Leverage AWS ecosystem services including Amazon S3, Glue, Redshift, and SQS for cloud-native storage and integration.
• Support enterprise-scale data platform initiatives.

Required Skills

• Deep expertise in Apache Spark/Databricks architecture.
• Mastery of distributed processing concepts and performance tuning.
• Proficiency with AWS cloud services (S3, Glue, Redshift, SQS).
• Familiarity with data movement, ingestion strategies, and AI coding assistants.