Senior Data Engineer with Python

Join Intellectsoft as a Senior Data Engineer in Bengaluru, Karnataka, India, and contribute to an AI-powered platform designed to empower businesses with enhanced decision-making and operational efficiency. This platform leverages advanced analytics and machine learning to process vast datasets, providing actionable insights and predictive capabilities for industries like healthcare, process optimization, and customer experience improvement. It seamlessly integrates with existing systems, facilitating rapid, data-driven decisions.

Responsibilities

• Design and build robust, scalable data pipelines utilizing PySpark and big data technologies.
• Collaborate with data scientists to implement new features that boost model accuracy and performance.
• Develop standardized data models to ensure consistency across diverse deployments.
• Identify and resolve issues within existing ETL pipelines and optimize workflow efficiency.
• Conduct Proofs of Concept (POCs) to assess emerging technologies and integrate new data sources.
• Adhere to and champion software development best practices, ensuring the delivery of high-quality solutions on time.
• Document development progress and maintain comprehensive technical documentation.

Requirements

• A minimum of 4 years of professional experience, with at least 2 years focused on data engineering using Apache Spark and SQL.
• Strong proficiency in Python for data processing and automation tasks.
• Solid understanding of PySpark, distributed computing, analytical databases, and other big data technologies.
• Proven expertise in designing and managing ETL pipelines and distributed data processing frameworks.
• Extensive knowledge of database systems, data modeling, and analytical databases.
• Hands-on experience with workflow orchestration tools, such as Apache Airflow.
• Familiarity with major cloud platforms including AWS, GCP, or Azure.
• A firm grasp of software development lifecycles, including coding standards, version control, and testing methodologies.

Nice to Have Skills

• A Bachelor’s or Master’s degree in Computer Science or a related field.
• Exposure to data science and machine learning development processes.
• Understanding of Machine Learning pipelines or frameworks.

Benefits

• Engage in impactful projects with significant influence.
• Access Udemy courses tailored to your professional development goals.
• Participate in team-building events, marathons, and charity activities to foster connections and well-being.
• Benefit from workshops, trainings, and expert knowledge-sharing sessions to support continuous growth.
• Follow a clearly defined career path for advancement.
• Enjoy absence days dedicated to maintaining a healthy work-life balance.
• Experience flexible working hours and a remote-friendly setup, allowing you to work from anywhere and manage your day effectively.