Lead Data Engineer

About the Role

Talworx is seeking a Lead Data Engineer for a client in the Fintech Product space. This is an opportunity to join a dynamic team focused on supporting critical technology systems and guiding business partners with industry best practices and innovative solution design. You will contribute to creating long-term value for customers within an engaging, global, and collaborative work environment that encourages thoughtful risk-taking and self-initiative.

Job Summary

As a key member of the Cognitive Engineering team, you will be responsible for building and maintaining enterprise-scale data extraction, automation, and ML model deployment pipelines. You will design and implement resilient, production-ready systems within an AWS-based ecosystem. This role involves close collaboration with a hands-on, technically adept global team to tackle high-complexity problems from inception to completion.

Key Responsibilities

• Develop, deploy, and operate data extraction and automation pipelines in production environments.
• Integrate and deploy machine learning models into existing pipelines, supporting functionalities like inference services and batch scoring.
• Lead the successful delivery of complex projects involving data extraction, transformation, and ML model deployment.
• Scale data pipelines on AWS services such as EKS, ECS, and Lambda.
• Manage DataOps processes using tools like Celery, Redis, and Airflow.
• Implement robust CI/CD pipelines using Azure DevOps and ensure comprehensive test coverage for all developed solutions.
• Enhance data quality and reliability through effective logging, metrics tracking, and automated alerting systems.
• Collaborate closely with data scientists, ML engineers, and product teams to ensure alignment on requirements and delivery timelines.

Technical Requirements

• 2 to 6 years of relevant experience in data engineering, automation, or ML model deployment.
• Expert proficiency in Python, including experience building extraction libraries and RESTful APIs.
• Hands-on experience with task queues and orchestration tools such as Celery, Redis, and Airflow.
• Strong expertise in AWS services, including EKS/ECS, Lambda, S3, and RDS/DynamoDB.
• Experience with containerization and orchestration technologies, with Docker as mandatory and basic Kubernetes experience being preferred.
• Proven experience deploying machine learning models to production environments.
• Solid understanding of CI/CD practices and hands-on experience with Azure DevOps.
• Familiarity with both SQL and NoSQL data stores, such as PostgreSQL and MongoDB.