Data Engineer II
Responsibilities
Qualifications & Requirements
Experience Level: Mid Level
Full Job Description
Ai Palette seeks a passionate Data Engineer II to join our Data Engineering division in Bengaluru, Karnataka, India. In this role, you will contribute to the core development of our AI Platform, which is designed to ideate future consumer products. You will collaborate closely with Data Science and Full Stack teams to tackle complex challenges in data engineering and cloud computing, managing millions of data points, including social media data. This is an excellent opportunity to work with and scale a growing data platform, solving challenging problems and finding your 'ikigai' in a role that excites you.
Responsibilities
- Develop scalable data collection pipelines to acquire and manage millions of public data points from various sources using APIs and web extraction techniques.
- Construct robust data cleaning, quality, and integrity pipelines within the platform, utilizing Apache Spark with Python and AWS services.
- Architect and develop distributed systems capable of handling large-scale data processing.
- Programming languages: Python/Java, Apache Spark, Apache Flink (a plus).
- NoSQL databases: Elasticsearch, DynamoDB/MongoDB.
- Collaborate with the Data Science Team on preprocessing steps for AI models in production.
- Scale and automate the Data Platform Collection layer to manage continuous data ingestion.
- Implement comprehensive data quality checks and validation processes.
- Troubleshoot and resolve performance-related issues.
- Partner with cross-functional teams to support data initiatives.
- Document data pipelines, data models, and other technical processes.
Requirements
- 4-6 years of experience in building data engineering pipelines.
- Hands-on experience with Apache Spark, PySpark, SQL, and Python programming.
- Experience with NoSQL databases such as Elasticsearch, DynamoDB, or Cassandra.
- Strong experience with AWS Cloud Platform services including S3, EC2, Lambda, and Elasticsearch.
- A quick learner, excellent communicator, and a strong team player.
Ideal Candidate Profile
- Deep technical understanding of AWS PaaS and IaaS services.
- Prior experience with tools like Airflow, Spark, PySpark, Python, Elasticsearch, Web Crawling, and Docker.
- Previous experience working with social media data (e.g., Twitter, Reddit, blogs).
Benefits
- Opportunities for career growth and development as the company expands.
- Comprehensive health insurance, including hospital and surgical coverage.
- Sponsored training through MOOCs (e.g., Coursera, Udemy) for skill enhancement.
We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
Company
Ai Palette
Ai Palette is a pioneering Food AI company dedicated to transforming product innovation. We empower food companies to create products that truly resonate with consumers by leveraging an AI-powered Saa...