Data Engineer

Data Engineer - Data Integration in Pune

As a Data Engineer at IBM in Pune, you will be instrumental in transforming raw data into actionable insights. Your role will involve contributing to the entire data lifecycle, from gathering and storage to batch and real-time processing. You will collaborate with cross-functional teams to select optimal data management systems and identify critical data for analysis. This position offers the opportunity to address challenges in database integration and work with complex, unstructured datasets.

Key Responsibilities:

• Develop and maintain robust data pipelines for both batch and stream processing, utilizing tools such as Informatica PowerCenter or cloud-based ETL/ELT solutions.
• Liaise effectively with business teams and technical leads to gather requirements, identify data sources, diagnose data quality issues, design target data structures, develop processing routines, and support unit testing and User Acceptance Testing (UAT).
• Collaborate with data scientists and business analytics teams to facilitate data ingestion and resolve data-related technical challenges.
• Implement and validate predictive models, and create/maintain statistical models focusing on big data, incorporating various statistical and machine learning techniques.
• Design and implement enterprise search applications like Elasticsearch and Splunk according to client specifications.
• Operate within an Agile, collaborative environment, partnering with scientists, engineers, consultants, and database administrators from diverse backgrounds to apply analytical rigor and statistical methods to behavioral prediction challenges.
• Build teams or develop programs for efficient and reusable data cleansing and integration, create predictive or prescriptive models, and evaluate modeling outcomes.

Qualifications:

Required Education:

• Bachelor's Degree

Preferred Education:

• Master's Degree

Required Technical and Professional Expertise:

• Expertise in Data Warehousing, Information Management, Data Integration, and Business Intelligence using ETL tools like Informatica PowerCenter.
• Proficiency with cloud technologies, Power BI, and cloud data migration skills.
• Experience in Unix shell scripting and Python.
• Experience with relational SQL and Big Data technologies.

Preferred Technical and Professional Experience:

• Knowledge of Microsoft Azure Cloud.
• Experience with Informatica PowerCenter.
• Experience in Unix shell scripting and Python.