Big Data Engineer

Kiya.ai is seeking a talented Big Data Engineer to join our team in Mumbai (Airoli), Navi Mumbai. This role involves developing and maintaining robust data pipelines and applications for big data processing and analysis. You will be instrumental in leveraging the Hadoop ecosystem and visualization tools to extract valuable insights.

Key Responsibilities:

• Develop and maintain data pipelines utilizing the Hadoop ecosystem.
• Build and maintain Spark SQL applications for efficient big data processing and analysis.
• Utilize Tableau for data visualization, dashboard creation, and extracting insights from Hadoop data.
• Optimize data processing performance, including query optimization and data storage strategies.
• Collaborate with cross-functional teams and stakeholders to understand data requirements and deliver effective solutions.
• Troubleshoot and resolve issues related to data processing and visualization workflows.
• Stay abreast of the latest technologies and best practices in the big data domain.
• Act as a Hadoop Administrator, responsible for the installation, configuration, monitoring, security, and tuning of the Apache Hadoop ecosystem, including HDFS, YARN, MapReduce, and related components like Hive, HBase, Spark, and Kafka.

Required Skills and Experience:

• Strong understanding of Hadoop ecosystem components, including HDFS, MapReduce, YARN, Hive, HBase, Spark, Kafka, NiFi, Pig, and Oozie.
• Proficiency in Spark SQL for querying and manipulating large datasets.
• Experience with data visualization tools such as Tableau or Power BI.
• Experience with at least one scripting language like Python, Scala, or Java.
• Knowledge of data warehousing concepts and data modeling techniques, with familiarity in platforms like Amazon Redshift, Google BigQuery, or Snowflake.
• Experience with ETL tools (e.g., Informatica, Pentaho) is considered a plus.
• Familiarity with cloud platforms (AWS, Azure, GCP) is beneficial.
• Excellent problem-solving and communication skills.
• Ability to work effectively both independently and as part of a collaborative team.

Soft Skills:

• Excellent problem-solving abilities.
• Strong communication skills.
• Capacity to work collaboratively in a team-oriented environment.

Additional Skills (Good to Have):

• ETL Tools: Experience with various ETL technologies and frameworks.
• Data Modelling: Knowledge of data modeling techniques and best practices.
• Security and Compliance: Understanding of data security and privacy regulations.
• Continuous Integration/Continuous Deployment (CI/CD): Familiarity with CI/CD pipelines and version control systems.