Freshworks is seeking a skilled Data Engineer to lead the design, development, and management of scalable data pipelines and systems that drive analytics, insights, and business intelligence. This pivotal role involves direct ownership of our Data Lake, Databricks platform, and real-time data streaming pipelines utilizing technologies such as MySQL, Kafka, and Spark. The objective is to ensure the seamless, secure, and efficient flow of data across the organization.
Key Responsibilities:
- Oversee and manage the enterprise Data Lake infrastructure on AWS and Databricks, ensuring optimal reliability, scalability, and governance.
- Design, develop, and refine data ingestion and transformation pipelines, migrating data from MySQL to Kafka (via CDC pipelines) and from Kafka to Databricks using Spark Structured Streaming.
- Leverage strong expertise in designing and implementing MapReduce jobs for efficient processing and transformation of large-scale datasets in distributed environments.
- Optimize MapReduce workflows to enhance performance, fault tolerance, and scalability within big data ecosystems.
- Construct robust batch and near real-time data pipelines capable of efficiently managing high-volume, high-velocity data.
- Develop and maintain metadata-driven data processing frameworks to ensure consistency, lineage, and traceability throughout the data lifecycle.
- Implement and manage comprehensive observability and monitoring systems, including logging, metrics, and alerting, utilizing tools like Prometheus, Grafana, or equivalent.
- Collaborate closely with Product, Regulatory, and Security teams to uphold compliance, data privacy, and data quality standards across all data operations.
- Partner with cross-functional teams to build integrated, end-to-end data lakehouse solutions that incorporate multiple systems and data sources.
- Adhere to best practices in code quality, CI/CD automation (Jenkins, GitHub Actions), and Infrastructure as Code (IaC) to ensure consistent deployments.
- Guarantee system reliability and scalability through proactive monitoring, performance tuning, and fault-tolerant architectural designs.
- Stay abreast of the latest advancements in data engineering, streaming technologies, and distributed systems, driving continuous improvement initiatives.
Required Qualifications:
- Proficient programming skills in Scala, Spark, Java, or Python.
- A minimum of 6 to 9 years of relevant professional experience.
- Demonstrated experience working with Kafka (Confluent or Apache) for constructing event-driven or CDC-based pipelines.
- Hands-on experience with distributed data processing frameworks such as Apache Spark, Databricks, or Flink for large-scale data management.
- A strong understanding of Kubernetes for deploying and managing scalable, resilient data workloads, with EKS experience being a significant advantage.
- Practical experience with essential AWS Cloud Services, including S3, Lambda, EMR, Glue, IAM, and CloudWatch.
- Experience in designing and managing data lakehouse architectures, ideally using Databricks or comparable platforms.
- Familiarity with metadata-driven frameworks and the principles of data governance, lineage, and cataloging.
- Experience implementing CI/CD pipelines (Jenkins, GitHub Actions) for data workflow deployment and automation.
- Experience with monitoring and alerting frameworks like Prometheus, Grafana, or the ELK stack.
- Exceptional problem-solving, communication, and collaboration abilities.
