Senior Data Engineer

Nielsen is seeking a Senior Data Engineer to serve as a technical leader in designing and driving the architecture of our extensive data infrastructure. This role involves close collaboration with cross-functional teams to establish best practices, mentor fellow engineers, and ensure our data systems are scalable, reliable, and prepared for future demands. You will be instrumental in optimizing data workflows, implementing robust governance strategies, and making architectural decisions that align with strategic business objectives. The position is based in Bangalore and Gurgaon.

Responsibilities:

• Lead the design and implementation of scalable, high-performance data architectures to support analytics, machine learning, and operational data requirements.
• Define technical roadmaps and best practices for data engineering, ensuring alignment with business objectives.
• Architect and optimize real-time and batch data pipelines for petabyte-scale data processing.
• Drive initiatives for data quality, lineage, governance, and security across the organization.
• Mentor and coach Senior Data Engineers, fostering their technical and professional growth.
• Collaborate with ML Engineers, Product Managers, and Software Engineers to develop robust, production-ready data products.
• Evaluate and introduce emerging technologies such as Iceberg, Delta Lake, and Apache Hudi to future-proof our data stack.
• Enhance and optimize query performance and storage strategies for structured and unstructured data.
• Build self-service data platforms that empower analytics teams while ensuring adherence to security and compliance standards.

Requirements:

• Extensive expertise in distributed computing frameworks like Apache Spark, Flink, or Beam.
• Strong proficiency in programming languages such as Python, Spark, SQL, Scala, or Java, emphasizing performance and maintainability.
• Expert-level knowledge of data storage systems, including relational databases (PostgreSQL, MySQL), columnar stores (Parquet, ORC), and distributed storage solutions.
• Hands-on experience with streaming architectures (Kafka, Kinesis, Pulsar) and event-driven systems.
• Advanced SQL skills with a proven track record in query optimization and database indexing strategies.
• Strong cloud expertise across AWS, GCP, and Azure, with experience in building large-scale data platforms.
• Demonstrated experience in designing data lakehouse architectures, including Iceberg, Delta Lake, or Apache Hudi.
• Experience architecting multi-region, high-availability data systems.
• Experience building multi-tenant data platforms for large enterprises.
• Proficiency with Infrastructure as Code (Terraform, Kubernetes) for data workloads.
• Knowledge of privacy-preserving data techniques such as differential privacy and tokenization.
• Familiarity with feature engineering for machine learning models in production environments.