Streaming Data engineer

We are seeking a skilled Streaming Data Engineer to join our team. In this role, you will be responsible for designing, developing, and maintaining robust streaming data pipelines. You will write code to address specific scenarios, leveraging your knowledge of Spark and Flink for efficient data processing and troubleshooting. Proficiency in core Python, including validation rules, string manipulation, collections, and basic constructs, is essential. You will also optimize Spark performance, utilize Spark Submit commands, and demonstrate strong SQL skills, particularly with join, aggregate, and window functions. A solid understanding of streaming pipeline fundamentals, including Spark Sessions, streaming processing, and data transformation, is required. Experience with Spark Streaming, Kafka, and Hive is crucial. Hands-on project experience with Spark Streaming or Flink Streaming combined with Kafka is highly desirable. Familiarity with Azure cloud services is also a plus. Excellent communication skills are necessary for collaboration and problem-solving. Candidates may be evaluated on their understanding of concepts like the difference between Spark Streaming and Batch Sessions, Spark Structured Streaming, the usage of spark.readStream() and writeStream() methods, and the action required to start reading from a Kafka queue (start()), along with printing streaming output to the console.