Streaming Data engineer

Responsibilities:

• Develop and implement code solutions for specified scenarios.
• Leverage expertise in Spark queries.
• Demonstrate proficiency in Apache Flink for coding and troubleshooting pipeline-level issues.
• Apply core Python skills, including validation rules for CSV files, string manipulation, collection handling, and fundamental programming constructs.
• Optimize Spark performance and utilize Spark Submit commands effectively.
• Apply strong SQL knowledge, including joins, aggregations, and window functions.
• Exhibit excellent communication skills.
• Understand the fundamentals of streaming pipelines.
• Possess a foundational understanding of Spark Sessions, streaming processing, and data transformations within streaming contexts.
• Gain hands-on experience with Spark Streaming, Kafka, and Hive, or other relevant streaming technologies.
• Familiarity with basic Spark and Hive concepts.
• Understand overarching Spark concepts, including Sessions and Contexts.
• Showcase hands-on project experience with Spark Streaming or Flink Streaming integrated with Kafka.
• Familiarity with Azure Cloud services is required.

Potential Evaluation Queries:

• Differentiate between Spark Streaming sessions and Batch Sessions.
• Explain Spark Structured Streaming.
• Describe the use of the spark.readStream() method for reading data in Spark Streaming applications.
• Detail the functionality of the writeStream method and its various arguments (e.g., format, location, writeMode) for writing DataFrames to sinks.
• Identify the action required to initiate reading data from a Kafka queue: start().
• Demonstrate how to print the output of streaming operations to the terminal using the console format.