Question 1

A data engineer observes that an upstream streaming source sends duplicate records, where duplicates share the same key and have at most a 30-minute difference inevent_timestamp. The engineer adds: dropDuplicatesWithinWatermark("event_timestamp", "30 minutes") What is the result?

Options :

A : It is not able to handle deduplication in this scenario

B : It removes duplicates that arrive within the 30-minute window specified by the watermark

C : It removes all duplicates regardless of when they arrive

D : It accepts watermarks in seconds and the code results in an error

Answer: B

Question 2

A data scientist of an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user. Before further processing the data, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns in this DataFrame. The PII columns in df_user are first_name, last_name, email, and birthdate. Which code snippet can be used to meet this requirement?

Options :

A : df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

B : df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

C : df_user_non_pii = df_user.dropfields("first_name", "last_name", "email", "birthdate")

D : df_user_non_pii = df_user.dropfields("first_name, last_name, email, birthdate")

Answer: A

Question 3

Which Spark configuration controls the number of tasks that can run in parallel on the executor? Options:

Options :

A : spark.executor.cores

B : spark.task.maxFailures

C : spark.driver.cores

D : spark.executor.memory

Answer: A

Question 4

A data engineer is working with a large JSON dataset containing order information. The dataset is stored in a distributed file system and needs to be loaded into a Spark DataFrame for analysis. The data engineer wants to ensure that the schema is correctly defined and that the data is read efficiently. Which approach should the data scientist use to efficiently load the JSON data into a Spark DataFrame with a predefined schema?

Options :

A : Use spark.read.json() to load the data, then use DataFrame.printSchema() to view the inferred schema, and finally use DataFrame.cast() to modify column types.

B : Use spark.read.json() with the inferSchema option set to true

C : Use spark.read.format("json").load() and then use DataFrame.withColumn() to cast each column to the desired data type.

D : Define a StructType schema and use spark.read.schema(predefinedSchema).json() to load the data.

Answer: D

Question 5

Given a DataFramedfthat has 10 partitions, after running the code: result = df.coalesce(20) How many partitions will the result DataFrame have?

Options :

A : 10

B : Same number as the cluster executors

C : 1

D : 20

Answer: A