Question 1

Where in the Spark UI can one diagnose a performance problem induced by not leveraging predicate pushdown?

Options :

A : In the Executor’s log file, by grepping for "predicate push-down"

B :

In the Stage’s Detail screen, in the Completed Stages table, by noting the size of data read from the Input column

C : In the Storage Detail screen, by noting which RDDs are not stored on disk

D : In the Delta Lake transaction log. by noting the column statistics

E : In the Query Detail screen, by interpreting the Physical Plan

Answer: E

Question 2

Which statement regarding static Delta tables in Stream-Static joins is correct?

Options :

A : Static Delta tables must be small enough to be broadcasted to all worker nodes in the cluster.

B : Static Delta tables need to be partitioned in order to be used in stream-static join.

C : Static Delta tables need to be refreshed with REFRESH TABLE command for each microbatch of a stream-static join

D : The latest version of the static Delta table is returned each time it is queried by a microbatch of the stream-static join

E : The latest version of the static Delta table is returned only for the first microbatch of the stream-static join. Then, it will be cached to be used by any upcoming microbatch.

Answer: D

Question 3

The data governance team is reviewing code used for deleting records for compliance with GDPR. The following logic has been implemented to propagate delete requests from the user_lookup table to the user_aggregates table.

Assuming that user_id is a unique identifying key and that all users that have requested deletion have been removed from the user_lookup table, which statement describes whether successfully executing the above logic guarantees that the records to be deleted from the user_aggregates table are no longer accessible and why?

Options :

A : No; the Delta Lake DELETE command only provides ACID guarantees when combined with the MERGE INTO command.

B : No; files containing deleted records may still be accessible with time travel until a VACUUM command is used to remove invalidated data files.

C : Yes; the change data feed uses foreign keys to ensure delete consistency throughout the Lakehouse.

D : Yes; Delta Lake ACID guarantees provide assurance that the DELETE command succeeded fully and permanently purged these records.

E : No; the change data feed only tracks inserts and updates, not deleted records.

Answer: B

Question 4

A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema:

user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING

The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched.

Which solution minimizes the compute costs to propagate this batch of data?

Options :

A : Perform a batch read on the reviews_raw table and perform an insert-only merge using the natural composite key user_id, review_id, product_id, review_timestamp.

B : Configure a Structured Streaming read against the reviews_raw table using the trigger once execution mode to process new records as a batch job.

C : Use Delta Lake version history to get the difference between the latest version of reviews_raw and one version prior, then write these records to the next table.

D : Filter all records in the reviews_raw table based on the review_timestamp; batch append those records produced in the last 48 hours.

E : Reprocess all records in reviews_raw and overwrite the next table in the pipeline.

Answer: A

Question 5

A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams. The proposed directory structure is displayed below:

Databricks-Certified-Professional-Data-Engineer-page61-image5

Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?

Options :

A : No; Delta Lake manages streaming checkpoints in the transaction log.

B : Yes; both of the streams can share a single checkpoint directory.

C : No; only one stream can write to a Delta Lake table

D : Yes; Delta Lake supports infinite concurrent writers.

E : No; each of the streams needs to have its own checkpoint directory.

Answer: E