Question 1

A data engineer wants to use Autoloader to ingest input data into a target table, and automatically evolve the schema of the table when new fields are detected.

They use the below query with a blank:

1. spark.readStream

2. .format("cloudFiles")

3. .option("cloudFiles.format", "json")

4. .option("cloudFiles.schemaLocation", checkpointPath)

5. .load(source_path)

6. .writeStream

7. .option("checkpointLocation", checkpointPath)

8. .___________

9. .start("target_table")

Which option correctly fills in the blank to meet the specified requirement ?

Options :

A : option("cloudFiles.schemaEvolutionMode", "addNewColumns")

B : option("cloudFiles.mergeSchema", True)

C : option("mergeSchema", True)

D : schema(schema_definition, mergeSchema=True)

E : Autoloader can not automatically evolve the schema of the table when new fields are detected

Answer: C

Question 2

A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema:

user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING

The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched.

Which solution minimizes the compute costs to propagate this batch of data?

Options :

A : Perform a batch read on the reviews_raw table and perform an insert-only merge using the natural composite key user_id, review_id, product_id, review_timestamp.

B : Configure a Structured Streaming read against the reviews_raw table using the trigger once execution mode to process new records as a batch job.

C : Use Delta Lake version history to get the difference between the latest version of reviews_raw and one version prior, then write these records to the next table.

D : Filter all records in the reviews_raw table based on the review_timestamp; batch append those records produced in the last 48 hours.

E : Reprocess all records in reviews_raw and overwrite the next table in the pipeline.

Answer: A

Question 3

The following code has been migrated to a Databricks notebook from a legacy workload:

The code executes successfully and provides the logically correct results, however, it takes over 20 minutes to extract and load around 1 GB of data. Which statement is a possible explanation for this behavior?

Options :

A : %sh triggers a cluster restart to collect and install Git. Most of the latency is related to cluster startup time

B :

Instead of cloning, the code should use %sh pip install so that the Python code can get executed in parallel across all nodes in a cluster.

C : %sh does not distribute file moving operations; the final line of code should be updated to use %fs instead.

D :

Python will always execute slower than Scala on Databricks. The run.py script should be refactored to Scala

E :

%sh executes shell code on the driver node. The code does not take advantage of the worker nodes or Databricks optimized Spark.

Answer: C

Question 4

Which of the following statements correctly describes End-to-End Testing ?

Options :

A : It’s an approach to simulate a user experience to ensure that the application can run properly under real-world scenarios

B : It’s an approach to test the interaction between subsystems of an application to ensure that modules work properly as a group.

C : It’s an approach to test individual units of code to determine whether they still work as expected if new changes are made to them in the future

D : It’s an approach to verify if each feature of the application works as per the business requirements

E : It’s an approach to measure the reliability, speed, scalability, and responsiveness of an application

Answer: A

Question 5

The data engineering team maintains the following code:

Databricks-Certified-Professional-Data-Engineer-page61-image9

Assuming that this code produces logically correct results and the data in the source tables has been de-duplicated and validated, which statement describes what will occur when this code is executed?

Options :

A :

A batch job will update the enriched_itemized_orders_by_account table, replacing only those rows that have different values than the current version of the table, using accountID as the primary key.

B :

The enriched_itemized_orders_by_account table will be overwritten using the current valid version of data in each of the three tables referenced in the join logic.

C :

An incremental job will leverage information in the state store to identify unjoined rows in the source tables and write these rows to the enriched_iteinized_orders_by_account table.

D :

An incremental job will detect if new rows have been written to any of the source tables; if new rows are detected, all results will be recalculated and used to overwrite the enriched_itemized_orders_by_account table

E :

No computation will occur until enriched_itemized_orders_by_account is queried; upon query materialization, results will be calculated using the current valid version of data in each of the three tables referenced in the join logic.

Answer: B