Question 1

The data engineering team has a singleplex bronze table called ‘orders_raw’ where new orders data is appended every night. They created a new Silver table called ‘orders_cleaned’ in order to provide a more refined view of the orders data.

The team wants to create a batch processing pipeline to process all new records inserted in the orders_raw table and propagate them to the orders_cleaned table.

Which solution minimizes the compute costs to propagate this batch of data?

Options :

A : Use time travel capabilities in Delta Lake to compare the latest version of orders_raw with one version prior, then write the difference to the orders_cleaned table.

B : Use Spark Structured Streaming to process the new records from orders_raw in batch mode using the trigger availableNow option

C : Use Spark Structured Streaming-s foreachBatch logic to process the new records from orders_raw using trigger(processingTime=”24 hours")

D : Use batch overwrite logic to reprocess all records in orders_raw and overwrite the orders_cleaned table

E : Use insert-only merge into the orders_cleaned table using orders_raw data based on a composite key

Answer: B

Question 2

A data engineer wants to use Autoloader to ingest input data into a target table, and automatically evolve the schema of the table when new fields are detected.

They use the below query with a blank:

1. spark.readStream

2. .format("cloudFiles")

3. .option("cloudFiles.format", "json")

4. .option("cloudFiles.schemaLocation", checkpointPath)

5. .load(source_path)

6. .writeStream

7. .option("checkpointLocation", checkpointPath)

8. .___________

9. .start("target_table")

Which option correctly fills in the blank to meet the specified requirement ?

Options :

A : option("cloudFiles.schemaEvolutionMode", "addNewColumns")

B : option("cloudFiles.mergeSchema", True)

C : option("mergeSchema", True)

D : schema(schema_definition, mergeSchema=True)

E : Autoloader can not automatically evolve the schema of the table when new fields are detected

Answer: C

Question 3

Which of the following statements correctly describes End-to-End Testing ?

Options :

A : It’s an approach to simulate a user experience to ensure that the application can run properly under real-world scenarios

B : It’s an approach to test the interaction between subsystems of an application to ensure that modules work properly as a group.

C : It’s an approach to test individual units of code to determine whether they still work as expected if new changes are made to them in the future

D : It’s an approach to verify if each feature of the application works as per the business requirements

E : It’s an approach to measure the reliability, speed, scalability, and responsiveness of an application

Answer: A

Question 4

Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.

Which benefit offsets this additional effort?

Options :

A : Improves the quality of your data

B : Validates a complete use case of your application

C : Troubleshooting is easier since all steps are isolated and tested individually

D : Yields faster deployment and execution times

E : Ensures that all steps interact correctly to achieve the desired end result

Answer: C

Question 5

Which of the following statements best describes Delta Lake Auto Compaction?

Options :

A : Auto Compaction occurs after a write to a table has succeeded to check if files can further be compacted; if yes, it runs an OPTIMIZE job with Z-Ordering toward a file size of 128 MB

B : Auto Compaction occurs after a write to a table has succeeded to check if files can further be compacted; if yes, it runs an OPTIMIZE job without Z-Ordering toward a file size of 128 MB

C : Auto Compaction occurs after a write to a table has succeeded to check if files can further be compacted; if yes, it runs an OPTIMIZE job with Z-Ordering toward a file size of 1 GB.

D : Auto Compaction occurs after a write to a table has succeeded to check if files can further be compacted; if yes, it runs an OPTIMIZE job without Z-Ordering toward a file size of 1 GB.

E : None of the above statements correctly describe Delta Lake Auto Compaction.

Answer: B