Question 1

If we want to create a constant integer 1 as a new column ‘new_column’ in a dataframe df, which code block we should select ?

Options :

A : df.withColumn(new_column, lit(1))

B : df.withColumn(”new_column”, lit(“1”))

C : df.withColumnRenamed('new_column', lit(1))

D : df.withColumn(“new_column”, 1)

E : df.withColumn(“new_column”, lit(1))

Answer: A

Question 2

Given the following statements regarding caching:

1. 1: The default storage level for a DataFrame is StorageLevel.MEMORY

2.

3. 2: The DataFrame class does have an unpersist() operation

4.

5. 3: The persist() method needs an action to load data from its source to materialize the DataFrame in cache

Which one is NOT TRUE ?

Options :

A : 2

B : 1,2,3

C : 1,3

D : 1

E : 1,2

Answer: D

Question 3

Choose the correct code block to unpersist a table named ‘table’.

Options :

A : spark.catalog.uncacheTable(table)

B : spark.uncacheTable(table)

C : spark.catalog.uncacheTable(“table”)

D : spark.uncacheTable(“table”)

Answer: C

Question 4

The code block displayed below contains an error. The code block should return DataFrame transactionsDf,

but with the column storeId renamed to storeNumber. Find the error.

Code block:

transactionsDf.withColumn("storeNumber", "storeId")

Options :

A : Instead of withColumn, the withColumnRenamed method should be used.

B : Arguments "storeNumber" and "storeId" each need to be wrapped in a col() operator.

C : Argument "storeId" should be the first and argument "storeNumber" should be the second argument to the withColumn method

D : The withColumn operator should be replaced with the copyDataFrame operator.

E : Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.

Answer: A

Question 5

The code block shown below should return the number of columns in the CSV file stored at location filePath. From the CSV file, only lines should be read that do not start with a # character. Choose the answer that correctly fills the blanks in the code block to accomplish this

Code block:

__1__(__2__.__3__.csv(filePath, __4__).__5__)

Options :

A :

1. size

2. spark

3. read()

4. escape='#'

5. columns

B :

1. DataFrame

2. spark

3. read()

4. escape='#'

5. shape[0]

C :

1. len

2. pyspark

3. DataFrameReader

4. comment='#'

5. columns

D :

1. size

2. pyspark

3. DataFrameReader

4. comment='#'

5. columns

E : 1. len

2. spark

3. read

4. comment='#'

5. columns

Answer: A