[May 30, 2023] Associate-Developer-Apache-Spark Test Prep Training Practice Exam Questions Practice Tests [Q94-Q110]

NEW QUESTION 94
Which of the following code blocks removes all rows in the 6-column DataFrame transactionsDf that have missing data in at least 3 columns?

transactionsDf.dropna(“any”)

transactionsDf.dropna(thresh=4)

transactionsDf.drop.na(“”,2)

transactionsDf.dropna(thresh=2)

transactionsDf.dropna(“”,4)

NEW QUESTION 95
The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)

1. filter
2. “transactionId”, “predError”, “value”, “f”

1. select
2. “transactionId, predError, value, f”

1. select
2. [“transactionId”, “predError”, “value”, “f”]

1. where
2. col(“transactionId”), col(“predError”), col(“value”), col(“f”)

1. select
2. col([“transactionId”, “predError”, “value”, “f”])

NEW QUESTION 96
Which of the following code blocks prints out in how many rows the expression Inc. appears in the string-type column supplier of DataFrame itemsDf?

1.counter = 0
2.
3.for index, row in itemsDf.iterrows():
4. if ‘Inc.’ in row[‘supplier’]:
5. counter = counter + 1
6.
7.print(counter)

1.counter = 0
2.
3.def count(x):
4. if ‘Inc.’ in x[‘supplier’]:
5. counter = counter + 1
6.
7.itemsDf.foreach(count)
8.print(counter)

print(itemsDf.foreach(lambda x: ‘Inc.’ in x))

print(itemsDf.foreach(lambda x: ‘Inc.’ in x).sum())

1.accum=sc.accumulator(0)
2.
3.def check_if_inc_in_supplier(row):
4. if ‘Inc.’ in row[‘supplier’]:
5. accum.add(1)
6.
7.itemsDf.foreach(check_if_inc_in_supplier)
8.print(accum.value)

NEW QUESTION 97
The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, “itemId”, how=”broadcast”)

The syntax is wrong, how= should be removed from the code block.

The join method should be replaced by the broadcast method.

Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.

The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.

broadcast is not a valid join type.

NEW QUESTION 98
The code block shown below should convert up to 5 rows in DataFrame transactionsDf that have the value 25 in column storeId into a Python list. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__).__3__(__4__)

1. filter
2. “storeId”==25
3. collect
4. 5

1. filter
2. col(“storeId”)==25
3. toLocalIterator
4. 5

1. select
2. storeId==25
3. head
4. 5

1. filter
2. col(“storeId”)==25
3. take
4. 5

1. filter
2. col(“storeId”)==25
3. collect
4. 5

NEW QUESTION 99
Which of the following statements about Spark’s configuration properties is incorrect?

The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.

The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.

The default value for spark.sql.autoBroadcastJoinThreshold is 10MB.

The default number of partitions to use when shuffling data for joins or aggregations is 300.

The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.

NEW QUESTION 100
The code block displayed below contains an error. The code block is intended to write DataFrame transactionsDf to disk as a parquet file in location /FileStore/transactions_split, using column storeId as key for partitioning. Find the error.
Code block:
transactionsDf.write.format(“parquet”).partitionOn(“storeId”).save(“/FileStore/transactions_split”)A.

The format(“parquet”) expression is inappropriate to use here, “parquet” should be passed as first argument to the save() operator and “/FileStore/transactions_split” as the second argument.

Partitioning data by storeId is possible with the partitionBy expression, so partitionOn should be replaced by partitionBy.

Partitioning data by storeId is possible with the bucketBy expression, so partitionOn should be replaced by bucketBy.

partitionOn(“storeId”) should be called before the write operation.

The format(“parquet”) expression should be removed and instead, the information should be added to the write expression like so: write(“parquet”).

NEW QUESTION 101
In which order should the code blocks shown below be run in order to create a table of all values in column attributes next to the respective values in column supplier in DataFrame itemsDf?
1. itemsDf.createOrReplaceView(“itemsDf”)
2. spark.sql(“FROM itemsDf SELECT ‘supplier’, explode(‘Attributes’)”)
3. spark.sql(“FROM itemsDf SELECT supplier, explode(attributes)”)
4. itemsDf.createOrReplaceTempView(“itemsDf”)

4, 3

1, 3

2

4, 2

1, 2

NEW QUESTION 102
Which of the following code blocks immediately removes the previously cached DataFrame transactionsDf from memory and disk?

array_remove(transactionsDf, “*”)

transactionsDf.unpersist()
(Correct)

del transactionsDf

transactionsDf.clearCache()

transactionsDf.persist()

NEW QUESTION 103
Which of the following code blocks returns all unique values of column storeId in DataFrame transactionsDf?

transactionsDf[“storeId”].distinct()

transactionsDf.select(“storeId”).distinct()
(Correct)

transactionsDf.filter(“storeId”).distinct()

transactionsDf.select(col(“storeId”).distinct())

transactionsDf.distinct(“storeId”)

NEW QUESTION 104
Which of the following describes a difference between Spark’s cluster and client execution modes?

In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client mode.

In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.

In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.

In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.

In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.

NEW QUESTION 105
Which of the following code blocks performs a join in which the small DataFrame transactionsDf is sent to all executors where it is joined with DataFrame itemsDf on columns storeId and itemId, respectively?

itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, “right_outer”)

itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, “broadcast”)

itemsDf.merge(transactionsDf, “itemsDf.itemId == transactionsDf.storeId”, “broadcast”)

itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)

itemsDf.join(transactionsDf, broadcast(itemsDf.itemId == transactionsDf.storeId))

NEW QUESTION 106
Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the
6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?

union(todayTransactionsDf, yesterdayTransactionsDf)

todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)

todayTransactionsDf.unionByName(yesterdayTransactionsDf)

todayTransactionsDf.concat(yesterdayTransactionsDf)

todayTransactionsDf.union(yesterdayTransactionsDf)

NEW QUESTION 107
The code block shown below should return a DataFrame with only columns from DataFrame transactionsDf for which there is a corresponding transactionId in DataFrame itemsDf. DataFrame itemsDf is very small and much smaller than DataFrame transactionsDf. The query should be executed in an optimized way. Choose the answer that correctly fills the blanks in the code block to accomplish this.
__1__.__2__(__3__, __4__, __5__)

1. transactionsDf
2. join
3. broadcast(itemsDf)
4. transactionsDf.transactionId==itemsDf.transactionId
5. “outer”

1. transactionsDf
2. join
3. itemsDf
4. transactionsDf.transactionId==itemsDf.transactionId
5. “anti”

1. transactionsDf
2. join
3. broadcast(itemsDf)
4. “transactionId”
5. “left_semi”

1. itemsDf
2. broadcast
3. transactionsDf
4. “transactionId”
5. “left_semi”

1. itemsDf
2. join
3. broadcast(transactionsDf)
4. “transactionId”
5. “left_semi”

NEW QUESTION 108
Which of the following code blocks creates a new DataFrame with two columns season and wind_speed_ms where column season is of data type string and column wind_speed_ms is of data type double?

spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

1. from pyspark.sql import types as T
2. spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,

CharType()), T.StructField(“season”, T.DoubleType())]))

spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

Explanation
spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) Correct. This command uses the Spark Session’s createDataFrame method to create a new DataFrame. Notice how rows, columns, and column names are passed in here: The rows are specified as a Python list. Every entry in the list is a new row. Columns are specified as Python tuples (for example (“summer”, 4.5)). Every column is one entry in the tuple.
The column names are specified as the second argument to createDataFrame(). The documentation (link below) shows that “when schema is a list of column names, the type of each column will be inferred from data” (the first argument). Since values 4.5 and 7.5 are both float variables, Spark will correctly infer the double type for column wind_speed_ms. Given that all values in column
“season” contain only strings, Spark will cast the column appropriately as string.
Find out more about SparkSession.createDataFrame() via the link below.
spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) No, the SparkSession does not have a newDataFrame method.
from pyspark.sql import types as T
spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,
T.CharType()), T.StructField(“season”, T.DoubleType())]))
No. pyspark.sql.types does not have a CharType type. See link below for available data types in Spark.
spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, this is not correct Spark syntax. If you have considered this option to be correct, you may have some experience with Python’s pandas package, in which this would be correct syntax. To create a Spark DataFrame from a Pandas DataFrame, you can simply use spark.createDataFrame(pandasDf) where pandasDf is the Pandas DataFrame.
Find out more about Spark syntax options using the examples in the documentation for SparkSession.createDataFrame linked below.
spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, the Spark Session (indicated by spark in the code above) does not have a DataFrame method.
More info: pyspark.sql.SparkSession.createDataFrame – PySpark 3.1.1 documentation and Data Types – Spark 3.1.2 Documentation Static notebook | Dynamic notebook: See test 1

NEW QUESTION 109
Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?

tranactionsDf.select(‘value’).join(transactionsDf.select(‘productId’), col(‘value’)==col(‘productId’),
‘outer’)

transactionsDf.select(col(‘value’), col(‘productId’)).agg({‘*’: ‘count’})

transactionsDf.select(‘value’, ‘productId’).distinct()

transactionsDf.select(‘value’).union(transactionsDf.select(‘productId’)).distinct()

transactionsDf.agg({‘value’: ‘collect_set’, ‘productId’: ‘collect_set’})

NEW QUESTION 110
Which of the following code blocks shows the structure of a DataFrame in a tree-like way, containing both column names and types?

Free Cram & Latest Exams Dumps

[May 30, 2023] Associate-Developer-Apache-Spark Test Prep Training Practice Exam Questions Practice Tests [Q94-Q110]

leave a comment Cancel reply

Related Certifications

Recent Posts

Archives

Categories