Databricks Certified Associate Developer for Apache Spark free question 10 / 10 · Domain 7: Using Pandas API on Spark
A developer writes a Pandas UDF to center a column: @pandas_udf("double") def center(s: pd.Series) -> pd.Series: return s - s.mean() df.withColumn("centered", center("amount")) The results do not match the column mean computed separately. According to the pandas_udf API reference, why?
- Pandas UDFs skip null values, so the mean is taken over a different set of rows.
- The returnType "double" makes Spark round the mean before subtracting it.
- A Series to Series UDF may return fewer rows than it received, which shifts them.
- Each call gets one internal batch, so s.mean() is the mean of that batch only.
Show the answer and explanation
Answer: D
Why this is correct
For Series to Series Pandas UDFs, the reference says the function takes one or more pandas.Series and outputs one pandas.Series of the same length. It adds a note: the length of the input is not that of the whole input column, but the length of an internal batch used for each call to the function. Each call of center therefore subtracts the mean of its own batch, which differs from the mean of the whole column.
Why the other options are wrong
"The returnType "double" makes Spark round the mean before subtracting it." … returnType only declares the type of the result, given as a DataType object or a DDL-formatted type string. A double type does not round the mean.
"A Series to Series UDF may return fewer rows than it received, which shifts them." … The reference says the output of a Series to Series function should always be of the same length as the input. The rows are not shifted by a shorter output.
"Pandas UDFs skip null values, so the mean is taken over a different set of rows." … The difference does not come from nulls. The reference's note on input length explains it: each call sees only part of the column.
A call sees a batch, not the column
Compute statistics over the whole column outside the UDF first, then use them in the calculation. Keep Series to Series UDFs for logic that needs only the values passed in each call.