Databricks Certified Associate Developer for Apache Spark free question 5 / 10 · Domain 3: Developing Apache Spark DataFrame/DataSet API Applications
A developer wraps a Python function that returns the length of a string: slen = F.udf(lambda s: len(s)) df2 = df.withColumn("name_length", slen("name")) No returnType is given. According to the PySpark API reference, what type does name_length have?
- No fixed type, since the column takes the type of each returned value.
- An integer type, since the return type is inferred from the Python result.
- A long type, since Python integers map to the widest integer type.
- A string type, since StringType is the default return type of udf.
Show the answer and explanation
Answer: D
Why this is correct
functions.udf(f=None, returnType=StringType(), ...) creates a user-defined function. The reference describes returnType as the return type of the user-defined function, given either as a DataType object or a DDL-formatted type string, and states that it defaults to StringType. Because the developer did not pass a returnType, name_length is a string column, even though the Python function returns numbers.
Why the other options are wrong
"An integer type, since the return type is inferred from the Python result." … The reference does not describe inferring the return type from the function. It gives a fixed default for returnType, so the Python int is not what decides the column type.
"A long type, since Python integers map to the widest integer type." … Mapping Python integers to a numeric type would need a returnType such as a DataType object or a DDL-formatted type string. Without one, the default applies.
"No fixed type, since the column takes the type of each returned value." … A DataFrame column has one type for all rows. The reference sets that type through returnType, with a default when it is left out.
Declaring the type
Pass the type when the result is not text, for example F.udf(lambda s: len(s), "int") or F.udf(lambda s: len(s), IntegerType()). Declaring it keeps sorting and arithmetic on the column numeric and avoids casting afterwards.