Databricks Certified Machine Learning Associate free question 10 / 10 · Domain 4: Model Deployment
A team has a trained model logged with MLflow. They want to score a whole table every night and also score events that keep arriving through Structured Streaming. Which way of using the MLflow model matches the Databricks documentation?
- Load the model once with mlflow.pyfunc.load_model, which returns a Spark UDF that can be applied to both batch and streaming DataFrames
- Use mlflow.pyfunc.spark_udf for the batch job and serve the model from an endpoint for streaming, because the Spark UDF cannot be applied to a streaming job
- Use mlflow.pyfunc.spark_udf for the batch job and rewrite the model logic as a hand-written pandas UDF for the streaming job
- Turn the model into a Spark UDF with mlflow.pyfunc.spark_udf and apply that UDF in both the batch job and the streaming job
Show the answer and explanation
Answer: D
Why this is correct
The Log, load, and register MLflow models page says a logged model can be exported as an Apache Spark UDF for scoring on a Spark cluster, either as a batch job or as a real-time Spark Streaming job. Its example reads a table into a DataFrame and adds a prediction column with mlflow.pyfunc.spark_udf(spark, model_path) and withColumn. The same UDF can be applied to the DataFrame of a streaming read, so one logged model serves both workloads.
Why the other options are wrong
"Use mlflow.pyfunc.spark_udf for the batch job and serve the model from an endpoint for streaming, because the Spark UDF cannot be applied to a streaming job" … The documentation says the Spark UDF can be used as a batch job or as a real-time Spark Streaming job, so the streaming job does not have to move to an endpoint. A REST call per row would also add a network round trip to every row.
"Load the model once with mlflow.pyfunc.load_model, which returns a Spark UDF that can be applied to both batch and streaming DataFrames" … mlflow.pyfunc.load_model returns a generic Python function whose predict method takes pandas-style input. The page presents the Spark UDF from spark_udf as the separate alternative for scoring on a Spark cluster.
"Use mlflow.pyfunc.spark_udf for the batch job and rewrite the model logic as a hand-written pandas UDF for the streaming job" … The page does not restrict spark_udf to batch DataFrames; it names Spark Streaming jobs as a use of the same UDF. Rewriting the logic as a hand-written pandas UDF is unnecessary work and loses the guarantee that the model behaves as logged.
Key point
The MLflow model format is a common standard that lets the same model be used for batch inference on Apache Spark and for real-time serving through a REST API. Choosing a different inference style does not require rebuilding the model, which is the benefit of the format.