Databricks Certified Associate Developer for Apache Spark free question 9 / 10 · Domain 6: Using Spark Connect to Deploy Applications
A developer runs DataFrame code in a notebook connected to a remote cluster through Spark Connect. According to the Spark Connect overview, what does the client send to the server, and how do results come back?
- The script's source code, which the server runs as a whole; results return as files written to shared storage.
- Physical plans optimized on the client; the server runs them and sends results back as plain text over HTTP.
- The Python objects of the session through Py4J; results return as Java objects that the client then converts.
- Unresolved logical plans encoded with protocol buffers over gRPC; results return as Arrow-encoded row batches.
Show the answer and explanation
Answer: D
Why this is correct
The overview describes the Spark Connect client as a thin API that builds on the DataFrame API, using unresolved logical plans as a language-agnostic protocol between the client and the Spark driver. The client translates DataFrame operations into unresolved logical query plans encoded with protocol buffers and sends them to the server with the gRPC framework. The server endpoint translates them into Spark's logical plan operators, the standard execution process runs, and results are streamed back through gRPC as Apache Arrow-encoded row batches.
Why the other options are wrong
"The Python objects of the session through Py4J; results return as Java objects that the client then converts." … The overview says the client does not run in the same process as the Spark driver, so it cannot reach the driver JVM directly. It sends plans to the server, and the server returns the results.
"The script's source code, which the server runs as a whole; results return as files written to shared storage." … The client translates each DataFrame operation into a plan rather than shipping the program. The overview describes results being streamed back to the client through gRPC, not written to storage.
"Physical plans optimized on the client; the server runs them and sends results back as plain text over HTTP." … Optimization happens on the server. The endpoint translates the unresolved plans into Spark's logical plan operators, and then the standard Spark execution process, with all of Spark's optimizations, kicks in.
Plans out, Arrow batches back
Because only plans travel, the client can be embedded in application servers, IDEs and notebooks, while Spark itself runs on the server with its usual optimizations.