Databricks Certified Associate Developer for Apache Spark free question 8 / 10 · Domain 5: Structured Streaming
A developer writes a streaming aggregation without a watermark: counts = events.groupBy("device").count() query = counts.writeStream.outputMode("append").format("console").start() According to the output mode compatibility matrix in the Structured Streaming programming guide, which statement is correct?
- Append works once a fixed trigger interval is set on the query as well.
- Append works, since each count is written once, when its device first appears.
- Complete or Update are supported; Append is not, since aggregates can update.
- Only Complete works here, since Update would need a watermark on the query.
Show the answer and explanation
Answer: C
Why this is correct
The compatibility matrix lists, for queries with aggregation other than aggregation on event time with a watermark, the supported output modes Complete and Update. Since no watermark is defined, old aggregation state is not dropped. The guide adds that Append mode is not supported because aggregates can update, which violates the semantics of Append, where each row is output only once.
Why the other options are wrong
"Append works, since each count is written once, when its device first appears." … Append mode outputs a row only once and suits queries whose rows never change. A count per device changes as new events arrive, so the guide does not support Append for this kind of aggregation.
"Append works once a fixed trigger interval is set on the query as well." … The guide ties output modes to the type of query. Its reason for excluding Append, that aggregates can update, holds whatever the trigger is.
"Only Complete works here, since Update would need a watermark on the query." … The matrix lists Complete and Update for aggregations without a watermark. Update does not need a watermark; without one, old aggregation state is simply not dropped.
Mode follows the query type
For an aggregation, use Complete to write the whole Result Table after every trigger, or Update to write only the rows that changed.