Databricks Certified Data Engineer Associate free question 5 / 10 · Domain 3: Data Transformation and Modeling
In Lakeflow Declarative Pipelines (formerly Delta Live Tables), the Gold table orders_gold for BI is defined as a streaming table and ingests daily. You rewrite the SELECT in the definition to add a currency conversion to the amount column, and run the pipeline with a normal update. Which statement about orders_gold right after this update is correct?
- The definition no longer matches the existing rows, so the update stops with an error and the table stays as it was before the change
- The earlier rows stay unconverted, and in addition the earlier rows of the source are appended again as converted rows
- Rows that came in with the update after the change are converted, and rows that came in before it remain unconverted
- Including the earlier rows, the data is re-read from the source with the new definition, and the rows in the table are all converted
Show the answer and explanation
Answer: C
Why this is correct
A streaming table handles each input row once. A row that has been appended is not read again in later updates. In the documentation example, when the definition is changed from LOWER to UPPER, the existing rows do not change and only new rows are uppercased. After a normal update, only the rows processed after the rewrite have the new conversion applied.
Why the other options are wrong
"Including the earlier rows, the data is re-read from the source with the new definition, and the rows in the table are all converted" … The documentation says: "A row that has already been appended to a streaming table will not be re-queried with later updates to the pipeline." In a normal update, earlier rows are not re-read from the source.
"The definition no longer matches the existing rows, so the update stops with an error and the table stays as it was before the change" … Describing the Limited evolution constraint, the documentation says: "You can change the query without recomputing the entire dataset." Changing the definition does not stop the update; rows that come in afterwards are processed with the new definition.
"The earlier rows stay unconverted, and in addition the earlier rows of the source are appended again as converted rows" … The documentation says: "Without a full refresh, a streaming table only sees each row once". The earlier rows of the source have already been processed, so a normal update does not read them again and append them.
Key point
In a streaming table, which version of the definition processed each row can differ from row to row. The documentation lists this as the Limited evolution constraint, and says that reprocessing rows that were processed before the change requires a full refresh. The Exam Guide uses the name Lakeflow Declarative Pipelines; the current docs also call it Lakeflow Spark Declarative Pipelines or Lakeflow pipelines.