Databricks Certified Data Engineer Associate free question 6 / 10 · Domain 4: Working with Lakeflow Jobs
In a Lakeflow Jobs job, three tasks are chained in series: ingest (no dependencies), notify_failure (depends only on ingest; Run if is At least one failed), and archive (depends only on notify_failure; Run if is the default, All succeeded). One day, ingest succeeds. Which statement about the results for notify_failure and archive is correct?
- notify_failure does not meet its condition and is skipped as Excluded, and archive, whose dependencies are all Excluded, is also skipped as Excluded
- notify_failure does not meet its condition and is skipped as Excluded, and because Excluded counts as success, archive runs
- notify_failure does not meet its condition and is skipped as Excluded, and archive becomes Upstream failed because its dependencies did not all succeed
- notify_failure does not meet its condition and becomes Upstream failed, and archive is also Upstream failed and does not run
Show the answer and explanation
Answer: A
Why this is correct
At least one failed is a condition that runs when any dependency failed, and when it is not met the task is skipped with the Excluded mark. ingest succeeded, so notify_failure is Excluded. archive depends only on notify_failure, which is Excluded. The documentation says that a task whose dependencies are all Excluded is Excluded regardless of its Run if condition, and gives an example in which, in a serial chain A to B to C, the exclusion of A propagates to C.
Why the other options are wrong
"notify_failure does not meet its condition and becomes Upstream failed, and archive is also Upstream failed and does not run" … When a condition that handles failure (At least one failed) is not met, the mark is Excluded, not Upstream failed. notify_failure becomes Excluded, and archive, whose dependencies are all Excluded, also becomes Excluded.
"notify_failure does not meet its condition and is skipped as Excluded, and archive becomes Upstream failed because its dependencies did not all succeed" … It is right that notify_failure becomes Excluded. But all of archive's dependencies are Excluded, so archive becomes Excluded regardless of Run if, before the check of whether All succeeded is met.
"notify_failure does not meet its condition and is skipped as Excluded, and because Excluded counts as success, archive runs" … The documentation does have a rule that, when evaluating Run if, an Excluded upstream is treated as a success. But the very next sentence of the same note says: "If all of a task's dependencies are excluded, the task itself is also excluded, regardless of its Run if condition." archive depends only on notify_failure, which is Excluded, so archive is Excluded and does not run.
Design point
If you chain a task that should always run behind a task that runs only on failure, such as a failure notification, it is skipped together with it on days when everything succeeds. If you want archive to run every time, add ingest to its dependencies and choose Run if again, or connect it directly to ingest in parallel with notify_failure. The outcome depends on how the DAG lines are drawn, so trace the mark of each task on both a success day and a failure day to confirm.