Databricks Certified Data Engineer Associate free question 7 / 10 · Domain 4: Working with Lakeflow Jobs
A job has a file arrival trigger watching a Unity Catalog volume, with Minimum time between triggers in seconds set to 600 and Wait after last change in seconds set to 90. On this day, files arrive at 10:00:00, 10:00:40 and 10:01:10, and then at 10:08:00 and 10:08:30. The first run of the day finishes in about a minute. Which are the correct start times of the first and second runs?
- The first at about 10:02:40, the second at about 10:13:40 (600 seconds after the first run finishes)
- The first at about 10:01:30, the second at about 10:12:40 (600 seconds after the first run starts)
- The first at about 10:02:40, the second at about 10:12:40 (600 seconds after the first run starts)
- The first at about 10:01:30, the second at about 10:13:40 (600 seconds after the first run finishes)
Show the answer and explanation
Answer: A
Why this is correct
Wait after last change in seconds starts the run only after the given number of seconds has passed since the last file arrived, and the wait is recounted whenever a file arrives. The last file of the first group is at 10:01:10, so the first run starts about 90 seconds later, at about 10:02:40, and finishes at about 10:03:40. Minimum time between triggers in seconds is a cooldown after a run: files that arrive after a run has finished do not start a new run until 600 seconds have passed. The files at about 10:08 arrive during this cooldown, so the second run starts at about 10:13:40.
Why the other options are wrong
"The first at about 10:01:30, the second at about 10:13:40 (600 seconds after the first run finishes)" … The documentation describes Wait after last change as "each new arrival resets the timer (debouncing)". Files at 10:00:40 and 10:01:10 reset the wait, so the run does not start at 10:01:30, 90 seconds after the first file. The time of the second run is right.
"The first at about 10:01:30, the second at about 10:12:40 (600 seconds after the first run starts)" … The first run does not start at 10:01:30, because the wait is reset by the files at 10:00:40 and 10:01:10 ("each new arrival resets the timer"). The second is also wrong: the cooldown is counted from when the run finishes, not from when it started, so it is not 10:12:40. Both settings are misapplied.
"The first at about 10:02:40, the second at about 10:12:40 (600 seconds after the first run starts)" … The documentation example says "After each run completes, the trigger waits 900 seconds (15 minutes) before it starts another run", counting the cooldown from run completion. The first run finishes at about 10:03:40, so the 600-second cooldown ends at about 10:13:40, not 10:12:40 counted from the start. The time of the first run is right.
Common trap
Setting both options overlaps two behaviors: waiting for a group of files to finish arriving, and spacing out runs. When the cooldown is long, processing of files that have arrived waits until the cooldown ends, so include the length of the cooldown in your estimate of the delay from arrival to processing.