Databricks Certified Data Engineer Professional free question 4 / 10 · Domain 4: Monitoring and Alerting
For a pipeline that runs on serverless compute, you queried the event log with event_type = 'autoscale', but no rows come back. The same query returns the requested number of executors for another team's pipeline. Which explanation is correct?
- autoscale events are emitted when a resize fails, so a pipeline whose resizes keep succeeding does not produce them
- autoscale events are not written by default; they are recorded when the event log is published in the pipeline settings
- On serverless pipelines the same information appears as optimal_num_executors in event_type = 'cluster_resources' events, so look there
- autoscale events are recorded for classic-compute pipelines with enhanced autoscaling enabled, and do not appear for serverless pipelines
Show the answer and explanation
Answer: D
Why this is correct
The event log captures cluster resizes for pipelines that use classic compute (in other words, that do not use serverless compute) when enhanced autoscaling is enabled. The details:autoscale object of events with event type autoscale holds the status (RESIZING, SUCCEEDED, PARTIALLY_SUCCEEDED, or FAILED) and the requested number of executors. A serverless pipeline does not produce this kind of event, so the query returns nothing.
Why the other options are wrong
"autoscale events are not written by default; they are recorded when the event log is published in the pipeline settings" … Publishing the event log only exposes the hidden table under a name; it does not change which events are recorded. For a classic pipeline with enhanced autoscaling, autoscale events are recorded even if the log is not published.
"On serverless pipelines the same information appears as optimal_num_executors in event_type = 'cluster_resources' events, so look there" … The documentation also describes cluster_resources events as metrics for classic compute resources. Autoscaling algorithm values such as optimal_num_executors are present when enhanced autoscaling is enabled.
"autoscale events are emitted when a resize fails, so a pipeline whose resizes keep succeeding does not produce them" … autoscale events are not limited to failures; they are recorded for resize requests (RESIZING), successes (SUCCEEDED), and partial successes too, which is why the other team's query returns requested numbers.
Key point
For classic pipelines, cluster_resources events show the number of task slots, how much they are used, and how many tasks are waiting to be scheduled. With enhanced autoscaling enabled they also carry algorithm states such as CLUSTER_AT_DESIRED_SIZE, so reading them together with autoscale events gives the whole picture of resizing.