Databricks Certified Machine Learning Associate free question 4 / 10 · Domain 2: Data Processing
A notebook re-runs summary statistics on a Spark DataFrame with many rows using dbutils.data.summarize(df, precise=True). The result is more accurate than the default run, but some statistics remain approximate even with precise=True. Which ones?
- The number of distinct values and the frequent value counts of categorical columns
- The number of distinct values and the number of missing values of categorical columns
- The histograms and the percentile estimates of numeric columns
- The mean and the standard deviation of numeric columns
Show the answer and explanation
Answer: C
Why this is correct
In Databricks Runtime 10.4 LTS and above, dbutils.data.summarize takes a precise parameter. When precise is false (the default), some statistics are approximations to reduce run time. When precise is true, the statistics are computed with higher precision, and all statistics except the histograms and percentiles for numeric columns are exact. The histograms and percentile estimates stay approximate, with an error of up to 0.0001% relative to the total number of rows, down from up to 0.01% in the default mode.
Why the other options are wrong
"The number of distinct values and the frequent value counts of categorical columns" … Both are approximations only in the default mode: the distinct count may have about 5% relative error for high-cardinality columns, and frequent value counts may be off by up to 0.01% above 10,000 distinct values. With precise=True they become exact.
"The mean and the standard deviation of numeric columns" … The mean and the standard deviation fall under "all statistics except the histograms and percentiles for numeric columns", which are exact when precise is true.
"The number of distinct values and the number of missing values of categorical columns" … The distinct count of a categorical column becomes exact with precise=True, so it does not remain approximate. The only statistics left approximate are the histograms and percentile estimates of numeric columns.
Key point
The tooltip at the top of the data summary output shows the mode of the current run, so you can check on the output itself whether the numbers are approximate. When you rely on quartiles or percentiles as outlier boundaries, remember that they stay approximate even in precise mode.