Databricks Certified Data Engineer Associate free question 2 / 10 · Domain 2: Data Ingestion and Loading
You use Auto Loader to ingest a partner's JSON into a Unity Catalog table, and you need three things: (1) the stream must not stop when the partner adds a field, (2) the table's column layout must not change because of anything the ingestion does, and (3) the values of added fields must not be discarded but kept in the table so they can be checked later. Which read configuration meets all three?
- Pass the columns and types with .schema(), and set neither schemaEvolutionMode nor rescuedDataColumn
- Pass the columns and types with .schema(), do not set schemaEvolutionMode, and specify rescuedDataColumn
- Pass the columns and types with schemaHints, and set neither schemaEvolutionMode nor rescuedDataColumn
- Pass the columns and types with .schema(), set schemaEvolutionMode to addNewColumns, and specify rescuedDataColumn
Show the answer and explanation
Answer: B
Why this is correct
When you pass a schema with .schema(), the default schemaEvolutionMode is none. Mode none does not evolve the schema, and the stream does not stop because the schema changes. New columns are ignored, but if the rescuedDataColumn option is set, their values are placed in the rescued data column as JSON. The table's columns stay as the schema you passed, so all of (1) to (3) are met.
Why the other options are wrong
"Pass the columns and types with schemaHints, and set neither schemaEvolutionMode nor rescuedDataColumn" … schemaHints adds type instructions to an inferred schema; it does not provide a schema. Because no schema is provided, schemaEvolutionMode defaults to addNewColumns, so on finding a new column the stream adds it to the schema and then stops with an UnknownFieldException. That fails (1) and (2).
"Pass the columns and types with .schema(), and set neither schemaEvolutionMode nor rescuedDataColumn" … Providing the schema makes the default none, which satisfies (1) and (2). But the documentation table says that for none, data is not rescued unless the rescuedDataColumn option is set. With no rescuedDataColumn, the values are dropped, which fails (3).
"Pass the columns and types with .schema(), set schemaEvolutionMode to addNewColumns, and specify rescuedDataColumn" … The documentation says addNewColumns is not allowed when the schema of the stream is provided, but does work if you provide your schema as a schema hint. A stream whose schema is passed with .schema() cannot be given addNewColumns.
Related setting
The rescued data column also collects values that do not match the schema's types and values from columns whose names differ only in case, not just columns missing from the schema. A rescued value is kept as JSON together with the path of the source file.