Databricks Certified Associate Developer for Apache Spark free question 4 / 10 · Domain 2: Using Spark SQL
A developer reads a folder of plain log files with: logs = spark.read.text("/logs/app/") No options are set. According to the Spark text data source guide, what does the logs DataFrame look like?
- One row per line, with columns split on spaces and types inferred.
- One row per line, with the line text in a string column named value.
- One row per line, with the line text in a string column named _c0.
- One row per file, with the full file text in a single string column.
Show the answer and explanation
Answer: B
Why this is correct
Spark SQL provides spark.read.text to read a file or a directory of text files into a DataFrame, and dataframe.write.text to write one. When reading a text file, each line becomes a row that has a string "value" column by default. The path can be a single text file or a directory of text files, so reading /logs/app/ produces one row per line across all the files, each with the line in value.
Why the other options are wrong
"One row per file, with the full file text in a single string column." … One row per file is what the wholetext option gives when it is set to true: it reads each file from the input path as a single row. Its default is false, and no options are set here.
"One row per line, with columns split on spaces and types inferred." … The text source does not split lines into columns or infer types; each line becomes one row. Splitting on a separator and inferring types are CSV reading features.
"One row per line, with the line text in a string column named _c0." … The guide names the column of the text source: each row has a string value column by default. The name _c0 is not what the text source uses.
Options that change the shape
lineSep changes the line separator used for reading or writing, and wholetext changes the unit from a line to a whole file. Further parsing, such as splitting value into fields, is done afterwards with DataFrame functions.