Databricks Certified Generative AI Engineer Associate Practice Questions

Databricks Certified Generative AI Engineer Associate exam preparation

Databricks Certified Generative AI Engineer Associate practice questions in English: 162 questions with explanations. The mock exam has 45 questions and 90 minutes.

Databricks Certified Generative AI Engineer Associate free question 10 / 10 · Domain 6: Evaluation and Monitoring

A team is deciding whether to use a small or a large LLM for answers. It serves both candidates on Model Serving chat endpoints and wants to run the same evaluation data through both with mlflow.genai.evaluate() and compare metrics. It builds each predict_fn with mlflow.genai.to_predict_fn("endpoints:/<endpoint-name>"), and each row of the evaluation data has the form {"inputs": {"question": "..."}}. When it runs, evaluation stops on both endpoints with an error that the input keys are not recognized. What is the fix?

  1. to_predict_fn passes the contents of inputs straight to the endpoint, so reshape the inputs of each row into a messages key holding role and content
  2. Move the evaluation data from a list of dictionaries into an evaluation dataset (mlflow.genai.datasets); the question key is then wrapped into the messages form automatically
  3. Keep the question key in inputs so that the judges can read the question, and add a messages key to the inputs of the same row so that both are passed
  4. to_predict_fn works only with endpoints built with Agent Framework, so write your own predict_fn that calls the REST API for chat endpoints
Show the answer and explanation

Answer: A

Why this is correct
The Docs note that to_predict_fn performs a kwargs pass-through directly to your endpoint. Your evaluation data must match the input format that your endpoint expects, and if the formats do not match, the evaluation fails with an error about unrecognized input keys. Model Serving chat endpoints require data formatted with the messages key, so if you reshape each row into {"inputs": {"messages": [{"role": "user", "content": "..."}]}}, the same data can evaluate both endpoints and the metrics can be lined up to choose a candidate.

Why the other options are wrong
"to_predict_fn works only with endpoints built with Agent Framework, so write your own predict_fn that calls the REST API for chat endpoints" … to_predict_fn works not only with endpoints built with Agent Framework (now Custom Agents) but also with Model Serving chat endpoints and custom endpoints. You do not need to write a function that calls REST yourself; it is enough to match the shape of the data.
"Move the evaluation data from a list of dictionaries into an evaluation dataset (mlflow.genai.datasets); the question key is then wrapped into the messages form automatically" … An evaluation dataset is a place that manages versions and lineage, and it does not rewrap the input to fit an endpoint. to_predict_fn passes inputs as they are, so the question key would be sent unchanged and evaluation would stop with the same error.
"Keep the question key in inputs so that the judges can read the question, and add a messages key to the inputs of the same row so that both are passed" … to_predict_fn passes the keys of inputs straight to the endpoint, so the leftover question key also reaches the endpoint and does not match the input format the chat endpoint expects. When the format does not match, evaluation stops with the unrecognized input keys error.

Key point
to_predict_fn also pulls traces out of endpoints that have tracing enabled, so you can follow a difference in metrics down to the step where it arises. When the input format differs between endpoints, the Docs also show a way to wrap the call in a function that converts the shape of the evaluation data to the shape of the endpoint, instead of fixing the data.

Go to the Databricks Certified Generative AI Engineer Associate practice app (10 free questions; purchase unlocks all questions)

See the full practice set