The expected behavior when a batch of data containing data that violates the expectation is

processed is that the job will fail. This is because the expectation clause has the ON VIOLATION FAIL

UPDATE option, which means that if any record in the batch does not meet the expectation, the

entire batch will be rejected and the job will fail. This option is useful for enforcing strict data quality

rules and preventing invalid data from entering the target dataset.

Option A is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that

violate the expectation, but fails the entire batch. To drop the records that violate the expectation

and record them as invalid in the event log, the ON VIOLATION DROP RECORD option should be used.

Option C is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that

violate the expectation, but fails the entire batch. To drop the records that violate the expectation

and load them into a quarantine table, the ON VIOLATION QUARANTINE RECORD option should be

used.

Option D is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that

violate the expectation, but fails the entire batch. To add the records that violate the expectation and

record them as invalid in the event log, the ON VIOLATION LOG RECORD option should be used.

Option E is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that

violate the expectation, but fails the entire batch. To add the records that violate the expectation and

flag them as invalid in a field added to the target dataset, the ON VIOLATION FLAG RECORD option

should be used.

Reference:

Delta Live Tables Expectations

[Databricks Data Engineer Professional Exam Guide]

Delta Lake tables use the Parquet format as their underlying storage format. Delta Lake enhances

Parquet by adding a transaction log that keeps track of all the operations performed on the table.

This allows features like ACID transactions, scalable metadata handling, and schema enforcement,

making it an ideal choice for big data processing and management in environments like Databricks.

Reference:

Databricks documentation on Delta Lake: Delta Lake Overview

Databricks cluster pools are a set of idle, ready-to-use instances that can reduce cluster start and

auto-scaling times. This is useful for scenarios where a data team needs to run an automated report

as quickly as possible, without waiting for the cluster to launch or scale up. Cluster pools can also

help save costs by reusing idle instances across different clusters and avoiding DBU charges for idle

instances in the pool. Reference: Best practices: pools | Databricks on AWS, Best practices: pools -

Azure Databricks | Microsoft Learn, Best practices: pools | Databricks on Google Cloud

:

The only option that is stored in the Databricks customer’s cloud account is data. Data is stored in the

customer’s cloud storage service, such as AWS S3 or Azure Data Lake Storage. The customer has full

control and ownership of their data and can access it directly from their cloud account.

Option A is not correct, as the Databricks web application is hosted and managed by Databricks on

their own cloud infrastructure. The customer does not need to install or maintain the web

application, but only needs to access it through a web browser.

Option B is not correct, as the cluster management metadata is stored and managed by Databricks

on their own cloud infrastructure. The cluster management metadata includes information such as

cluster configuration, status, logs, and metrics. The customer can view and manage their clusters

through the Databricks web application, but does not have direct access to the cluster management

metadata.

Option C is not correct, as the repos are stored and managed by Databricks on their own cloud

infrastructure. Repos are version-controlled repositories that store code and data files for Databricks

projects. The customer can create and manage their repos through the Databricks web application,

but does not have direct access to the repos.

Option E is not correct, as the notebooks are stored and managed by Databricks on their own cloud

infrastructure. Notebooks are interactive documents that contain code, text, and visualizations for

Databricks workflows. The customer can create and manage their notebooks through the Databricks

web application, but does not have direct access to the notebooks.

Reference:

Databricks Architecture

Databricks Data Sources

Databricks Repos

[Databricks Notebooks]

[Databricks Data Engineer Professional Exam Guide]

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as commit, pull, push, branch management, and visual comparison of diffs when

committing1. However, merge is not supported in the Git dialog2. You need to use the Repos UI or

your Git provider to merge branches3. Merge is a way to combine the commit history from one

branch into another branch1. During a merge, a merge conflict is encountered when Git cannot

automatically combine code from one branch into another. Merge conflicts require manual

resolution before a merge can be completed1. Reference: 4: Run Git operations on Databricks

Repos4, 1: CI/CD techniques with Git and Databricks Repos1, 3: Collaborate in Repos3, 2: Databricks

Repos - What it is and how we can use it2.

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as commit, pull, push, merge, and branch management. However, to clone a remote Git

repository to a Databricks repo, you must use the Databricks UI or API. You cannot clone a Git repo

using the CLI through a cluster’s web terminal, as the files won’t display in the Databricks

UI1. Reference: 1: Run Git operations on Databricks Repos | Databricks on AWS2

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as cloning, committing, pushing, pulling, and branch management. Databricks Notebooks

versioning is a legacy feature that allows users to link notebooks to GitHub repositories and perform

basic Git operations. However, Databricks Notebooks versioning does not support the use of multiple

branches for development work, which is an advantage of using Databricks Repos. With Databricks

Repos, users can create and manage branches for different features, experiments, or bug fixes, and

merge, rebase, or resolve conflicts between them. Databricks recommends using a separate branch

for each notebook and following data science and engineering code development best practices

using Git for version control, collaboration, and CI/CD. Reference: Git integration with Databricks

Repos - Azure Databricks | Microsoft Learn, Git version control for notebooks (legacy) | Databricks on

AWS, Databricks Repos Is Now Generally Available - New ‘Files’ Feature in …, Databricks Repos -

What it is and how we can use it | Adatis.

The MERGE command can be used to upsert data from a source table, view, or DataFrame into a

target Delta table. It allows you to specify conditions for matching and updating existing records, and

inserting new records when no match is found. This way, you can avoid writing duplicate records into

a Delta table1. The other commands (DROP, IGNORE, APPEND, INSERT) do not have this functionality

and may result in duplicate records or data loss234. Reference: 1: Upsert into a Delta Lake table using

merge | Databricks on AWS 2: SQL DELETE | Databricks on AWS 3: SQL INSERT INTO | Databricks on

AWS 4: SQL UPDATE | Databricks on AWS

:

In a medallion architecture, a common data design pattern for lakehouses, data flows from Bronze

to Silver to Gold layer tables, with each layer progressively improving the structure and quality of

data. Bronze tables store raw data ingested from various sources, while Silver tables apply minimal

transformations and cleansing to create an enterprise view of the data. Silver tables can also join and

enrich data from different Bronze tables to provide a more complete and consistent view of the data.

Therefore, option D is the correct answer, as Silver tables contain a more refined and cleaner view of

data than Bronze tables. Option A is incorrect, as it is the opposite of the correct answer. Option B is

incorrect, as Silver tables do not necessarily contain aggregates, but can also store detailed records.

Option C is incorrect, as Silver tables may contain less data than Bronze tables, depending on the

transformations and cleansing applied. Option E is incorrect, as Silver tables may contain more data

than Bronze tables, depending on the joins and enrichments applied. Reference: What is a Medallion

Architecture?, Transforming Bronze Tables in Silver Tables, What is the medallion lakehouse

architecture?