Free Databricks Certified Data Engineer Associate Practice Questions – 2026 Updated

https://spark.apache.org/docs/latest/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.D

ataStreamWriter.trigger.html

The processingTime option specifies a time-based trigger interval for fixed interval micro-batches.

This means that the query will execute a micro-batch to process data every 5 seconds, regardless of

how much data is available. This option is suitable for near-real time processing workloads that

require low latency and consistent processing frequency. The other options are either invalid syntax

(A, C), default behavior (B), or experimental feature (E). Reference: Databricks Documentation -

Configure Structured Streaming trigger intervals, Databricks Documentation - Trigger.

The DELETE command in Delta Lake allows you to remove data that matches a predicate from a

Delta table. This command will delete all the rows where the value in the column age is greater than

25 from the existing Delta table my_table and save the updated table. The other options are either

incorrect or do not achieve the desired result. Option A will only select the rows that match the

predicate, but not delete them. Option B will update the rows that match the predicate, but not

delete them. Option D will update the rows that do not match the predicate, but not delete them.

Option E will delete the rows that do not match the predicate, which is the opposite of what we

want. Reference: Table deletes, updates, and merges — Delta Lake Documentation

Option A is the correct answer because it uses the FILTER higher-order function correctly to filter out

employees with more than 5 years of experience from the array column “employees”. It applies a

lambda function i -> i.years_exp > 5 that checks if the years of experience of each employee in the

array is greater than 5. If this condition is met, the employee is included in the new array column

“exp_employees”.

Reference: The use of higher-order functions like FILTER can be referenced from Databricks

documentation on Higher-Order Functions.

Structured Streaming uses checkpointing and write-ahead logs to record the offset range of the data

being processed in each trigger. This ensures that the engine can reliably track the exact progress of

the processing and handle any kind of failure by restarting and/or reprocessing. Checkpointing is the

mechanism of saving the state of a streaming query to fault-tolerant storage (such as HDFS) so that it

can be recovered after a failure. Write-ahead logs are files that record the offset range of the data

being processed in each trigger and are written to the checkpoint location before the processing

starts. These logs are used to recover the query state and resume processing from the last processed

offset range in case of a failure. Reference: Structured Streaming Programming Guide, Fault

Tolerance Semantics

This approach would allow the data engineering team to use the existing SQL program and add some

logic to control the execution of the final query based on the day of the week. They could use

the datetime module in Python to get the current date and check if it is a Sunday. If so, they could

run the final query, otherwise they could skip it. This way, they could schedule the program to run

every day without changing the data model or the source table. Reference: PySpark SQL

Module, Python datetime Module, Databricks Jobs

A data lakehouse is a data management architecture that combines the flexibility, cost-efficiency,

and scale of data lakes with the data management and ACID transactions of data warehouses,

enabling business intelligence (BI) and machine learning (ML) on all data12. By using a data

lakehouse, both the data analysis and data engineering teams can access the same data sources and

formats, ensuring data consistency and quality across their reports. A data lakehouse also supports

schema enforcement and evolution, data validation, and time travel to old table versions, which can

help resolve data conflicts and errors1. Reference: 1: What is a Data Lakehouse? - Databricks 2: What

is a data lakehouse? | IBM

Delta Live Tables is a declarative framework for building reliable, maintainable, and testable data

processing pipelines. You define the transformations to perform on your data and Delta Live Tables

manages task orchestration, cluster management, monitoring, data quality, and error handling. Delta

Live Tables supports both SQL and Python as the languages for defining your datasets and

expectations. Delta Live Tables also supports both streaming and batch sources, and can handle both

append-only and upsert data patterns. Delta Live Tables follows the medallion lakehouse

architecture, which consists of three layers of data: bronze, silver, and gold. Therefore, migrating to

Delta Live Tables does not require any of the changes listed in the options B, C, D, or E. The data

engineer and data analyst can use the same languages, sources, and architecture as before, and

simply declare their datasets and expectations using Delta Live Tables syntax. Reference:

What is Delta Live Tables?

Transform data with Delta Live Tables

What is the medallion lakehouse architecture?

To grant full permissions on a table to a user or a group, you can use the GRANT ALL PRIVILEGES ON

TABLE statement. This statement will grant all the possible privileges on the table, such

as SELECT, CREATE, MODIFY, DROP, ALTER, etc. Option A is the only code block that follows this syntax

correctly. Option B is incorrect, as it does not grant all the possible privileges on the table, but only a

subset of them. Option C is incorrect, as it only grants the SELECT privilege on the table, which is not

enough to fully manage the project. Option D is incorrect, as it grants the USAGE privilege on the

table, which is not a valid privilege for tables. Option E is incorrect, as it grants all the privileges on

the table team to the user or group sales, which is the opposite of what the question

asks. Reference: Grant privileges on a table using SQL | Databricks on AWS, Grant privileges on a

table using SQL - Azure Databricks, SQL Privileges - Databricks

The query performing a streaming hop from raw data to a Bronze table is identified by using the

Spark streaming read capability and then writing to a Bronze table. Let's analyze the options:

Option A: Utilizes .writeStream but performs a complete aggregation which is more characteristic of

a roll-up into a summarized table rather than a hop into a Bronze table.

Option B: Also uses .writeStream but calculates an average, which again does not typically represent

the raw to Bronze transformation, which usually involves minimal transformations.

Option C: This uses a basic .write with .mode("append") which is not a streaming operation, and

hence not suitable for real-time streaming data transformation to a Bronze table.

Option D: It employs spark.readStream.load() to ingest raw data as a stream and then writes it out

with .writeStream, which is a typical pattern for streaming data into a Bronze table where raw data is

captured in real-time and minimal transformation is applied. This approach aligns with the concept

of a Bronze table in a modern data architecture, where raw data is ingested continuously and stored

in a more accessible format.

Reference:

Databricks documentation on Structured Streaming: Structured Streaming in Databricks

The expected behavior when a batch of data containing data that violates the expectation is

processed is that the job will fail. This is because the expectation clause has the ON VIOLATION FAIL

UPDATE option, which means that if any record in the batch does not meet the expectation, the

entire batch will be rejected and the job will fail. This option is useful for enforcing strict data quality

rules and preventing invalid data from entering the target dataset.

Option A is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that

violate the expectation, but fails the entire batch. To drop the records that violate the expectation

and record them as invalid in the event log, the ON VIOLATION DROP RECORD option should be used.

Option C is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that

violate the expectation, but fails the entire batch. To drop the records that violate the expectation

and load them into a quarantine table, the ON VIOLATION QUARANTINE RECORD option should be

used.

Option D is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that

violate the expectation, but fails the entire batch. To add the records that violate the expectation and

record them as invalid in the event log, the ON VIOLATION LOG RECORD option should be used.

Option E is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that

violate the expectation, but fails the entire batch. To add the records that violate the expectation and

flag them as invalid in a field added to the target dataset, the ON VIOLATION FLAG RECORD option

should be used.

Reference:

Delta Live Tables Expectations

[Databricks Data Engineer Professional Exam Guide]

Delta Lake tables use the Parquet format as their underlying storage format. Delta Lake enhances

Parquet by adding a transaction log that keeps track of all the operations performed on the table.

This allows features like ACID transactions, scalable metadata handling, and schema enforcement,

making it an ideal choice for big data processing and management in environments like Databricks.

Reference:

Databricks documentation on Delta Lake: Delta Lake Overview

Databricks cluster pools are a set of idle, ready-to-use instances that can reduce cluster start and

auto-scaling times. This is useful for scenarios where a data team needs to run an automated report

as quickly as possible, without waiting for the cluster to launch or scale up. Cluster pools can also

help save costs by reusing idle instances across different clusters and avoiding DBU charges for idle

instances in the pool. Reference: Best practices: pools | Databricks on AWS, Best practices: pools -

Azure Databricks | Microsoft Learn, Best practices: pools | Databricks on Google Cloud

:

The only option that is stored in the Databricks customer’s cloud account is data. Data is stored in the

customer’s cloud storage service, such as AWS S3 or Azure Data Lake Storage. The customer has full

control and ownership of their data and can access it directly from their cloud account.

Option A is not correct, as the Databricks web application is hosted and managed by Databricks on

their own cloud infrastructure. The customer does not need to install or maintain the web

application, but only needs to access it through a web browser.

Option B is not correct, as the cluster management metadata is stored and managed by Databricks

on their own cloud infrastructure. The cluster management metadata includes information such as

cluster configuration, status, logs, and metrics. The customer can view and manage their clusters

through the Databricks web application, but does not have direct access to the cluster management

metadata.

Option C is not correct, as the repos are stored and managed by Databricks on their own cloud

infrastructure. Repos are version-controlled repositories that store code and data files for Databricks

projects. The customer can create and manage their repos through the Databricks web application,

but does not have direct access to the repos.

Option E is not correct, as the notebooks are stored and managed by Databricks on their own cloud

infrastructure. Notebooks are interactive documents that contain code, text, and visualizations for

Databricks workflows. The customer can create and manage their notebooks through the Databricks

web application, but does not have direct access to the notebooks.

Reference:

Databricks Architecture

Databricks Data Sources

Databricks Repos

[Databricks Notebooks]

[Databricks Data Engineer Professional Exam Guide]

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as commit, pull, push, branch management, and visual comparison of diffs when

committing1. However, merge is not supported in the Git dialog2. You need to use the Repos UI or

your Git provider to merge branches3. Merge is a way to combine the commit history from one

branch into another branch1. During a merge, a merge conflict is encountered when Git cannot

automatically combine code from one branch into another. Merge conflicts require manual

resolution before a merge can be completed1. Reference: 4: Run Git operations on Databricks

Repos4, 1: CI/CD techniques with Git and Databricks Repos1, 3: Collaborate in Repos3, 2: Databricks

Repos - What it is and how we can use it2.

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as commit, pull, push, merge, and branch management. However, to clone a remote Git

repository to a Databricks repo, you must use the Databricks UI or API. You cannot clone a Git repo

using the CLI through a cluster’s web terminal, as the files won’t display in the Databricks

UI1. Reference: 1: Run Git operations on Databricks Repos | Databricks on AWS2

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations

such as cloning, committing, pushing, pulling, and branch management. Databricks Notebooks

versioning is a legacy feature that allows users to link notebooks to GitHub repositories and perform

basic Git operations. However, Databricks Notebooks versioning does not support the use of multiple

branches for development work, which is an advantage of using Databricks Repos. With Databricks

Repos, users can create and manage branches for different features, experiments, or bug fixes, and

merge, rebase, or resolve conflicts between them. Databricks recommends using a separate branch

for each notebook and following data science and engineering code development best practices

using Git for version control, collaboration, and CI/CD. Reference: Git integration with Databricks

Repos - Azure Databricks | Microsoft Learn, Git version control for notebooks (legacy) | Databricks on

AWS, Databricks Repos Is Now Generally Available - New ‘Files’ Feature in …, Databricks Repos -

What it is and how we can use it | Adatis.

The MERGE command can be used to upsert data from a source table, view, or DataFrame into a

target Delta table. It allows you to specify conditions for matching and updating existing records, and

inserting new records when no match is found. This way, you can avoid writing duplicate records into

a Delta table1. The other commands (DROP, IGNORE, APPEND, INSERT) do not have this functionality

and may result in duplicate records or data loss234. Reference: 1: Upsert into a Delta Lake table using

merge | Databricks on AWS 2: SQL DELETE | Databricks on AWS 3: SQL INSERT INTO | Databricks on

AWS 4: SQL UPDATE | Databricks on AWS

:

In a medallion architecture, a common data design pattern for lakehouses, data flows from Bronze

to Silver to Gold layer tables, with each layer progressively improving the structure and quality of

data. Bronze tables store raw data ingested from various sources, while Silver tables apply minimal

transformations and cleansing to create an enterprise view of the data. Silver tables can also join and

enrich data from different Bronze tables to provide a more complete and consistent view of the data.

Therefore, option D is the correct answer, as Silver tables contain a more refined and cleaner view of

data than Bronze tables. Option A is incorrect, as it is the opposite of the correct answer. Option B is

incorrect, as Silver tables do not necessarily contain aggregates, but can also store detailed records.

Option C is incorrect, as Silver tables may contain less data than Bronze tables, depending on the

transformations and cleansing applied. Option E is incorrect, as Silver tables may contain more data

than Bronze tables, depending on the joins and enrichments applied. Reference: What is a Medallion

Architecture?, Transforming Bronze Tables in Silver Tables, What is the medallion lakehouse

architecture?

Auto Loader provides a Structured Streaming source called cloudFiles that can process new data files

as they arrive in cloud storage without any additional setup. Auto Loader uses a scalable key-value

store to track ingestion progress and ensure exactly-once semantics. Auto Loader can ingest various

file formats and load them into Delta Lake tables. Auto Loader is recommended for incremental data

ingestion with Delta Live Tables, which extends the functionality of Structured Streaming and allows

you to write declarative Python or SQL code to deploy a production-quality data

pipeline. Reference: What is Auto Loader?, What is Auto Loader? | Databricks on AWS, Solved: How

does Auto Loader ingest data? - Databricks - 5629

The keyword that can be used to compact the small files associated with a Delta table

is OPTIMIZE. The OPTIMIZE command performs file compaction on a Delta table by rewriting a set of

small files into a set of larger files1. This can improve the performance of queries that scan the table

by reducing the number of files that need to be read and the amount of metadata that needs to be

processed1. The OPTIMIZE command can also optionally sort the data within each file by a given set

of columns, which can further improve the query performance by enabling data skipping and

predicate pushdown1. The OPTIMIZE command can be applied to the whole table or to a specific

partition of the table1.

The other keywords are not suitable for compacting the small files associated with a Delta

table. REDUCE is a keyword used in the SQL syntax for aggregating data using a user-defined

function2. COMPACTION is not a valid keyword in SQL or Python. REPARTITION is a keyword used in

the Python syntax for changing the number of partitions of a DataFrame or an RDD3. VACUUM is a

keyword used to remove files that are no longer referenced by a Delta table and are older than a

retention threshold4.

Reference:

1: OPTIMIZE | Databricks on AWS

2: REDUCE | Databricks on AWS

3: repartition | Databricks on AWS

4: VACUUM | Databricks on AWS

https://www.w3schools.com/python/python_functions.asp

https://www.geeksforgeeks.org/python-functions/

In Databricks, you can use different languages within the same notebook by using magic commands.

Magic commands are special commands that start with a percentage sign (%) and allow you to

change the behavior of the cell. To use SQL within a cell of a Python notebook, you can add %sql to

the first line of the cell. This will tell Databricks to interpret the rest of the cell as SQL code and

execute it against the default database. You can also specify a different database by using the USE

statement. The result of the SQL query will be displayed as a table or a chart, depending on the

output mode. You can also assign the result to a Python variable by using the -o option. For example,

%sql -o df SELECT * FROM my_table will run the SQL query and store the result as a pandas

DataFrame in the Python variable df. Option A is incorrect, as it is possible to use SQL in a Python

notebook using magic commands. Option B is incorrect, as attaching the cell to a SQL endpoint is not

necessary and will not change the language of the cell. Option C is incorrect, as simply writing SQL

syntax in the cell will result in a syntax error, as the cell will still be interpreted as Python code.

Option E is incorrect, as changing the default language of the notebook to SQL will affect all the cells,

not just one. Reference: Use SQL in Notebooks - Knowledge Base - Noteable, [SQL magic commands -

Databricks], [Databricks SQL Guide - Databricks]

The spark.sql method can be used to execute SQL queries programmatically and return the result as

a DataFrame. The spark.sql method accepts a string argument that contains a valid SQL statement.

The data engineer can use a formatted string literal (f-string) to insert the Python variable

table_name into the SQL query. The other methods are either invalid or not suitable for running SQL

queries. Reference: Running SQL Queries Programmatically, Formatted string literals, spark.sql