You need to schedule the population of the medallion layers to meet the technical requirements. What should you do?

The medallion architecture involves a sequential data flow (Bronze -> Silver -> Gold) where each layer's population is dependent on the successful completion of the previous one. The most robust and manageable way to implement this is through orchestration. Scheduling a single "master" or "controller" data pipeline that, in turn, calls other pipelines for each specific layer (e.g., a Bronze pipeline, a Silver pipeline) ensures proper dependency management. This pattern allows the Silver pipeline to run only after the Bronze pipeline succeeds, and the Gold pipeline only after Silver succeeds, meeting the technical requirements for data integrity and process flow.

B. Schedule a notebook.

Scheduling a single notebook creates a monolithic process, which is difficult to debug, manage, and reuse for individual layers.

C. Schedule an Apache Spark job.

This is similar to scheduling a notebook; it packages the logic into a single job, lacking the high-level orchestration and dependency management features of a pipeline.

D. Schedule multiple data pipelines.

Scheduling pipelines independently (e.g., based on time) does not enforce dependencies. If the Bronze pipeline is delayed or fails, the Silver pipeline might run with stale or incomplete data.

1. Microsoft Docs

"Execute Pipeline activity in Azure Data Factory and Azure Synapse Analytics": This document describes the activity used to invoke one pipeline from another. It states

"The Execute Pipeline activity allows a Synapse pipeline or Data Factory pipeline to invoke another pipeline." This is the core mechanism for the correct answer

enabling a master pipeline to orchestrate child pipelines.

2. Microsoft Learn

"Orchestrate data movement and transformation in Azure Synapse Pipelines": This learning module explains how to build complex workflows. It details using triggers to start a pipeline and chaining activities to ensure sequential execution. The concept of a master pipeline calling other pipelines is a primary pattern for managing complex

dependent workflows like populating a medallion architecture.

3. Microsoft Cloud Adoption Framework

"Data management and analytics scenario data platform operations": In the section on data orchestration

it discusses the need for "orchestration of data pipelines to run in a dependent order." This principle directly supports using a master pipeline to control the sequence of layer population in the medallion architecture.