Azure Data Engineering Project: MySQL to ADLS with ADF

Build a realistic Azure data engineering project that ingests financial services data from MySQL into Azure Data Lake Storage Gen2 using Azure Data Factory.

You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, designing incremental loads, setting up Azure resources, implementing the solution, validating the output, and scheduling repeatable runs.

In this project, you will:

• Prepare the source: Set up multiple financial services tables in MySQL.

• Connect the source: Configure a self-hosted integration runtime and ADF linked services.

• Build the pipeline: Ingest multiple source tables using Azure Data Factory.

• Load data incrementally: Use watermarks to process new and updated records.

• Store the data: Write the raw data to Azure Data Lake Storage Gen2.

• Track each run: Maintain configuration, run history, and watermarks in Azure SQL.

• Validate the output: Query and verify the ingested data using Azure Synapse.

• Schedule the solution: Use ADF triggers to run the pipeline on a schedule.

You will build the project in your own Azure account using the provided source code, sample MySQL data, configuration files, and infrastructure setup scripts.

The focus is not only on getting the pipeline to run. You will understand why each Azure service is used, how the components work together, and how the pipeline handles new and updated source records during subsequent runs.

By the end of the course, you will have a complete Azure data engineering project that you can practice, adapt for your portfolio, and explain clearly in interviews.

This course contains a promotion.