Eclasess

Course available for 4 years after purchase

Duration: 4 Hours Hands-on Workshop
Objective (📅 Workshop Schedule
✅ Sat, 11 th April 2026
✅ Sun, 12 th April 2026

⏰ Time: 2:00 PM – 4:00 PM)
In this workshop students learn how to build an end-to-end data pipeline using Azure Data Lake
Gen2 and Azure Databricks with Medallion Architecture (Bronze, Silver, Gold).
Students will learn data storage, governance using Unity Catalog, notebook-based transformations,
presentation layer creation, reporting, and workflow automation.
1. Modern Data Architecture (Lakehouse)
The Lakehouse architecture combines the scalability of data lakes with the performance and
structure of data warehouses.
Key components include Azure Data Lake Gen2 for storage, Unity Catalog for governance, Azure
Databricks for processing, Delta Lake for reliability, and Databricks SQL for analytics and reporting.
2. Medallion Architecture
Data is organized into three layers:
Bronze: Raw data stored exactly as received.
Silver: Cleaned and transformed data.
Gold: Aggregated business-ready data used for reporting and analytics.
3. Azure Data Lake Gen2
Azure Data Lake Gen2 is a scalable storage service used to store structured and unstructured data.
Typical folder structure:
raw/ – stores original data files
processed/ – stores cleaned data
presentation/ – stores business-ready datasets
4. Unity Catalog
Unity Catalog provides centralized governance and security.
Hierarchy: Metastore → Catalog → Schema → Tables.
Example: catalog: workshop_catalog, schema: sales_schema, table: sales_table.
5. Connecting Data Lake to Databricks
Connection requires Access Connector, Storage Credential, and External Location.
Example SQL:
CREATE CATALOG workshop_catalog;
CREATE SCHEMA workshop_catalog.sales_schema;
6. Azure Databricks Workspace
Main components include clusters for compute, notebooks for code, libraries for dependencies, and
workflows for automation.
7. Creating a Cluster
Example configuration:
Cluster Name: workshop-cluster
Runtime Version: Databricks Runtime 13.x
Language: Python
8. Reading Data from Data Lake
Example Spark code:
df = spark.read.format("csv").option("header","true").load("abfss://raw@storageaccount.dfs.core.wi
ndows.net/sales_data.csv")
9. Data Cleaning
Cleaning includes removing duplicates, filtering invalid records, and handling missing values.
Example: dropDuplicates() and filter(sales_amount > 0).
10. Data Transformation
Transformations prepare data for analytics such as aggregations, joins, and calculations.
Example: group sales by region and calculate total sales.
11. Processed Layer (Silver Layer)
Cleaned data is stored in Delta format in the processed layer.
Delta Lake provides ACID transactions, schema enforcement, and time travel capabilities.
12. Presentation Layer (Gold Layer)
Business-ready aggregated datasets are stored in the presentation layer and used for reporting
tools or SQL queries.
13. Reporting using Databricks SQL
Example query:
SELECT region, SUM(total_sales)
FROM gold_sales
GROUP BY region;
14. Databricks Jobs and Workflows
Automation pipeline example:
Task 1 – Load Raw Data
Task 2 – Clean Data
Task 3 – Transform Data
Task 4 – Load Gold Layer
Jobs can be scheduled (e.g., daily at 8 AM).
Summary
Students learn Azure Data Lake Gen2 storage, Unity Catalog governance, Databricks notebook
processing, Medallion architecture, transformations, reporting, and workflow automation.

Project oriented DATA BRICKS & PYAPARK work shop 

U2FsdGVkX19OrqAkYjS6+ipn7KjbKtbQVlj0kPpHppOqxvi4W9YbYnsS2CkXKFOV

Day-1 Data bricks and pysprak workshop

Maharashtra

Hey! I’m learning with Eclasess and you can too. Here’s a course you might like: Project oriented DATA BRICKS & PYAPARK work shop  by Eclasess
    https://learn.eclasess.com/courses/814341?utm_source=whatsapp&utm_medium=social&utm_campaign=web-course-share

Hey! I’m learning with Eclasess and you can too. Here’s a course you might like: Project oriented DATA BRICKS & PYAPARK work shop  by Eclasess
    https://learn.eclasess.com/courses/814341?utm_source=facebook&utm_medium=social-fb&utm_campaign=web-course-share

Hey! I’m learning with Eclasess and you can too. Here’s a course you might like: Project oriented DATA BRICKS & PYAPARK work shop  by Eclasess
    https://learn.eclasess.com/courses/814341?utm_source=telegram&utm_medium=social&utm_campaign=web-course-share

Hey! I’m learning with Eclasess and you can too. Here’s a course you might like: Project oriented DATA BRICKS & PYAPARK work shop  by Eclasess
    https://learn.eclasess.com/courses/814341?utm_source=linkedin&utm_medium=social-link&utm_campaign=web-course-share

Hey! I’m learning with Eclasess and you can too. Here’s a course you might like: Project oriented DATA BRICKS & PYAPARK work shop  by Eclasess
    https://learn.eclasess.com/courses/814341?utm_source=copy-link&utm_medium=social&utm_campaign=web-course-share

You have Successfully Published Project oriented DATA BRICKS &amp; PYAPARK work shop 