Snowflake Data Engineer Training From SQL School Training Institute

Complete Practical and Real-time Snowflake Training with Basic to Advanced Cloud Concepts, ETL (Extract, Transform and Load), DWH (DataWarehouse), Big Data Storage, Big Data Analytics, Incremental Loads, External Stages, SnowPipes, Failsafe Concepts, Data Recovery, Security Management, ETL Management, Administration, Azure Integrations, Power BI Integrations, External Data Connections and more..

This Snowflake Training Classes also includes Concept wise FAQs, Real-time Project, Project Solution, Project FAQs for your Job Interviews, Snowflake Core Certification Guidance with Real-Life Scenarios.

Snowflake Data Engineer Training

	PLAN A	PLAN B	PLAN C
Course includes	Snowflake	Snowflake, Azure Data Engineer	Snowflake, Azure Data Engineer Power BI
Total Duration	4 Weeks	11 Weeks	15 Weeks
Snowflake: DWH and ETL	✔	✔	✔
Snowflake: Cloud Configurations	✔	✔	✔
Snowflake: DWH Architecture	✔	✔	✔
Snowflake: Database, Schemas	✔	✔	✔
Snowflake: Constraints, SP, Tasks	✔	✔	✔
Snowflake: Partitions & Tuning	✔	✔	✔
Snowflake: Security Management	✔	✔	✔
Snowflake: Automated Backups	✔	✔	✔
Snowflake: SnowSQL Concepts	✔	✔	✔
Snowflake: Variables & Tasks	✔	✔	✔
Snowflake: File Formats, Stages	✔	✔	✔
Snowflake: External Stages	✔	✔	✔
Snowflake: Azure Integration	✔	✔	✔
Snowflake: Snow Pipes & Upserts	✔	✔	✔
Snowflake: Power BI Analytics	✔	✔	✔
ADF : Azure Data Factory	❌	✔	✔
ADF : Data Imports, ETL	❌	✔	✔
ADF : Data Flows, Wrangling	❌	✔	✔
ADF : Transformations, ETL	❌	✔	✔
Synapse: Configuration, Loads	❌	✔	✔
Synapse: ETL with ADF, DWH	❌	✔	✔
Synapse: Performance Tuning	❌	✔	✔
Synapse: MPP, cDWH, DIUs	❌	✔	✔
ADB : Azure Data Bricks	❌	✔	✔
ADB : Architecture, Data Loads	❌	✔	✔
ADB : Run Spark Jobs, Pools	❌	✔	✔
ADB : Workspace, Delta Tables	❌	✔	✔
DP 203 Certification Guidance	❌	✔	✔
Power BI Report Design	❌	❌	✔
Power BI Desktop, Custom Visuals	❌	❌	✔
Data Modelling with Power Query	❌	❌	✔
Data Modelling with DAX	❌	❌	✔
Power BI Cloud, Excel Analysis	❌	❌	✔
Power BI Mobile, R, REST API	❌	❌	✔
PL 300 Exam Guidance	❌	❌	✔
Real-Time Project in Power BI	❌	❌	✔

Trainer: Mr.Sai Phanindra (17+ Yrs of Exper)

All Session Are Completely Practical & Real Time

S No	Time (IST, Sat & Sun)	Start Date
Snowflake Training Schedules
1	9 AM - 10 AM	June 6th	Register
2	6 PM - 7 PM	May 27th	Register

S No	Time (IST, Mon - Fri)	Start Date
Azure Data Engineer Training Schedules
3	7 AM - 8 AM	Apr 25th	Register
4	7 PM - 8 PM	May 8th	Register

S No	Time (IST, Mon - Fri)	Start Date
Power BI Training Schedules
5	8 AM - 9 AM	June 6th	Register
6	6 PM - 7 PM	Apr 27th	Register
7	9 PM - 10 PM	May 19th	Register

If above schedules does not work for you, please register for Power BI Training Videos

TRAINING HIGHLIGHTS

Custom Visuals	Power BI Mobile
Big Data Sources	R, Python, JSON
Excel Analysis	Excel Online
Google Big Query	Report Server
Gateways, Rest API	Power BI Admin
✔ Tabular Mode	✔ Data Models
✔ Tabular Mode OLAP	✔ Tabular Mode DAX
✔ Cube Design	✔ Excel Analytics

Mod 1: Snowflake Architecture, Basics	Mod 2: Snowflake Concepts, Admin	Mod 3: ETL, Stages & Pipes
Ch 1: Introduction to Cloud & Snowflake Database Introduction & Database Types OLTP, OLAP and Warehouse Databases Need for Datawarehouses; Advantages Popular Database Technologies Popular Datawarehouse Technologies Need for Cloud & Remote DWH Store Cloud Implementation Types & Usage IaaS: Infrastructure As A Service PaaS: Platform As A Service SaaS: Software As A Service Snowflake Cloud: Introduction Snowflake: Popular SaaS Platform Cloud Datawarehouse Concepts Advantages, Dependencies in Snowflake	Ch 7: Constraints and Data Types Snowflake Constraints, Data Validations NULL and NOT NULL Properties Unique, Primary and Foreign Keys Working with Named Constraints Single Column, Multi Column Constraints Inline and Out Of Line Constraints Table Constraints Types, Real-time Use Constraint Properties: ENFORCED DEFERRED, IMMEDIATE Options Numeric, String and Binary Data Types Boolean Data Types and Usage Date and Time Data Types Semi Structured Data Types Geospatial & Variant Data Types	Ch 13: Snowflake Tasks, Partitions Snowflake Tasks and Real-time Use Snowflake Serverless Compute Model User Managed & Snowflake Managed Tasks Tree: Root and Link Concepts Directed Acyclic Graph (DAG) Tasks Schedules and RESUME Options CREATE, ALTER, DESCRIBE, SHOW CREATE TASK … AFTER Statement ALTER TASK … ADD AFTER Statement CRON Syntax with Tasks, Procedures Virtual Warehouse Concepts (DWH) Multi Cluster Warehouse Config Client Utilities, Drivers & Connectors Auto Scale Options, Billing (Pricing)
Ch 2: Snowflake Account & Editions Creating Snowflake Account (Cloud) Snowflake Trail Account: Limitations, Uses Snowflake Account Components Cloud Platform: Azure, AWS, Google Cloud Regions and Availability with Snowflake Snowflake Editions, Credits; UI Usage Default Accounts; Web UI & Snow Sight Snowflake Storage: Ondemand, Capacity Snowflake Editions and Comparisons Standard Edition: Features, Advantages Enterprise Edition: Features, Advantages Business Critical Edition: Features, Usage Virtual Private Edition (VPS) & Pricing Snowflake Pricing: Ondemand Vs Capacity	Ch 8: Snowflake Cloning (Zero Copy) Cloning Operations with Snowflake Zero Copy and Schema Level Cloning Real-time Uses: Cloning in Snowflake Snapshot Concept, Metadata Options Possible Objects for Snowflake Clone Permissions for Snowflake Cloning Accessing, Controlling Cloned Objects Real-time Considerations @ Cloning Storage Layer and Metadata Layer Cloning with Foreign Key Constraints Cloning Snowflake Databases Cloning Snowflake Schemas, Tables Security and MANAGE GRANTS Cloning and COPY GRANTS Options	Ch 14: SnowSQL and Variables SnowSQL: Concepts, Client Installation SnowSQL Tool: Configuration Options SnowSQL: Account Authorization DDL, DML and SELECT Operations Snowflake SQL Query Syntax Format SnowSQL Command Line Options Working with DB, Schema and Tables Snowflake Variables, Batch Processing Snowflake SQL Data Types, Usage DECLARE, LET, BEGIN and END EXECUTE IMMEDIATE, FOR, END FOR Creating Warehouse, Database, Tables Granting Permissions, Query Execution Writing Output to Files (Win, MAC OS)
Ch 3: Architecture, Warehouse (DWH) Snowflake Architecture & Compute Shared Disk, Shared Nothing Architecture Cluster Nodes and Snowflake Clusters CPU & Memory Resources in Clusters Disk Storage and Network Communication Storage Layer and Cloud Service Layer Database Query Layer & Data Cycle Snowflake Datawarehouse Architecture MPP: Massively Parallel Processing Compute and Storage Components Column Store and Virtual Warehouse Datawarehouse Creation in SnowSight Classic UI with Snowflake; Navigations Data Load and Billing; Auto Suspend	Ch 9: Snowflake Procedures & Views Snowflake Procedures and Functions Creating and Using Stored Procedures Using CALL Command in Snowflake SQL and JavaScript Options with SPs Snowflake Deferred Name Resolution Overloading with Snowflake SPs Transactions & Injection Variables & CALL Command Using execute() with Procedures sqlText:command, createStatement Cursoring Operations Dynamic DML Operations with SPs Loops & next.scan() RETURN and RETURNS Statements Views & Query Storage DML and SELECT Operations on Views Regular Views, System Predefined Views Recursive Views, Parameterized Views	Ch 15: Snowflake Partitions, Stages Snowflake Partitions, Real-time use Service and Storage Layer Concepts Micro Partition with DML, CDC Cluster Key: Usage, ReClusters Depth and Overlap Properties Internal Partition Types & Usage List, Range and Hash Partitions SYSTEM$CLUSTERING_INFORMATION Snowflake Stages and Usage Internal and External Stages User Stages : Creation, Usage Table Stages: Creation, Usage Internal Named Stages, Usage COPY Command, Bulk Data Loads
Ch 4: Snowflake Databases & Tables Snowflake Databases and Data Storage Need for Snowflake Warehouse, Compute Database Objects and Hierarchy Snowflake Worksheet Parameters Database Creation with Snowflake UI Retention Time, DB List & Connections Permanent and Transient Databases Snowflake Tables and their Usage Permanent, Transient & Temp Tables Creating Tables with SnowSight Describe Options, Data Inserts CREATE TABLE AS SELECT Cloning Tables, Case Sensitivity Options Collation, ALTER, DROP & UNDROP	Ch 10: Security Management Security Management Concepts Security Entities with Snowflake Securable Objects, Users & Roles Prevligies and Privilege Groups Snowflake Security Hierarchy Organization, Account, Users, Roles Schema, Tables, Other DB Objects Creating and Using Roles, Users System Defined Roles and Usage Role Hierarchy and Dependency Creating and Working with Users Auditing Users and Password Policy RBAC: Role Based Access Control DAC: Discretionary Access Control	Ch 16: Azure & External Storage Working with Azure Storage Azure Subscription, Resources Create, Use Azure Storage Account Storage Containers, BLOB Data Using SnowSQL with Azure BLOB SAS [Shared Access Signature] Using SAS Key and FILE PATH External Stages in Snowflake File Formats: Creation, Usage Creating & Using External Stages Azure Storage with BLOB COPY INTO Command Usage Listing Stages with Snowflake Snowflake Patterns & RegEx
Ch 5: Time Travel & Transient Tables Time Travel Feature in Snowflake DML Operations and Silent Audits Continuous Data Protection Life Cycle Invoking Time Travel Feature in Snowflake Timestamp, Offset & Query ID Options Time Travel using Offset Feature Time Travel using Query ID Feature Data Recovery using TIMESTAMP Using OFFSET options in Real-world Fail Safe and UNDROP Operations Transient Tables and Real-time Usage Permanent Table and Real-time Usage Restrictions with Permanent Tables Identical Names and Naming Conflicts	Ch 11: Snowflake Transactions Working with Transactions (ACID) Atomicity, Consistency, Isolation Durability and Data Storage Options Transaction Types with Snowflake Implicit, Explicit and Auto Commit BEGIN TRANSACTION & COMMIT DDL Statements and Transactions BEGIN WORK Versus START current_transaction() and Usage to_timestamp_ltz and Usage Failed Transactions with SPs Batches Versus Transactions Transactions with Stored Procedures Scoped and INNER Transactions	Ch 17: SnowPipes & Incremental Loads Snowflake SnowPipes: Incremental Loads & SnowPipe Creating Azure Account For BLOB Creating Container, Generate SAS Key Creating Azure Queue and Event Grid Notification Integration in Snowflake Integrations: Show, Describe, Use Azure Active Directory & IAM Linking Azure AD with Snowflake Enterprise Application, Authentication File Formats with Regular Expr External Data Stages Data Unloading Concepts PUT and GET with SnowSQL Bulk Unloads; Data Preparation & Stages Data Unloading to User Stages Data Unloading to Table Stages
Ch 6: Schemas and Session Context Schemas: Creation, Real-time Usage Permanant and Transient Schemas Managed Schemas in Real-time Usage Verifying and Cloning Snowflake Schemas Invoking Schemas & Cloning Operations ALTER SCHEMA.. IF EXISTS Options Creating, Working with Managed Schema Snowflake Sessions (Workspaces) Session Context: Role and Warehouse Session Context: Database & Schema Working with Fully Qualified Names CTAS: Create Table As Select Data Loading with GUI, SQL Scripts Using Query and History Tab in GUI	Ch 12: Snowflake Streams & Audits Working with Snowflake Streams Stream Object and DML Auditing Snapshot Creation, Offset Options METADATA$ACTION Parameter METADATA$ISUPDATE Parameter METADATA$ROW_ID Parameter Stream Types: Standard Stream Append Only Stream & Usage Insert Only Stream & Usage Data Flow with Snowflake Streams Auditing INSERT, UPDATE, DELETE show streams; desc streams Streams on Transient Tables Time Travel; Using Stream Tables	Ch 18 : REAL-TIME PROJECT Phase 1: Legacy Data Systems Heterogenous Data Platforms File Data Sources Phase 2 Desining DWH in Snowflake Incremental Loads Capacity Planning External Data, Integrations SnowPipes and Azure BLOB Phase 3 Data Visualizations, Analytics End to End Implementations Project Solution & FAQs Resume & Certification Guidance

Mod 1: Snowflake Architecture, Basics

Mod 2: Snowflake Concepts, Admin

Mod 3: ETL, Stages & Pipes

Ch 1: Introduction to Cloud & Snowflake

Database Introduction & Database Types
OLTP, OLAP and Warehouse Databases
Need for Datawarehouses; Advantages
Popular Database Technologies
Popular Datawarehouse Technologies
Need for Cloud & Remote DWH Store
Cloud Implementation Types & Usage

IaaS: Infrastructure As A Service
PaaS: Platform As A Service
SaaS: Software As A Service
Snowflake Cloud: Introduction
Snowflake: Popular SaaS Platform
Cloud Datawarehouse Concepts
Advantages, Dependencies in Snowflake

Ch 7: Constraints and Data Types

Snowflake Constraints, Data Validations
NULL and NOT NULL Properties
Unique, Primary and Foreign Keys
Working with Named Constraints
Single Column, Multi Column Constraints
Inline and Out Of Line Constraints
Table Constraints Types, Real-time Use

Constraint Properties: ENFORCED
DEFERRED, IMMEDIATE Options
Numeric, String and Binary Data Types
Boolean Data Types and Usage
Date and Time Data Types
Semi Structured Data Types
Geospatial & Variant Data Types

Ch 13: Snowflake Tasks, Partitions

Snowflake Tasks and Real-time Use
Snowflake Serverless Compute Model
User Managed & Snowflake Managed
Tasks Tree: Root and Link Concepts
Directed Acyclic Graph (DAG)
Tasks Schedules and RESUME Options
CREATE, ALTER, DESCRIBE, SHOW

CREATE TASK … AFTER Statement
ALTER TASK … ADD AFTER Statement
CRON Syntax with Tasks, Procedures
Virtual Warehouse Concepts (DWH)
Multi Cluster Warehouse Config
Client Utilities, Drivers & Connectors
Auto Scale Options, Billing (Pricing)

Ch 2: Snowflake Account & Editions

Creating Snowflake Account (Cloud)
Snowflake Trail Account: Limitations, Uses
Snowflake Account Components
Cloud Platform: Azure, AWS, Google Cloud
Regions and Availability with Snowflake
Snowflake Editions, Credits; UI Usage
Default Accounts; Web UI & Snow Sight

Snowflake Storage: Ondemand, Capacity
Snowflake Editions and Comparisons
Standard Edition: Features, Advantages
Enterprise Edition: Features, Advantages
Business Critical Edition: Features, Usage
Virtual Private Edition (VPS) & Pricing
Snowflake Pricing: Ondemand Vs Capacity

Ch 8: Snowflake Cloning (Zero Copy)

Cloning Operations with Snowflake
Zero Copy and Schema Level Cloning
Real-time Uses: Cloning in Snowflake
Snapshot Concept, Metadata Options
Possible Objects for Snowflake Clone
Permissions for Snowflake Cloning
Accessing, Controlling Cloned Objects

Real-time Considerations @ Cloning
Storage Layer and Metadata Layer
Cloning with Foreign Key Constraints
Cloning Snowflake Databases
Cloning Snowflake Schemas, Tables
Security and MANAGE GRANTS
Cloning and COPY GRANTS Options

Ch 14: SnowSQL and Variables

SnowSQL: Concepts, Client Installation
SnowSQL Tool: Configuration Options
SnowSQL: Account Authorization
DDL, DML and SELECT Operations
Snowflake SQL Query Syntax Format
SnowSQL Command Line Options
Working with DB, Schema and Tables

Snowflake Variables, Batch Processing
Snowflake SQL Data Types, Usage
DECLARE, LET, BEGIN and END
EXECUTE IMMEDIATE, FOR, END FOR
Creating Warehouse, Database, Tables
Granting Permissions, Query Execution
Writing Output to Files (Win, MAC OS)

Ch 3: Architecture, Warehouse (DWH)

Snowflake Architecture & Compute
Shared Disk, Shared Nothing Architecture
Cluster Nodes and Snowflake Clusters
CPU & Memory Resources in Clusters
Disk Storage and Network Communication
Storage Layer and Cloud Service Layer
Database Query Layer & Data Cycle

Snowflake Datawarehouse Architecture
MPP: Massively Parallel Processing
Compute and Storage Components
Column Store and Virtual Warehouse
Datawarehouse Creation in SnowSight
Classic UI with Snowflake; Navigations
Data Load and Billing; Auto Suspend

Ch 9: Snowflake Procedures & Views

Snowflake Procedures and Functions
Creating and Using Stored Procedures
Using CALL Command in Snowflake
SQL and JavaScript Options with SPs
Snowflake Deferred Name Resolution
Overloading with Snowflake SPs
Transactions & Injection
Variables & CALL Command
Using execute() with Procedures

sqlText:command, createStatement
Cursoring Operations
Dynamic DML Operations with SPs
Loops & next.scan()
RETURN and RETURNS Statements
Views & Query Storage
DML and SELECT Operations on Views
Regular Views, System Predefined Views
Recursive Views, Parameterized Views

Ch 15: Snowflake Partitions, Stages

Snowflake Partitions, Real-time use
Service and Storage Layer Concepts
Micro Partition with DML, CDC
Cluster Key: Usage, ReClusters
Depth and Overlap Properties
Internal Partition Types & Usage
List, Range and Hash Partitions

SYSTEM$CLUSTERING_INFORMATION
Snowflake Stages and Usage
Internal and External Stages
User Stages : Creation, Usage
Table Stages: Creation, Usage
Internal Named Stages, Usage
COPY Command, Bulk Data Loads

Ch 4: Snowflake Databases & Tables

Snowflake Databases and Data Storage
Need for Snowflake Warehouse, Compute
Database Objects and Hierarchy
Snowflake Worksheet Parameters
Database Creation with Snowflake UI
Retention Time, DB List & Connections
Permanent and Transient Databases

Snowflake Tables and their Usage
Permanent, Transient & Temp Tables
Creating Tables with SnowSight
Describe Options, Data Inserts
CREATE TABLE AS SELECT
Cloning Tables, Case Sensitivity Options
Collation, ALTER, DROP & UNDROP

Ch 10: Security Management

Security Management Concepts
Security Entities with Snowflake
Securable Objects, Users & Roles
Prevligies and Privilege Groups
Snowflake Security Hierarchy
Organization, Account, Users, Roles
Schema, Tables, Other DB Objects

Creating and Using Roles, Users
System Defined Roles and Usage
Role Hierarchy and Dependency
Creating and Working with Users
Auditing Users and Password Policy
RBAC: Role Based Access Control
DAC: Discretionary Access Control

Ch 16: Azure & External Storage

Working with Azure Storage
Azure Subscription, Resources
Create, Use Azure Storage Account
Storage Containers, BLOB Data
Using SnowSQL with Azure BLOB
SAS [Shared Access Signature]
Using SAS Key and FILE PATH

External Stages in Snowflake
File Formats: Creation, Usage
Creating & Using External Stages
Azure Storage with BLOB
COPY INTO Command Usage
Listing Stages with Snowflake
Snowflake Patterns & RegEx

Ch 5: Time Travel & Transient Tables

Time Travel Feature in Snowflake
DML Operations and Silent Audits
Continuous Data Protection Life Cycle
Invoking Time Travel Feature in Snowflake
Timestamp, Offset & Query ID Options
Time Travel using Offset Feature
Time Travel using Query ID Feature

Data Recovery using TIMESTAMP
Using OFFSET options in Real-world
Fail Safe and UNDROP Operations
Transient Tables and Real-time Usage
Permanent Table and Real-time Usage
Restrictions with Permanent Tables
Identical Names and Naming Conflicts

Ch 11: Snowflake Transactions

Working with Transactions (ACID)
Atomicity, Consistency, Isolation
Durability and Data Storage Options
Transaction Types with Snowflake
Implicit, Explicit and Auto Commit
BEGIN TRANSACTION & COMMIT
DDL Statements and Transactions

BEGIN WORK Versus START
current_transaction() and Usage
to_timestamp_ltz and Usage
Failed Transactions with SPs
Batches Versus Transactions
Transactions with Stored Procedures
Scoped and INNER Transactions

Ch 17: SnowPipes & Incremental Loads

Snowflake SnowPipes: Incremental Loads & SnowPipe
Creating Azure Account For BLOB
Creating Container, Generate SAS Key
Creating Azure Queue and Event Grid
Notification Integration in Snowflake
Integrations: Show, Describe, Use
Azure Active Directory & IAM
Linking Azure AD with Snowflake

Enterprise Application, Authentication
File Formats with Regular Expr
External Data Stages
Data Unloading Concepts
PUT and GET with SnowSQL
Bulk Unloads; Data Preparation & Stages
Data Unloading to User Stages
Data Unloading to Table Stages

Ch 6: Schemas and Session Context

Schemas: Creation, Real-time Usage
Permanant and Transient Schemas
Managed Schemas in Real-time Usage
Verifying and Cloning Snowflake Schemas
Invoking Schemas & Cloning Operations
ALTER SCHEMA.. IF EXISTS Options
Creating, Working with Managed Schema

Snowflake Sessions (Workspaces)
Session Context: Role and Warehouse
Session Context: Database & Schema
Working with Fully Qualified Names
CTAS: Create Table As Select
Data Loading with GUI, SQL Scripts
Using Query and History Tab in GUI

Ch 12: Snowflake Streams & Audits

Working with Snowflake Streams
Stream Object and DML Auditing
Snapshot Creation, Offset Options
METADATA$ACTION Parameter
METADATA$ISUPDATE Parameter
METADATA$ROW_ID Parameter
Stream Types: Standard Stream

Append Only Stream & Usage
Insert Only Stream & Usage
Data Flow with Snowflake Streams
Auditing INSERT, UPDATE, DELETE
show streams; desc streams
Streams on Transient Tables
Time Travel; Using Stream Tables

Ch 18 : REAL-TIME PROJECT

Phase 1:
Legacy Data Systems
Heterogenous Data Platforms
File Data Sources
Phase 2
Desining DWH in Snowflake
Incremental Loads
Capacity Planning
External Data, Integrations
SnowPipes and Azure BLOB
Phase 3
Data Visualizations, Analytics
End to End Implementations
Project Solution & FAQs
Resume & Certification Guidance

Module 2 : Azure Data Engineer [Applicable for Plans B, C]

Mod 1: Azure Data Factory [ADF], Synapse	Mod 2: Azure Storage & Stream Analytics	Mod 3: Azure Databricks & SparkSQL
Chapter 1: Cloud Basics, Azure SQL DB Cloud Introduction and Azure Basics Azure Implementation: IaaS, PaaS, SaaS Benefits of Azure Cloud Environment Azure Data Engineer: Job Roles Azure Storage Components Azure ETL & Streaming Components Need for Azure Data Factory (ADF) Need for Azure Synapse Analytics Azure Resources and Resource Types Resource Groups in Azure Portal Azure SQL Server [Logical Server] Firewall Rules and Azure Services Connections with SSMS & ADS Tools Working with Azure Portal Resource Group Navigations, Options DP 203 Certification Guidance	Chapter 1: Azure Storage & Containers Storage Components in Microsoft Azure Azure Storage Services and Types - Uses High Availability, Durability & Scalability Blob: Binary Large Object Storage General Purpose: Gen 1 & Gen 2 Versions Blobs, File Share, Queues and Tables Data Lake Gen 2 Operations with Azure Azure Storage Account Creation Azure Storage Container: Usage Azure Data Explorer: Operations File Uploads, Edits and Access URLs Azure Storage Explorer Tool Usage Azure Account Options in Explorer Directory Creation, File Operations End User Access Options With Files Data Explorer Vs Storage Explorer Tool	Chapter 1: Azure Intro, Azure Databricks Azure Databricks : Purpose & Config Need for Azure Databricks (ADB) Azure Databricks Service Creation Azure Databricks Workspace & Usage Spark Cluster Configurations & Capacity Driver Nodes and Worker Nodes in Spark Master Node & Cluster Creation Process Cluster Types and Capacity Options Standard, High Concurrency Clusters Databricks Runtime Service & DBUs Databricks File System (DBFS) and Usage Azure Databricks Workspace Operations ETL and Data Storage Components Spark Concepts and Spark SQL Spark Context and Spark Session DataFrame, Dataset and Real-time Use
Chapter 2: Synapse SQL Pools (DWH) Dedicated SQL Pools in Azure Enterprise Data Warehouse with Synapse DWU: Data Warehouse Units, Resources Massively Parallel Processing (MPP) Control Nodes and Compute Nodes SQL Pool Access from SSMS Tool T-SQL Queries @ SQL Pools Start/Resume/Pause, Scaling Options Creating Tables in Azure SQL Pool Compression, MAX DOP & Indexes Distributions: Round Robin, Hash Distributions: Replicate and Usage Data Imports with COPY Table Dynamic Views (DMV) with PDW Data Loads Monitoring, Resource Class	Chapter 2: Azure Migration, BLOB Imports SQL Server (On-Premise) to Azure Migration Source Database Scripts & Validations BACPAC File Generation From SSMS Tool Azure Data Lake Storage and SSMS Access Azure Storage Container, BACPAC Files Azure SQL Server Creation From Portal Azure SQL DB Imports, Storage SAS Keys Azure SQL Database Migrations, Verification BLOB Data Access from On-Premise Data Imports From Excel and CSV Files BLOB Data Imports using T-SQL Queries SAS - Shared Access Signature Generation CSV File - Uploads, Downloads, Edits, Keys Master Keys, Credentials, External Sources BULK INSERT Statement and Data Imports T-SQL Imports : Practical Limitations	Chapter 2: SQL Notebooks & Python Notebooks: Concept, Usage Options Creating SQL Notebooks in Databricks Using DBFS Tables in SQL Notebooks Data Access and Analytics Options SparkSQL Queries: SELECT, GROUP BY SparkSQL Queries: Aggregates, Conditions Notebook Operations: Download, Clone Notebook Operations: Upload, Reuse SQL Notebooks with Python Code Using DBFS Sample Data Sources (CSV) Dataframes: Creation and Real-time Use Pandas Dataframe, Virtual Table Creation Dataframe Data Access, Caching Options Take() and Display() Functions in PySpark Temporary View Creation and Access SparkSQL Queries, Analytics, Chart Reports
Chapter 3: Azure Data Factory Concepts Azure Data Factory (ADF) Concepts Hybrid Data Integration at Scale ADF Pipeline Components & Usage Configure ADF Resource in Azure Understanding ADF Portal and IR Linked Services and Connections Datasets and Tables / Files for ETL ADF Pipelines: Design, Publish & Trigger ADF Pipeline with Copy Data Tool Creating Azure Storage Account Storage Container, BLOB File Uploads Data Loads with Azure BLOB Files DIU Allocations and Concurrency Creating Linked Services, Datasets Pipeline Trigger, Author and Monitor	Chapter 3: Azure Tables, Shares Azure Tables - Real-time Usage Schema-less Design and Access Options Structured and Relational Data Storage Tables, Entities and Properties Concepts Azure Tables: Creation and Data Inserts Azure Tables in Portal - GUI and Data Types Azure Tables: Data Imports in Explorer Data Edits, Queries & Delete Operations Azure Files - SMB Protocol, Creation, Usage Shared Access, Fully Managed & Resiliency Performance, Size Requirements for Shares Azure Storage Explorer Tool for File Shares Azure Queues: Message Queues, Limitations Adding Messages, Queuing and De-Queuing Data Access & Clear Queue from Explorer End Points for Azure Message Queues	Chapter 3: Python Notebooks Azure SQL Server Configurations Azure SQL Database Creation Azure Firewall Rules and IP Address Allow Azure Services, Remote Access Connection Tests with SSMS Tool Python Notebooks with Azure Databricks Data Imports and Table Creations (Code) Parquet Files and Usage in Databricks Using Dataframes for Data Operations SparkSQL Queries with SELECT, TOP Establishing Connections to Azure SQL DB JDBC Connection Strings, DataframeWriter JDBC Properties, Port Settings & Options Data Extraction, SQLContext & Dataframes Pandas Data Frame for Big Data Analytics JDBC URL Options & PySparkSQL Modules
Chapter 4: ADF Pipelines, Polybase Copy Data Tool For ETL Operations Azure SQL DB to Synapse Data Loads Working with Multi Tables Data Loads Query Options for Source Datasets Transformations with Copy Data Tool Rename, Rearrange & Remove Options Pipeline Execution: DTU & DOCP ADF Pipeline Monitoring Options ADF Pipelines: Execution Settings ADF Logging Options, Consistency Check Compression Option, DOP and DOCP ETL Staging Advantages & Performance Staging with Storage Account, Container ADF Pipeline Triggers and Monitoring Polybase For Azure Synapse, Advantages	Chapter 4: Azure Storage Security, Admin Azure Data Lake Storage Security Options Shared Access Keys - Primary, Secondary Keys SAS Key Generation: Container, Tables, Files SAS Key Permissions, Validation Options Access Keys: Account Level Permissions Azure Active Directory (AAD): Users, Groups Azure AD Security: RBAC with IAM, ACLs Owner Role, Contributor and Reader Role Azure Data Lake Storage Security Options ACL : Access Control Lists & Security Azure BLOB Storage Containers & ACLs Folder Level and File Level Security ACL Permissions: Read, Write & Execute Access Policy: Creation and Realtime Use Permissions: rwacdl; Azure Principals, CORS Comparing IAM and ACLs in Data Lake Store	Chapter 4: Open Data Sources, DeltaLakes Creating Python Notebooks with Databricks Spark Dataframes with Azure OpenDatasets Windows Azure Storage Blob [wasb] Sources Creating Dataframes & Temporary Views Using Print and Display Functions with ADB Big Data Analysis with BLOB Data & Charts Keys, Values, Aggregations, Display Type Databricks Notebooks, Jobs and Stages Azure DeltaLake Implementation ACID Properties and Upsert Advantages Delta Engine Optimizations & Uses Pipeline Creation with JSON Files in DBFS Delta Tables Creation, Data Loads Spark Cluster Settings: Auto Optimize Auto Compact and Delta Table Optimize Delta Locations; Data Retrieval, Versions
Chapter 5: OnPremise Data with ADF On-Premise Data Sources with Azure Self Hosted Integration Runtime (IR) Access Keys, Remote Linked Services Synapse SQL Pool (DW) with OnPremise Staged Data Copy and Performance Pipeline Executions and Monitoring Pipeline RunIDs and Audits / Tracing Incompatible Rows Skips, Fault Tolerance Incremental Loads with Files (BLOB) Pipeline Executions and Schedules Regular Schedules and Tumbling Window Execution Retry and Delay Options Binary Copy, Last Modified Date in Blob Automated Loops and Trigger Schedules Incremental Loads Verification Tests	Chapter 5: Azure Monitoring, Power BI Azure Monitor, Metrics & Logs Monitoring Azure Storage Namespaces Add KQL Metrics; Account, Blob and File Total Ingress and Egress Metrics: Charts Average Latency, Transaction Count Request Breakdowns, Signal Logic Options Azure Alerts and Conditions, Notifications Signal Logic Conditions and Emails Power BI Desktop Tool Installation Binary Data and Record Data Access Azure Data Lake Storage: Access Keys Azure Data Lake Storage with Power BI BLOB File Access with Power BI Azure Tables Creation and File Imports Azure Table Access with Power BI	Chapter 5: Databricks Security & Jobs Azure Databricks Security Operations Azure Active Directory (Azure AD) AD Users and RBAC with IAM Owner, Contributor & Reader Roles Workspace Admin Permissions Notebook Permissions and Share Options Shared Notebooks, User Access Options Notebook Operations: Clone & Export Databricks Jobs: Creation Options, Usage Job Limits, Workspace, Concurrency Limits Notebooks with and without Parameters Jobs with Default Parameters, Executions Interactive, Automated Clusters for Jobs Job Schedules and Manual Executions Active Jobs, Recently Run Jobs, Monitoring ADB Jobs with Azure OData Sources, BLOB
Chapter 6: ADF Data Flow - 1 Limitations with Copy Data Tool Data Flow Task, Data Flow Activity Transformations with Data Flow Spark Cluster For Debugging Cluster Node Configurations Data Preview Options with DFT SELECT Transformation & Options JOIN Transformation and Usage Conditional Split Transformation Aggregate & Group By Transformations Synapse Sink Options with DFT DFT Optimization Techniques Pipeline Debug Runs and ETL Testing Spark Cluster For Pipeline Executions Pipeline Monitoring & Run IDs	Chapter 6: Azure Stream Analytics, IoT Azure Stream Analytics: Real-time Usage Real-time Data Processing, Event Tracking\ Ingest, Deliver and Analysis Operations Azure Stream Analytics Jobs Concept Understanding Input & Output Options SAQL Queries for Stream Analytics Jobs IoT: Internet Of Things For Real-time Data Need for IoT Hubs and Event Hubs Creating IoT Device for Data Inputs Creating Azure Strean Analytics Resource Stream Analytics Jobs for Historical Data Azure SQL Database Options for ASA Jobs SAQL: Query Formatting and Validation Historical Data Uploads, ASA Job Execution Stream Analytics Job Monitoring Options	Chapter 6: Databricks @ BLOB, Power BI BLOB Data Access with Databricks Accessing Storage Account, Container Gerate, Use SAS: Shared Access Signature dbutils.fs.mount() with DBFS Store fs.azure.sas.container.strorageaccount spark.read() and DBFS Mounts Scala Transformations, Create Temp View Spark SQL Queries with Temp Views dataframe.write.jdbc() & JVM Properties spark.read.jdbc() with Azure SQL DB Power BI Integration with Databricks Server Host Name, Port and Http Path Cluster Configurations and JDBC User Access Token Generation, Usage Spark ClusterAccess, Power BI Analytics
Chapter 7: ADF Data Flow - 2 ADF Pipelines For ETL Operations Data Flow Tasks and Activities in Synapse Pivot Transformation For Normalization Generating Pivot Column, Aggregations Pivot Transformation and Pivot Settings Pivot Key Selection, Value and Nulls Pivoted Columns and Column Pattern Column Prefix, Help Graphic & Metadata Window Functions & Usage in Data Flow Rank / DenseRank / Row Number Over Clause and Input Options Derived Column Transformations Exists & Lookup Transformations Reusing Data Flow Tasks in Synapse Pipeline Validations & Executions	Chapter 7: IoT Hubs & Event Hubs Azure Stream Analytics For API Data IoT Hubs & IoT Devices, Connection Strings Rasberry APP Connections with IoT Hub Azure Storage Account and Container Creating Azure Stream Analytics Job Configuring Input Aliases with IoT Hub Configuring Output Alias with ADLS Gen 2 SAQL Query and Job Executions; Monitoring Azure Event Hubs and Event Instances Event Hub Namespaces, Partition Counts Access Policies, Permissions & Defaults RootManageSharedAccessKey & Options Connection Strings & Event Service Bus Telco App Installation, Executions. LIVE Data On-Premise App Integration with ASA Jobs	Chapter 7: Databricks Integrations Azure Databricks with Data Lake Storage Handling Unstructured Data in Azure Data Preparation and Staging Operations Azure App (Service Principal) Registration Azure Key Vault Creation & Key Usage Service Principal Permissions @ Data Loads Tenants and Authorization Settings Client Credentials, Token Provider Options Spark Notebooks For Dynamic Connections Parameterized Options & Blob Access Data Preparation & Big Data Ingestion Data Extraction and ADLS Storage show(), transformations, wasbs Options Azure SQL Server & Synapse Creations Data Loads with Incremental Changes
Chapter 8: Azure Synapse Analytics Azure Synapse Analytics Resource Azure Synapse Analytics Workspace Managed Resource Group, SQL Account SQL Admin Account and its Purpose Operations with Synapse Workspace ADLS Gen 2 Storage Account, Container Synapse Studio (Synapse Portal) Dedicated SQL Pools & Spark Pools Creating Dedicated SQL Pools Synapse Tables, Data Loads with T-SQL COPY INTO Statements with T-SQL Clustered Column Store Indexes Row Terminator and Compressions T-SQL Queries and Aggregations Aggregation Data Loads in Synapse	Chapter 8: Azure Stream Analytics Security Azure Key Vaults & ADLS [Data Lake] Security Azure Passwords, Keys and Certificates Azure Key Vaults - Name and Vault URI Inbuilt Managed Key and Azure Key Vault Standard Type, Premium Type Azure Key Vaults Secret Page, Key Backups and Key Restores Adding Keys to Azure Vaults. Key Type, Size Using Azure Key Vaults to secure Resources Azure Storage: Replications and DR Options LRS: Locally Redundant Storage GRS: Globally Redundant Storage ZRS: Zone Redundant Storage Replication Options and Advantages Replication Verification and Modifications Azure Storage Endpoints, Failover Partner	Real-Time Project ADF Integration, Real-time Project Azure Databricks Integrations with ADF Defining Scala Notebooks in ADB Using Notebooks in Azure Data Factory spark.conf.set & fs.azure.account.key spark.read.format, Option() and Head() Online Retail Database Data Source Azure Migrations and ETL Concepts Azure SQL Pool (Synapse DWH) Tables Apache Spark Pool : Databases, Tables Azure Data Lake Storage (ADLS Gen 2) Azure Stream Analytics Jobs with IoT Azure Data Bricks and DBFS, Notebooks Concept wise FAQs, Resume Guidance Project Requirement, Solution, FAQs DP 203 Certification Guidance
Chapter 9: Synapse Analytics with Spark Apache Spark Pool in Azure Synapse Spark Cluster Nodes: Vcores, Memory Creating Spark Clusters @ Synapse Studio Python Notebooks For Remote Access Creating Databases in Apache Spark Pool Data Loads from Dedicated SQL Pools Table Creations, Aggregation Operations PySpark Code for Data Operations, Writes Serverless Pool in Azure Synapse Connections, Usage with Serverless Pool Using Azure OpenDatasets in Synapse OPENROWSET and BULK Data Loads Azure Storage Account : Data Analysis Working with Parquet Files in Synapse Python Notebooks (Pyspark) in Synapse
Chapter 10: Incremental Loads @ Synapse Incremental Loads with Synapse Studio Multi Table Merge Operations On-Premise Data Sources & Timestamps Azure SQL DB Destinations, Watermarks Watermark Table Usage & Audits Stored Procedures for Timestamp Updates Table Data Type and Dynamic MERGE SQL Queries for Datasets and Fetch Lookup Activity and its Usage un Synapse Expressions in ADF Portal for Lookup Expressions in ADF Portal for Source Output Pipeline Expression, Data Window Concat Function, Run IDs Expressions JSON Parameters, Pipeline Scheduling Pipeline Validation, Trigger and Monitoring
Chapter 11: Optimizations, Power Query ADF ETL with GUI : Power Query Power Query Resoruce Creation, Use Source Data Configurations & Settings Rename, Remove, Pivot, Group By, Order Index, Filter, Remove Error Rows Using Power Query Activity, ADF Pipelines Spark Cluster Configurations for Pipelines Concurrency, Big Data Recommendations Storage Optimization Techniques ETL Optimization Techniques SQL Pool (Synapse) Optimizations Indexes, Partitions, Distributions, DOP Pipeline Optimization Techniques Partitions, DOCP, Compressions, DIU Staging, Polybase and Core Counts
Chapter 12: Pipeline Monitoring, Security Azure Monitor Resource and Usage Pipeline Monitoring Techniques ADF: Pipeline Monitoring and Alerts Synapse: Pipeline Monitoring and Alerts Synapse: Storage Monitoring and Alerts Conditions, Signal Rules and Metrics Email Notifications with Azure Concurrency, Big Data Recommendations Azure Active Directory (AAD) Users, Groups IAM: Identity & Access Management Synapse Workspace Security with RBAC ADF Security with RBAC: Owner, Contributor Azure Synapse SQL Pool Security: Logins Users, Roles and Resource Classes (RC) ADF V1 to V2 Migrations, Considerations

Mod 1: Azure Data Factory [ADF], Synapse

Mod 2: Azure Storage & Stream Analytics

Mod 3: Azure Databricks & SparkSQL

Chapter 1: Cloud Basics, Azure SQL DB

Cloud Introduction and Azure Basics
Azure Implementation: IaaS, PaaS, SaaS
Benefits of Azure Cloud Environment
Azure Data Engineer: Job Roles
Azure Storage Components
Azure ETL & Streaming Components
Need for Azure Data Factory (ADF)
Need for Azure Synapse Analytics

Azure Resources and Resource Types
Resource Groups in Azure Portal
Azure SQL Server [Logical Server]
Firewall Rules and Azure Services
Connections with SSMS & ADS Tools
Working with Azure Portal
Resource Group Navigations, Options
DP 203 Certification Guidance

Chapter 1: Azure Storage & Containers

Storage Components in Microsoft Azure
Azure Storage Services and Types - Uses
High Availability, Durability & Scalability
Blob: Binary Large Object Storage
General Purpose: Gen 1 & Gen 2 Versions
Blobs, File Share, Queues and Tables
Data Lake Gen 2 Operations with Azure
Azure Storage Account Creation

Azure Storage Container: Usage
Azure Data Explorer: Operations
File Uploads, Edits and Access URLs
Azure Storage Explorer Tool Usage
Azure Account Options in Explorer
Directory Creation, File Operations
End User Access Options With Files
Data Explorer Vs Storage Explorer Tool

Chapter 1: Azure Intro, Azure Databricks

Azure Databricks : Purpose & Config
Need for Azure Databricks (ADB)
Azure Databricks Service Creation
Azure Databricks Workspace & Usage
Spark Cluster Configurations & Capacity
Driver Nodes and Worker Nodes in Spark
Master Node & Cluster Creation Process
Cluster Types and Capacity Options

Standard, High Concurrency Clusters
Databricks Runtime Service & DBUs
Databricks File System (DBFS) and Usage
Azure Databricks Workspace Operations
ETL and Data Storage Components
Spark Concepts and Spark SQL
Spark Context and Spark Session
DataFrame, Dataset and Real-time Use

Chapter 2: Synapse SQL Pools (DWH)

Dedicated SQL Pools in Azure
Enterprise Data Warehouse with Synapse
DWU: Data Warehouse Units, Resources
Massively Parallel Processing (MPP)
Control Nodes and Compute Nodes
SQL Pool Access from SSMS Tool
T-SQL Queries @ SQL Pools
Start/Resume/Pause, Scaling Options

Creating Tables in Azure SQL Pool
Compression, MAX DOP & Indexes
Distributions: Round Robin, Hash
Distributions: Replicate and Usage
Data Imports with COPY Table
Dynamic Views (DMV) with PDW
Data Loads Monitoring, Resource Class

Chapter 2: Azure Migration, BLOB Imports

SQL Server (On-Premise) to Azure Migration
Source Database Scripts & Validations
BACPAC File Generation From SSMS Tool
Azure Data Lake Storage and SSMS Access
Azure Storage Container, BACPAC Files
Azure SQL Server Creation From Portal
Azure SQL DB Imports, Storage SAS Keys
Azure SQL Database Migrations, Verification

BLOB Data Access from On-Premise
Data Imports From Excel and CSV Files
BLOB Data Imports using T-SQL Queries
SAS - Shared Access Signature Generation
CSV File - Uploads, Downloads, Edits, Keys
Master Keys, Credentials, External Sources
BULK INSERT Statement and Data Imports
T-SQL Imports : Practical Limitations

Chapter 2: SQL Notebooks & Python

Notebooks: Concept, Usage Options
Creating SQL Notebooks in Databricks
Using DBFS Tables in SQL Notebooks
Data Access and Analytics Options
SparkSQL Queries: SELECT, GROUP BY
SparkSQL Queries: Aggregates, Conditions
Notebook Operations: Download, Clone
Notebook Operations: Upload, Reuse

SQL Notebooks with Python Code
Using DBFS Sample Data Sources (CSV)
Dataframes: Creation and Real-time Use
Pandas Dataframe, Virtual Table Creation
Dataframe Data Access, Caching Options
Take() and Display() Functions in PySpark
Temporary View Creation and Access
SparkSQL Queries, Analytics, Chart Reports

Chapter 3: Azure Data Factory Concepts

Azure Data Factory (ADF) Concepts
Hybrid Data Integration at Scale
ADF Pipeline Components & Usage
Configure ADF Resource in Azure
Understanding ADF Portal and IR
Linked Services and Connections
Datasets and Tables / Files for ETL
ADF Pipelines: Design, Publish & Trigger

ADF Pipeline with Copy Data Tool
Creating Azure Storage Account
Storage Container, BLOB File Uploads
Data Loads with Azure BLOB Files
DIU Allocations and Concurrency
Creating Linked Services, Datasets
Pipeline Trigger, Author and Monitor

Chapter 3: Azure Tables, Shares

Azure Tables - Real-time Usage
Schema-less Design and Access Options
Structured and Relational Data Storage
Tables, Entities and Properties Concepts
Azure Tables: Creation and Data Inserts
Azure Tables in Portal - GUI and Data Types
Azure Tables: Data Imports in Explorer
Data Edits, Queries & Delete Operations

Azure Files - SMB Protocol, Creation, Usage
Shared Access, Fully Managed & Resiliency
Performance, Size Requirements for Shares
Azure Storage Explorer Tool for File Shares
Azure Queues: Message Queues, Limitations
Adding Messages, Queuing and De-Queuing
Data Access & Clear Queue from Explorer
End Points for Azure Message Queues

Chapter 3: Python Notebooks

Azure SQL Server Configurations
Azure SQL Database Creation
Azure Firewall Rules and IP Address
Allow Azure Services, Remote Access
Connection Tests with SSMS Tool
Python Notebooks with Azure Databricks
Data Imports and Table Creations (Code)
Parquet Files and Usage in Databricks

Using Dataframes for Data Operations
SparkSQL Queries with SELECT, TOP
Establishing Connections to Azure SQL DB
JDBC Connection Strings, DataframeWriter
JDBC Properties, Port Settings & Options
Data Extraction, SQLContext & Dataframes
Pandas Data Frame for Big Data Analytics
JDBC URL Options & PySparkSQL Modules

Chapter 4: ADF Pipelines, Polybase

Copy Data Tool For ETL Operations
Azure SQL DB to Synapse Data Loads
Working with Multi Tables Data Loads
Query Options for Source Datasets
Transformations with Copy Data Tool
Rename, Rearrange & Remove Options
Pipeline Execution: DTU & DOCP
ADF Pipeline Monitoring Options

ADF Pipelines: Execution Settings
ADF Logging Options, Consistency Check
Compression Option, DOP and DOCP
ETL Staging Advantages & Performance
Staging with Storage Account, Container
ADF Pipeline Triggers and Monitoring
Polybase For Azure Synapse, Advantages

Chapter 4: Azure Storage Security, Admin

Azure Data Lake Storage Security Options
Shared Access Keys - Primary, Secondary Keys
SAS Key Generation: Container, Tables, Files
SAS Key Permissions, Validation Options
Access Keys: Account Level Permissions
Azure Active Directory (AAD): Users, Groups
Azure AD Security: RBAC with IAM, ACLs
Owner Role, Contributor and Reader Role

Azure Data Lake Storage Security Options
ACL : Access Control Lists & Security
Azure BLOB Storage Containers & ACLs
Folder Level and File Level Security
ACL Permissions: Read, Write & Execute
Access Policy: Creation and Realtime Use
Permissions: rwacdl; Azure Principals, CORS
Comparing IAM and ACLs in Data Lake Store

Chapter 4: Open Data Sources, DeltaLakes

Creating Python Notebooks with Databricks
Spark Dataframes with Azure OpenDatasets
Windows Azure Storage Blob [wasb] Sources
Creating Dataframes & Temporary Views
Using Print and Display Functions with ADB
Big Data Analysis with BLOB Data & Charts
Keys, Values, Aggregations, Display Type
Databricks Notebooks, Jobs and Stages

Azure DeltaLake Implementation
ACID Properties and Upsert Advantages
Delta Engine Optimizations & Uses
Pipeline Creation with JSON Files in DBFS
Delta Tables Creation, Data Loads
Spark Cluster Settings: Auto Optimize
Auto Compact and Delta Table Optimize
Delta Locations; Data Retrieval, Versions

Chapter 5: OnPremise Data with ADF

On-Premise Data Sources with Azure
Self Hosted Integration Runtime (IR)
Access Keys, Remote Linked Services
Synapse SQL Pool (DW) with OnPremise
Staged Data Copy and Performance
Pipeline Executions and Monitoring
Pipeline RunIDs and Audits / Tracing
Incompatible Rows Skips, Fault Tolerance

Incremental Loads with Files (BLOB)
Pipeline Executions and Schedules
Regular Schedules and Tumbling Window
Execution Retry and Delay Options
Binary Copy, Last Modified Date in Blob
Automated Loops and Trigger Schedules
Incremental Loads Verification Tests

Chapter 5: Azure Monitoring, Power BI

Azure Monitor, Metrics & Logs
Monitoring Azure Storage Namespaces
Add KQL Metrics; Account, Blob and File
Total Ingress and Egress Metrics: Charts
Average Latency, Transaction Count
Request Breakdowns, Signal Logic Options
Azure Alerts and Conditions, Notifications
Signal Logic Conditions and Emails

Power BI Desktop Tool Installation
Binary Data and Record Data Access
Azure Data Lake Storage: Access Keys
Azure Data Lake Storage with Power BI
BLOB File Access with Power BI
Azure Tables Creation and File Imports
Azure Table Access with Power BI

Chapter 5: Databricks Security & Jobs

Azure Databricks Security Operations
Azure Active Directory (Azure AD)
AD Users and RBAC with IAM
Owner, Contributor & Reader Roles
Workspace Admin Permissions
Notebook Permissions and Share Options
Shared Notebooks, User Access Options
Notebook Operations: Clone & Export

Databricks Jobs: Creation Options, Usage
Job Limits, Workspace, Concurrency Limits
Notebooks with and without Parameters
Jobs with Default Parameters, Executions
Interactive, Automated Clusters for Jobs
Job Schedules and Manual Executions
Active Jobs, Recently Run Jobs, Monitoring
ADB Jobs with Azure OData Sources, BLOB

Chapter 6: ADF Data Flow - 1

Limitations with Copy Data Tool
Data Flow Task, Data Flow Activity
Transformations with Data Flow
Spark Cluster For Debugging
Cluster Node Configurations
Data Preview Options with DFT
SELECT Transformation & Options
JOIN Transformation and Usage

Conditional Split Transformation
Aggregate & Group By Transformations
Synapse Sink Options with DFT
DFT Optimization Techniques
Pipeline Debug Runs and ETL Testing
Spark Cluster For Pipeline Executions
Pipeline Monitoring & Run IDs

Chapter 6: Azure Stream Analytics, IoT

Azure Stream Analytics: Real-time Usage
Real-time Data Processing, Event Tracking\
Ingest, Deliver and Analysis Operations
Azure Stream Analytics Jobs Concept
Understanding Input & Output Options
SAQL Queries for Stream Analytics Jobs
IoT: Internet Of Things For Real-time Data
Need for IoT Hubs and Event Hubs

Creating IoT Device for Data Inputs
Creating Azure Strean Analytics Resource
Stream Analytics Jobs for Historical Data
Azure SQL Database Options for ASA Jobs
SAQL: Query Formatting and Validation
Historical Data Uploads, ASA Job Execution
Stream Analytics Job Monitoring Options

Chapter 6: Databricks @ BLOB, Power BI

BLOB Data Access with Databricks
Accessing Storage Account, Container
Gerate, Use SAS: Shared Access Signature
dbutils.fs.mount() with DBFS Store
fs.azure.sas.container.strorageaccount
spark.read() and DBFS Mounts
Scala Transformations, Create Temp View
Spark SQL Queries with Temp Views

dataframe.write.jdbc() & JVM Properties
spark.read.jdbc() with Azure SQL DB
Power BI Integration with Databricks
Server Host Name, Port and Http Path
Cluster Configurations and JDBC
User Access Token Generation, Usage
Spark ClusterAccess, Power BI Analytics

Chapter 7: ADF Data Flow - 2

ADF Pipelines For ETL Operations
Data Flow Tasks and Activities in Synapse
Pivot Transformation For Normalization
Generating Pivot Column, Aggregations
Pivot Transformation and Pivot Settings
Pivot Key Selection, Value and Nulls
Pivoted Columns and Column Pattern
Column Prefix, Help Graphic & Metadata

Window Functions & Usage in Data Flow
Rank / DenseRank / Row Number
Over Clause and Input Options
Derived Column Transformations
Exists & Lookup Transformations
Reusing Data Flow Tasks in Synapse
Pipeline Validations & Executions

Chapter 7: IoT Hubs & Event Hubs

Azure Stream Analytics For API Data
IoT Hubs & IoT Devices, Connection Strings
Rasberry APP Connections with IoT Hub
Azure Storage Account and Container
Creating Azure Stream Analytics Job
Configuring Input Aliases with IoT Hub
Configuring Output Alias with ADLS Gen 2
SAQL Query and Job Executions; Monitoring

Azure Event Hubs and Event Instances
Event Hub Namespaces, Partition Counts
Access Policies, Permissions & Defaults
RootManageSharedAccessKey & Options
Connection Strings & Event Service Bus
Telco App Installation, Executions. LIVE Data
On-Premise App Integration with ASA Jobs

Chapter 7: Databricks Integrations

Azure Databricks with Data Lake Storage
Handling Unstructured Data in Azure
Data Preparation and Staging Operations
Azure App (Service Principal) Registration
Azure Key Vault Creation & Key Usage
Service Principal Permissions @ Data Loads
Tenants and Authorization Settings
Client Credentials, Token Provider Options

Spark Notebooks For Dynamic Connections
Parameterized Options & Blob Access
Data Preparation & Big Data Ingestion
Data Extraction and ADLS Storage
show(), transformations, wasbs Options
Azure SQL Server & Synapse Creations
Data Loads with Incremental Changes

Chapter 8: Azure Synapse Analytics

Azure Synapse Analytics Resource
Azure Synapse Analytics Workspace
Managed Resource Group, SQL Account
SQL Admin Account and its Purpose
Operations with Synapse Workspace
ADLS Gen 2 Storage Account, Container
Synapse Studio (Synapse Portal)
Dedicated SQL Pools & Spark Pools

Creating Dedicated SQL Pools
Synapse Tables, Data Loads with T-SQL
COPY INTO Statements with T-SQL
Clustered Column Store Indexes
Row Terminator and Compressions
T-SQL Queries and Aggregations
Aggregation Data Loads in Synapse

Chapter 8: Azure Stream Analytics Security

Azure Key Vaults & ADLS [Data Lake] Security
Azure Passwords, Keys and Certificates
Azure Key Vaults - Name and Vault URI
Inbuilt Managed Key and Azure Key Vault
Standard Type, Premium Type Azure Key Vaults
Secret Page, Key Backups and Key Restores
Adding Keys to Azure Vaults. Key Type, Size
Using Azure Key Vaults to secure Resources

Azure Storage: Replications and DR Options
LRS: Locally Redundant Storage
GRS: Globally Redundant Storage
ZRS: Zone Redundant Storage
Replication Options and Advantages
Replication Verification and Modifications
Azure Storage Endpoints, Failover Partner

Real-Time Project

ADF Integration, Real-time Project
Azure Databricks Integrations with ADF
Defining Scala Notebooks in ADB
Using Notebooks in Azure Data Factory
spark.conf.set & fs.azure.account.key
spark.read.format, Option() and Head()
Online Retail Database Data Source
Azure Migrations and ETL Concepts
Azure SQL Pool (Synapse DWH) Tables

Apache Spark Pool : Databases, Tables
Azure Data Lake Storage (ADLS Gen 2)
Azure Stream Analytics Jobs with IoT
Azure Data Bricks and DBFS, Notebooks
Concept wise FAQs, Resume Guidance
Project Requirement, Solution, FAQs
DP 203 Certification Guidance

Chapter 9: Synapse Analytics with Spark

Apache Spark Pool in Azure Synapse
Spark Cluster Nodes: Vcores, Memory
Creating Spark Clusters @ Synapse Studio
Python Notebooks For Remote Access
Creating Databases in Apache Spark Pool
Data Loads from Dedicated SQL Pools
Table Creations, Aggregation Operations
PySpark Code for Data Operations, Writes

Serverless Pool in Azure Synapse
Connections, Usage with Serverless Pool
Using Azure OpenDatasets in Synapse
OPENROWSET and BULK Data Loads
Azure Storage Account : Data Analysis
Working with Parquet Files in Synapse
Python Notebooks (Pyspark) in Synapse

Chapter 10: Incremental Loads @ Synapse

Incremental Loads with Synapse Studio
Multi Table Merge Operations
On-Premise Data Sources & Timestamps
Azure SQL DB Destinations, Watermarks
Watermark Table Usage & Audits
Stored Procedures for Timestamp Updates
Table Data Type and Dynamic MERGE
SQL Queries for Datasets and Fetch

Lookup Activity and its Usage un Synapse
Expressions in ADF Portal for Lookup
Expressions in ADF Portal for Source
Output Pipeline Expression, Data Window
Concat Function, Run IDs Expressions
JSON Parameters, Pipeline Scheduling
Pipeline Validation, Trigger and Monitoring

Chapter 11: Optimizations, Power Query

ADF ETL with GUI : Power Query
Power Query Resoruce Creation, Use
Source Data Configurations & Settings
Rename, Remove, Pivot, Group By, Order
Index, Filter, Remove Error Rows
Using Power Query Activity, ADF Pipelines
Spark Cluster Configurations for Pipelines
Concurrency, Big Data Recommendations

Storage Optimization Techniques
ETL Optimization Techniques
SQL Pool (Synapse) Optimizations
Indexes, Partitions, Distributions, DOP
Pipeline Optimization Techniques
Partitions, DOCP, Compressions, DIU
Staging, Polybase and Core Counts

Chapter 12: Pipeline Monitoring, Security

Azure Monitor Resource and Usage
Pipeline Monitoring Techniques
ADF: Pipeline Monitoring and Alerts
Synapse: Pipeline Monitoring and Alerts
Synapse: Storage Monitoring and Alerts
Conditions, Signal Rules and Metrics
Email Notifications with Azure
Concurrency, Big Data Recommendations

Azure Active Directory (AAD) Users, Groups
IAM: Identity & Access Management
Synapse Workspace Security with RBAC
ADF Security with RBAC: Owner, Contributor
Azure Synapse SQL Pool Security: Logins
Users, Roles and Resource Classes (RC)
ADF V1 to V2 Migrations, Considerations

Module 3 : Power BI [Applicable for Plans C]

Part 1: Power BI Report Design	Part 2: ETL, Data Modeling, DAX	Part 3: Power BI Cloud, Admin
Ch 1 : POWER BI BASICS Power BI Job Roles in Real-time Power BI Data Analyst Job Roles Business Analyst - Job Roles Power BI Developer - Job Roles Power BI for Data Scientists Comparing MSBI and Power BI Comparing Tableau and Power BI PL 300 Certification Guidance Types of Reports in Real-World Interactive & Paginated Reports Analytical & Mobile Reports Data Sources Types in Power BI Power BI Licensing Plans - Types Power BI Training : Lab Plan Power BI Dev & Prod Environments	Ch 7 : POWER QUERY LEVEL 1 Power Query M Language Purpose Power Query Architecture and ETL Data Types, Literals and Values Power Query Transformation Types Table & Column Transformations Text & Number Transformations Date, Time and Structured Data List, Record and Table Structures let, source, in statements @ M Lang Power Query Functions, Parameters Invoke Functions, Execution Results Get Data, Table Creations and Edit Merge and Append Transformations Join Kinds, Advanced Editor, Apply ETL Operations with Power Query	Ch 13 : POWER BI CLOUD - 1 Power BI Service Architecture Power BI Cloud Components, Use App Workspaces, Report Publish Reports & Related Datasets Cloud Creating New Reports in Cloud Report Publish and Report Uploads Dashboards Creation and Usage Adding Tiles to Dashboards Pining Visuals and Report Pages Visual Pin Actions in Dashboards LIVE Page Interaction in Dashboard Adding Media: Images, Custom Links Adding Chs and Embed Links API Data Sources, Streaming Data Streaming Dataset Tiles (REST API)
Ch 2: BASIC REPORT DESIGN Power BI Desktop Installation Data Sources & Visual Types Canvas, Visualizations and Fields Get Data and Memory Tables In-Memory xvelocity Database Table and Tree Map Visuals Format Button and Data Labels Legend, Category and Grid PBIX and PBIT File Formats Visual Interaction, Data Points Disabling Visual Interactions Edit Interactions - Format Options SPOTLIGHT & FOCUSMODE CSV and PDF Exports. Tooltips Power BI EcoSystem, Architecture	Ch 8 : POWER QUERY LEVEL 2 Query Duplicate, Query Reference Group By and Advanced Options Aggregations with Power Query Transpose, Header Row Promotion Reverse Rows and Row Count Data Type Changes & Detection Replace Columns: Text, NonText Replace Nulls: Fill Up, Fill Down PIVOT, UNPIVOT Transformations Move Column and Split Column Extract, Format and Numbers Date & Time Transformations Deriving Year, Quarter, Month, Day Add Column : Query Expressions Query Step Inserts and Step Edits	Ch 14 : POWER BI CLOUD - 2 Dashboards Actions,Report Actions DataSet Actions: Create Report Share, Metrics and Exports Mobile View & Dashboard Themes Q & A [Cortana] and Pin Visuals Export, Subscribe, Subscribe Favorite, Insights, Embed Code Featured Dashboards and Refresh Gateways Configuration, PBI Service Gateway Types, Cloud Connections Gateway Clusters, Add Data Sources Data Refresh : Manual, Automatic PBIEngw Service, ODG Logs, Audits DataFlows, Power Query Expressions Adding Entities and JSON Files
Ch 3 : Visual Sync, Grouping Slicer Visual : Real-time Usage Orientation, Selection Properties Single & Multi Select, CTRL Options Slicer : Number, Text and Date Data Slicer List and Slicer Dropdowns Visual Sync Limitations with Slicer Disabling Slicers,Clear Selections Grouping : Real-time Use, Examples List Grouping and Binning Options Grouping Static / Fixed Data Values Grouping Dynamic / Changing Data Bin Size and Bin Limits (Max, Min) Bin Count and Grouping Options Grouping Binned Data, Classification	Ch 9 : POWER QUERY LEVEL 3 Creating Parameters in Power Query Parameter Data Types, Default Lists Static/Dynamic Lists For Parameters Removing Columns and Duplicates Convert Tables to List Queries Linking Parameters to Queries Testing Parameters and PBI Canvas Multi-Valued Parameter Lists Creating Lists in Power Query Converting Lists to Table Data Advanced Edits and Parameters Data Type Conversions, Expressions Columns From Examples, Indexes Conditional Columns, Expressions	Ch 15 : EXCEL & RLS Import and Upload Options in Excel Excel Workbooks and Dashboards Datasets in Excel and Dashboards Using Excel Analyzer in Power BI Using Excel Publisher in PBI Cloud Excel Workbooks, PINS in Power BI Excel ODC Connections, Power Pivot Row Level Security (RLS) with DAX Need for RLS in Power BI Cloud Data Modeling in Power BI Desktop DAX Roles Creation and Testing Adding Power BI Users to Roles Custom Visualizations in Cloud Histogram,Gantt Chart,Infographics
Ch 4 : Hierarchies, Filters Creating Hierarchies in Power BI Independent Drill-Down Options Dependant Drill-Down Options Conditional Drilldowns, Data Points Drill Up Buttons and Operations Expand & Show Next Level Options Dynamic Data Drills Limitations Show Data and See Records Filters : Types and Usage in Real-time Visual Filter, Page Filter, Report Filter Basic, Advanced and TOP N Filters Category and Summary Level Filters DrillThru Filters, Drill Thru Reports Keep All Filters" Options in DrillThru CrossReport Filters, Include, Exclude	Ch 10 : DAX Functions - Level 1 DAX : Importance in Real-time Real-world usage of Excel, DAX DAX Architecture, Entity Sets DAX Data Types, Syntax Rules DAX Measures and Calculations ROW Context and Filter Context DAX Operators, Special Characters DAX Functions, Types in Real-time Vertipaq Engine, DAX Cheat Sheet Creating, Using Measures with DAX Creating, Using Columns with DAX Quick Measures and Summaries Validation Errors, Runtime Errors SUM, AVERAGEX, KEEPFILTERS Dynamic Expressions, IF in DAX	Ch 16 : Report Server, RDL Need for Report Server in PROD Install, Configure Report Server Report Server DB, Temp Database Webservice URL, Webportal URL Creating Hybrid Cloud with Power BI Using Power BI DesktopRS Uploading Interactive Reports Report Builder For Report Server Report Builder For Power BI Cloud Designing Paginated Reports (RDL) Deploy to Power BI Report Server Data Source Connections, Report Power BI Report Server to Cloud Tenant IDs Generation and Use Mobile Report Publisher, Usage
Ch 5 : Bookmarks, Azure, Modeling Drill-thru Filters, Page Navigations Bookmarks : Real-time Usage Bookmarks for Visual Filters Bookmarks for Page Navigations Selection Pane with Bookmarks Buttons, Images with Actions Buttons, Actions and Text URLs Bookmarks View & Selection Pane OLTP Databases, Big Data Sources Azure Database Access, Reports Import & Direct Query with Power BI SQL Queries and Enter Data Data Modeling : Currency, Relations Summary, Format, Synonyms Web View & Mobile View in PBI	Ch 11 : DAX Functions - Level 2 Data Modeling Options in DAX Detecting Relations for DAX Using Calculated Columns in DAX Using Aggregated Measures in DAX Working with Facts & Measures Modeling : Missing Relations Modeling : Relation Management CALCULATE Function Conditions CALCULATE & ALL Member Scope RELATED & COUNTROWS in DAX Entity Sets and Slicing in DAX Dynamic Expressions, RETURN Date, Time and Text Functions Logical, Mathematical Functions Running Total & EARLIER Function	Ch 17: MSBI Integrations Power BI with SQL Server Source Power BI with SQL Data Warehouse Power BI with SSAS OLAP Server Power BI with Azure SQL DB Source Power BI with Azure SQL Warehouse Power BI with Azure Analysis Server Power BI with SSRS (RDL) Reports Power BI Report Builder Tool Installation & Configuration Paginated Reports Design, Use Data Sources, Datasets, RDL Report Publish (RDL) to Cloud Report Verifications, Data Sync Interactive Vs Paginated Reports Creating, Managing Alerts in Cloud
Ch 6 : Visualization Properties Stacked Charts and Clustered Charts Line Charts, Area Charts, Bar Charts 100% Stacked Bar & Column Charts Map Visuals: Tree, Filled, Bubble Cards, Funnel, Table, Matrix Scatter Chart : Play Axis, Labels Series Clusters & Selections Waterfall Chart and ArcGIS Maps Infographics, Icons and Labels Color Saturation, Sentiment Colors Column Series, Column Axis in Lines Join Types : Round, Bevel, Miter Shapes, Markers, Axis, Plot Area Display Units,Data Colors,Shapes Series, Custom Series and Legends	Ch 12 : DAX FUNCTIONS Level 3 1:1, 1:M and M:1 Relations Connection with CSV, MS Access AVERAGEX and AVERAGE in DAX KEEPFILTERS and CALCUALTE COUNTROWS, RELATED, DIVIDE PARALLELPERIOD, DATEDADD CALCULATE & PREVIOUSMONTH USERELATIONSHIP, DAX Variables TOTALYTD , TOTALQTD DIVIDE, CALCULATE, Conditions IF..ELSE..THEN Statement SELECTEDVALUE, FORMAT SUM, DATEDIFF Examples in DAX TODAY, DATE, DAY with DAX Time Intelligence Functions - DAX	Ch 18 : REAL-TIME PROJECT Project Requirement Analysis Implementing SDLC Phases Requirement Gathering, FSA Phase 1: PBIX Report Design Visualizations, Properties Analytics and Formating Phase 2: Data Modeling, Power Query Dynamic Connections, Azure DB Parameters and M Lang Scripts Phase 3: DAX Requriements, Analysis Cloud and Report Server Project FAQs and Solutions