What is Data Engineering: Definition and key concepts: Different Data Engineering Technologies: Tools and platforms used in data processing: Importance of Python in Data Engineering: Scripting, automation, and data manipulation: Importance of SQL in Data Engineering: Querying, managing, and transforming data: Importance of Linux in Data Engineering: Command-line usage, scripting, and system management: Cloud Computing Introduction: Overview of cloud platforms and their relevance to data engineering.

SQL Introduction: Provisioning MySQL using RDS: Create, Alter, Drop Database: Create, Alter, Insert tables: Key Constraints: Select Queries: Joins.

Python Introduction: Python Programming Introduction: Interactive Mode Development & Script Mode Development: Python Installation on Windows Operating System & First Python Program using Python Shell: Anaconda Python Installation and First Python Application using Jupyter Notebook & Spider IDE: Python Editors & IDE Software’s Installation & First Python Application using Editors & IDE’s – Notepad++, PyCharm IDE, VSCode

AWS Introduction: AWS Global Infrastructure (Regions & Availability Zones): AWS Account Creation: AWS Free-tier: AWS EC2: AWS IAM

PySpark Foundation: PySpark Core Programming – RDD Programming, Transformations & Actions: PySpark SQL – DataFrames, Tables, DSL & Native SQL: PySpark Streaming Programming: PySpark RDS Integration: PySpark S3 Integration

Introduction to S3: Overview and use cases; S3 Buckets and Objects: Creating and managing buckets and objects; S3 Versioning: Enabling and managing versioning; S3 Security and Bucket Policies: Setting up bucket policies and access control; Static Website Hosting: Hosting a static website on S3; Cross-Region and Same-Region Replication: Configuring replication for data redundancy; S3 Storage Classes: Overview of storage classes and cost optimization; Configuring S3 Lifecycle Rules: Automating transitions and expiration

AWS S3 Introduction: AWS S3 Bucket using AWS Web Console: Create AWS S3 Bucket: Setup Data Set locally to upload into AWS S3: Adding AWS S3 Buckets and Objects using AWS Web Console: Version Control of AWS S3 Objects or Files: AWS S3 Cross-Region Replication for fault tolerance: Overview of AWS S3 Storage Classes or Storage Tiers: Overview of Glacier in AWS S3: Managing AWS S3 buckets and objects using AWS CLI: AWS S3 Integration with PySpark

Hello World using AWS Lambda: Setup Project for local development: Deploy Project to AWS Lambda console: Develop download functionality using requests: Using 3rd party libraries in AWS Lambda: Validating S3 access for local development: Develop upload functionality to S3: Validating using AWS Lambda Console: Run using AWS Lambda Console: Validating files incrementally: Reading and Writing Bookmark using S3: Maintaining Bookmark using S3: Review the incremental upload logic: Deploying Lambda function: Schedule Lambda Function using AWS Event Bridge

AWS Glue Components: Create Crawler and Catalog Table; Create and Run the Glue Job; Create and Run Glue Trigger; Create Glue Workflow; Run Glue Workflow and Validate; Setup Spark History Server on AWS; Build Glue Spark UI Container; Update IAM Policy Permissions; Start Glue Spark UI Container; Steps for Creating Catalog Tables; Create Glue Catalog Database; Crawling Multiple Folders; Managing Glue Catalog using AWS CLI

Amazon Athena Introduction: Glue Catalog Databases and Tables; Access Glue Catalog Databases and Tables using Athena Query Editor; Create Database and Table using Athena; Populate Data into Table using Athena; Using CTAS to create tables using Athena; Amazon Athena Architecture; Amazon Athena Resources and relationship with Hive; Create Partitioned Table using Athena; Develop Query for Partitioned Column; Insert into Partitioned Tables using Athena; Validate Data Partitioning using Athena; Drop Athena Tables and Delete Data Files; Drop Partitioned Table using Athena; Data Partitioning in Athena using CTAS

Building Streaming Pipeline using Kinesis: Rotating Logs; Setup Kinesis Firehose Agent; Create Kinesis Firehose Delivery Stream; Planning the Pipeline; Create IAM Group and User; Granting Permissions to IAM User using Policy; Configure Kinesis Firehose Agent; Start and Validate Agent; Building Simple Streaming Pipeline by Integrating PySpark

Amazon Redshift – Introduction: Create Redshift Cluster using Free Trial; Connecting to Database using Redshift Query Editor; Get list of tables querying information schema; Run Queries against Redshift Tables using Query Editor; Create Redshift Table; CRUD Operations; Insert Data into Redshift Tables; Update Data in Redshift Tables; Delete Data from Redshift tables; Redshift Saved Queries using Query Editor

Deleting Redshift Cluster: Copy Data from s3 to Redshift – Introduction; Setup Data in s3 for Redshift Copy; Copy Database and Table for Redshift Copy Command; Run Copy Command to copy data from s3 to Redshift Table; Copy JSON Data from s3 to Redshift table using IAM Role; Redshift Architecture; Create multi-node Redshift Cluster; Connect to Redshift Cluster using Query Editor; Create Redshift Database; Create Redshift Database User; Create Redshift Database Schema; Integrating PySpark with Redshift

End to End Project Work: Resume Guidance; Interview FAQs; Mock Interview (On-Request)