Purchasing the Spark SQL and PySpark using Python 3 Course

Introduction to Udemy Course Landing Page

Introduction to Spark SQL and PySpark 3 using Python 3

Curriculum for Spark SQL and Pyspark 3 using Python 3

Adding Notes to Course Lectures

Using Course Sidebar to move between lectures

Overview of Udemy Course or Video Player

Overview of Support to ITVersity courses on Udemy

Resources for Spark SQL and Pyspark 3 using Python 3

Material for Spark SQL and PySpark 3 using Python 3

Become Part of ITVersity Data Engineering Community

Best Practices to get ITVersity Support using Udemy

Udemy for Business Customers - Important Information for about labs for practice

Rate and Leave Feedback - Spark SQL and PySpark 3 using Python 3

Setup Development Environment using VS Code Remote Development Extension Pack

Review Data Sets Provided as part of Gateway Nodes of Hadoop and Spark Cluster

Validate Hive on Hadoop and Spark Multinode Cluster

Review Hadoop HDFS and YARN Property Files using Visual Studio Code Editor

Validate HDFS on Multi Node Hadoop and Spark Cluster from Gateway Node

Review Hadoop HDFS and YARN Property Files on Hadoop and Spark Cluster

Review Hive Property Files on Multinode Hadoop and Spark Cluster

Review Spark 2 Property Files and Important Properties

Validate Spark Shell CLI using Spark 2

Validate Pyspark CLI using Spark 2

Validate Spark SQL CLI using Spark 2

Review Spark 3 Property Files and Important Properties

Validate Spark Shell CLI using Spark 3

Validate Pyspark CLI using Spark 3

Validate Spark SQL CLI using Spark 3

Prerequisites for Single Node Hadoop and Spark Cluster on Windows

Overview of Windows System Configuration

Setup Ubuntu on Windows 11 using wsl

Setup and Validate Ubuntu VM on Windows using wsl

Overview of Docker Desktop on Windows 11

Install Docker Desktop on Windows 11 using wsl2

Validate Docker Commands using Windows Powershell as well as wsl Ubuntu

Setup Visual Studio Code IDE on Windows

Clone GitHub Repository for Pyspark Course using Visual Studio Code

Install Visual Studio Code Extension for Remote Development

Launching Terminal using Visual Studio Code and WSL

Review Docker Compose File to setup Hadoop and Spark Lab

Review the resource utilization of Windows for Hadoop and Spark Lab

Start Hadoop and Spark Lab along with Jupyter Lab on Windows 11

Review Docker Desktop for Hadoop and Spark Lab using Docker

Validate Hadoop and Spark setup using Docker on Windows

Overview of Docker Compose Commands to manage Hadoop and Spark Lab

Getting Started with AWS Cloud9

Warming up with AWS Cloud9 IDE

Creating AWS Cloud9 Environment

Review Operating System Details on AWS Cloud9

Opening ports for AWS Cloud9 Instance

Setup Docker Compose on AWS Cloud9 Instance

Associating Elastic IPs to AWS Cloud9 Instance

Increase EBS Volume Size of AWS Cloud9 Instance

Clone GitHub Repository on AWS Cloud9 for the Course Material

Review Docker Compose File to setup Hadoop and Spark Lab

Overview of Docker Compose Commands to manage Hadoop and Spark Lab

Start Hadoop and Spark Lab along with Jupyter Lab on Windows 11

Validate Hadoop and Spark setup using Docker

Introduction and Setting up Python

Functions in Python

Python Collections

Basic Programming Constructs

Map Reduce operations on Python Collections

Setting up Data Sets for Basic IO Operations

Basic IO operations and processing data using Collections

Getting help or usage

Listing HDFS Files

Copying files from HDFS to local

Managing HDFS Directories

Previewing Data in HDFS File

Copying files from local to HDFS

HDFS Block Size

Getting HDFS Storage Usage

Getting File Metadata

HDFS Replication Factor

Using HDFS Stat Commands

HDFS File Permissions

Overriding Properties

Introduction

Review of Setup Steps for Spark Environment

Spark Modules

Apache Spark Official Documentation (Very Important)

Using ITVersity labs

Quick Review of Spark APIs

Spark Data Structures - RDDs and Data Frames

Develop Simple Application

Create Data Frames from Text Files

Apache Spark - Framework

Create Data Frames from Hive Tables

Getting Started - Overview

Overview of Spark Documentation

Launching and using Spark SQL CLI

Running OS Commands using Spark SQL

Understanding Spark Metastore Warehouse Directory

Overview of Spark SQL Properties

Managing Spark Metastore Tables using Spark SQL

Retrieve Metadata of Spark Metastore Tables using Spark SQL Describe Command

Managing Spark Metastore Databases using Spark SQL

Role of Spark Metastore or Hive Metastore

Exercise - Getting Started with Spark SQL

Basic Transformations using Spark SQL - Introduction

Spark SQL - Overview

Define Problem Statement

Filtering Data using Spark SQL Where Clause

Prepare Spark Metastore Tables for Basic Transformations using Spark SQL

Projecting Data using Spark SQL Select Clause

Joining Tables using Spark SQL - Outer

Sorting Data using Order By in Spark SQL

Aggregating Data using Group By in Spark SQL

Conclusion - Final Solution for the problem statement using Spark SQL

Joining Tables using Spark SQL - Inner

Introduction to Basic DDL and DML in Spark SQL

Adding Comments to Spark Metastore Tables using Spark SQL

Loading Data from Local File System Into Tables using Spark SQL Load Statement

Create Spark Metastore Tables using Spark SQL Create Statement

Overview of Data Types used in Spark Metastore Tables

Creating External Spark Metastore Tables using Spark SQL

Loading Data from HDFS Folders Into Tables using Spark SQL Load Statement

Difference between Managed and External Spark Metastore Tables

Difference between Load with Append and Overwrite using Spark SQL Load Statement

Truncating Spark Metastore Tables

Drop Spark Metastore Tables and Databases using Spark SQL

Overview of File Formats used in Spark Metastore Tables

Exercise - Managed Spark Metastore Tables

Introduction to DML and Partitioning using Spark SQL on Spark Metastore Tables

Introduction to Partitioning of Spark Metastore Tables using Spark SQL

Difference between Load and Insert to get data into Spark Metastore Tables

Creating Spark Metastore Tables using Parquet File Format

Creating Spark Metastore Partitioned Tables using Spark SQL

Inserting Data using Stage Table leveraging Spark SQL

Adding Partitions to Spark Metastore Tables using Spark SQL

Inserting Data into Spark Metastore Partitions using Spark SQL Insert Statement

Exercise - Partitioned Tables using Spark SQL

Loading Data into Spark Metastore Partitioned Tables using Spark SQL

Using Dynamic Partition Mode while inserting into Spark Partitioned Tables

Introduction - Overview of Spark SQL Pre-defined Functions

Overview of Spark SQL Pre-defined Functions

Validating Spark SQL Functions

Overview of Numeric Functions in Spark SQL

String Manipulation using Spark SQL Functions

Data Type Conversion using Spark SQL

Dealing with Nulls using Spark SQL

Date Manipulation using Spark SQL Functions

Using CASE and WHEN in Spark SQL Queries

Query Example - Word Count using Spark SQL

Introduction to Windowing Functions in Spark SQL

Prepare HR Database for Windowing Functions in Spark SQL

Overview of Windowing Functions using Spark SQL

Aggregations using Spark SQL Windowing Functions

Getting first and last values using Spark SQL Windowing Functions

Ranking using Spark SQL Windowing Functions - rank, dense rank and row number

Using LEAD or LAG in Spark SQL Windowing Functions

Order of execution of Spark SQL Queries

Overview of Subqueries in Spark SQL

Filtering Window Function Results using Spark SQL

Starting Spark Context - pyspark

Understanding airlines data

Overview of Spark Read APIs

Inferring Schema using Spark Data Frame APIs

Overview of Data Frame APIs

Previewing Airlines Data using Spark Data Frame APIs

Overview of Functions on Spark Data Frames

Overview of Spark Write APIs

Create Dummy Data Frame to explore Functions on Data Frame Columns

Overview of Predefined Functions on Spark Data Frame Columns

Categories of Predefined Functions used on Spark Data Frame Columns

Common String Manipulation Functions for Spark Data Frame Columns

Special Functions for Spark Data Frame Columns - col and lit

Extracting Strings using substring from Spark Data Frame Columns

Extracting Strings using split from Spark Data Frame Columns

Padding Characters around Strings in Spark Data Frame Columns

Trimming Characters from Strings in Spark Data Frame Columns

Date and Time Manipulation Functions for Spark Data Frame Columns

Date and Time Arithmetic on Spark Data Frame Columns

Using Date and Time Trunc Functions on Spark Data Frame Columns

Date and Time Extract Functions for Spark Data Frame Columns

Using date format Function on Spark Data Frame Columns

Using to date and to timestamp on Spark Data Frame Columns

Using CASE and WHEN on Spark Data Frame Columns

Dealing with Nulls in Spark Data Frame Columns

Dealing with Unix Timestamp in Spark Data Frame Columns

Overview of Basic Transformations on Spark Data Frames

Spark Data Frames for basic transformations

Basic Filtering of Data or rows using where from Spark Data Frames

Filtering Example using dates on Spark Data Frames

Boolean Operators while filtering from Spark Data Frames

Using IN Operator or isin Function while filtering from Spark Data Frames

Using LIKE Operator or like Function while filtering from Spark Data Frames

Using BETWEEN Operator while filtering from Spark Data Frames

Dealing with Nulls while Filtering from Spark Data Frames

Total Aggregations on Spark Data Frames

Aggregate data using groupBy from Spark Data Frames

Aggregate data using rollup on Spark Data Frames

Aggregate data using cube on Spark Data Frames

Solution - Problem 2 - Get Total Aggregations By FlightDate

Solution - Problem 1 - Get Total Aggregations

Overview of Sorting Spark Data Frames

Prepare Datasets for Joining Spark Data Frames

Problem Statements for Joining Spark Data Frames

Overview of Joins on Spark Data Frames

Analyze Datasets for Joining Spark Data Frames

Using Inner Joins on Spark Data Frames

Left or Right Outer Join on Spark Data Frames

Solution - Get Flight Count Per US State using Spark Data Frame APIs

Solution - Get Flight Count Per US Airport using Spark Data Frame APIs

Solution - Get Dormant US Airports using Spark Data Frame APIs

Solution - Get Count of Flights per Airport without master data

Solution - Get Count of Flights without master data using Spark Data Frame APIs

Solution - Get Origins without master data using Spark Data Frame APIs

Solution - Get Daily Revenue using Spark Data Frame APIs

Solution - Get Daily Revenue rolled up till Yearly using Spark Data Frame APIs

Exploring Spark Catalog

Overview of APIs to deal with Spark Metastore

Creating Spark Metastore Tables using catalog

Inferring Schema while creating Spark Metastore Tables using Spark Catalog

Define Schema for Spark Metastore Tables using StructType

Read and Process data from Spark Metastore Tables using Data Frame APIs

Inserting into Existing Spark Metastore Tables using Spark Data Frame APIs

Create Spark Metastore Partitioned Tables using Data Frame APIs

Saving as Spark Metastore Partitioned Table using Data Frame APIs

Creating Temporary Views on top of Spark Data Frames

Using Spark SQL against Temporary Views on Spark Data Frames

Introduction to Getting Started with Semi Structured Data using Spark

Overview of ARRAY Type in Spark Metastore Table

Create Spark Metastore Table with Special Data Types

Overview of MAP and STRUCT Type in Spark Metastore Table

Insert Data into Spark Metastore Table with Special Type Columns

Create Spark Data Frame with Special Data Types

Create Spark Data Frame with Special Types using Python List

Review Data in the JSON File with Special Data Types

Setup JSON Data Set to explore Spark APIs on Special Data Type Columns

Insert Spark Data Frame with Special Types into Spark Metastore Table

Read JSON Data with Special Types into Spark Data Frame

Flatten Array Fields in Spark Data Frames using explode and explode outer

Concatenate Array Values into Delimited String using Spark APIs

Get Size or Length of Array Type Columns in Spark Data Frame

Convert Delimited Strings from Spark Data Frame Columns to Arrays

Build Arrays from Flattened Rows of Spark Data Frame

Setup Data Sets to Build Arrays using Spark

Read JSON Data into Spark Data Frame and Review Aggregate Operations

Getting Started with Spark Data Frames with Struct Columns

Concatenate Struct Column Values in Spark Data Frame

Filter Data on Struct Column Attributes in Spark Data Frame

Create Spark Data Frame using Map Type Column

Project Map Values as Columns using Spark Data Frame APIs

Conclusion of Getting Started with Semi Structured Data using Spark

Introduction to Process Semi Structured Data using Spark Data Frame APIs

Setup JSON Data Sets in HDFS using HDFS Command

Review the Data Sets to generate denormalized JSON Data using Spark

Create Spark Data Frames using Data Frame APIs

Generate Struct Field for Order Details using Spark

Generate Array of Struct Field for Order Details using Spark

Join Data Sets to generate denormalized JSON Data using Spark

Join Orders and Order Items using Spark Data Frame APIs

Denormalize Join Results using Spark Data Frame APIs

Publish JSON Files for downstream applications

Read Denormalized Data into Spark Data Frame

Write Denormalized Customer Details to JSON Files using Spark

Filter Denormalized Data Frame using Spark APIs

Flatten Semi Structured Data or Denormalized Data using Spark

Perform Aggregations on Denormalized Data Frame using Spark

Conclusion of Processing Semi Structured Data using Spark Data Frame APIs

Compute Monthly Customer Revenue using Spark on Denormalized Data

Setup Virtual Environment and Install Pyspark

Passing Run Time Arguments

Getting Started with Pycharm

Accessing OS Environment Variables

Getting Started with Spark

Setup Sample Data

Create Function for Spark Session

Read data from files

Validating Writing Data to Files

Write data to files

Process data using Spark APIs

Productionizing the Code

Setting up Data for Production Validation

Detailed Validation of the Application

Running the application using YARN

Deploying and Monitoring Spark Applications - Introduction

Setup EMR Cluster with Hadoop and Spark

Overview of Types of Spark Cluster Managers

Overall Capacity of Big Data Cluster with Hadoop and Spark

Understanding YARN Capacity of an Enterprise Cluster

Overview of Hadoop HDFS and YARN Setup on Multi-node Cluster

Develop Word Count Application

Review Deployment Process of Spark Application

Overview of Spark Setup on top of Hadoop

Setup Data Set for Word Count application

Switch between Python Versions to run Spark Applications or launch Pyspark CLI

Review Spark Configuration Properties at Run Time

Overview of Spark Submit Command

Switch between Pyspark Versions to run Spark Applications or launch Pyspark CLI

Run Spark Application and review default executors

Develop Shell Script to run Spark Application

Overview of Spark History Server UI

Setup SSH Proxy to access Spark Application logs - Introduction

Overview of Private and Public ips of servers in the cluster

Overview of SSH Proxy

Proxy using sshuttle on Mac or Linux

Setup sshuttle on Mac or Linux

Steps to setup SSH Proxy on Windows to access Spark Application Logs

Accessing Spark Application logs via SSH Proxy using sshuttle on Mac or Linux

Setup PuTTY and PuTTYgen on Windows

Side effects of using SSH Proxy to access Spark Application Logs

Quick Tour of PuTTY on Windows

Configure Passwordless Login using PuTTYGen Keys on Windows

Configure Tunnel to Gateway Node using PuTTY on Windows for SSH Proxy

Run Spark Application on Gateway Node using PuTTY

Setup Proxy on Windows and validate using Microsoft Edge browser

Understanding Proxying Network Traffic overcoming Windows Caveats

Overview of performing tasks related to Spark Applications using Mac

Access Spark Application logs using SSH Proxy

Update Hosts file for worker nodes using private ips

Deployment Modes of Spark Applications - Introduction

Launch Pyspark using local mode

Default Execution Master Type for Spark Applications

Overview of Spark CLI Commands such as Pyspark

Running Spark Applications using Local Mode

Overview of submitting spark application using cluster deployment mode

Overview of submitting spark application using client deployment mode

Review the default logging while submitting Spark Applications

Accessing Local Files using Spark CLI or Spark Applications

Submit Spark Application using client mode with log level info

Changing Spark Application Log Level using custom log4j properties

Submit Spark Application using cluster mode with log level info

Submit Spark Applications using SPARK CONF DIR with custom properties files

Submit Spark Applications using Properties File

Passing Application Properties Files and External Dependencies - Introduction

Steps to pass application properties using JSON

Setup Working Directory to pass application properties using JSON

Explore APIs to process JSON Data using Pyspark

Build the JSON with Application Properties

Refactor the Spark Application Code to use properties from JSON

Pass Application Properties to Spark Application using local files in client mod

Pass Application Properties to Spark Application using local files in cluster mo

Create required YAML File to externalize application properties

Pass Application Properties to Spark Application using HDFS files

Install PyYAML into specific folder and build zip

Steps to pass external Python Libraries using pyfiles

Refactor the Spark Application Code to use properties from YAML

Explore APIs to process YAML Data using Pyspark

Pass External Dependencies to Spark Application using local files in client mode

Pass External Dependencies to Spark Apps using local files in cluster mode

Pass External Dependencies to Spark Application using HDFS files