Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• An overview of the Big Data landscape and Spark’s role in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs, and determining when to apply each method
• Establishing and configuring SparkSession, along with an understanding of basic application settings

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats, including CSV, JSON, Parquet, and Delta
• Managing PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Executing advanced operations like window functions, timestamp handling, and managing nested data
• Implementing data quality checks and crafting reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Basics for Spark Users

• A conceptual look at how Docker images and Kubernetes pods relate to Spark driver and executor processes, tailored for cluster users rather than administrators
• Insights into Spark job execution on Kubernetes: pod scheduling, resource requests and limits, and how session settings align with these resources
• Clarifying what you can control via Spark session parameters versus what is managed by the platform team through cluster provisioning, and why this distinction is critical for tuning
• Practical Exercise: Examine a live Spark-on-Kubernetes job in a sandbox cluster and identify the driver and executor pods

Module 4: Deep Dive into Spark Session Configuration on Kubernetes

• Analysing executor count versus executor size: the trade-offs between memory and cores, and how to evaluate them
• Allocating memory for drivers and executors, including memory overhead, and translating these into pod resource specifications
• Dynamic allocation: its behaviour on Kubernetes, scenarios where it saves resources, and where it may not
• Practical Exercise: Execute the same job with varying executor and core configurations, comparing runtime and resource utilisation

Module 5: Scaling Behaviour & Cost Optimisation

• Understanding how adding or removing nodes impacts job behaviour, completion times, and resource consumption
• Comparing many small executors against fewer large ones: the balance between performance and cost
• Shuffle dynamics and partition tuning, including estimating the cost implications of different configurations
• Practical Exercise: Expand the sandbox cluster from five to ten nodes and monitor the impact on job completion time and resource usage

Module 6: Efficient Data Ingestion & Partitioning

• The small-files problem: why sources split into numerous 1–5 MB Parquet files reduce performance and disrupt partitioning
• Strategies for repartitioning and coalescing data
• Managing partition sizes during both reading and writing operations
• Efficiently writing Parquet files to prevent the recurrence of small-file issues downstream
• Practical Exercise: Load a dataset comprising many small Parquet partitions, test various repartitioning strategies, and compare performance before and after optimisation

Module 7: Pandas Memory Management & Diagnosing Failures

• Identifying the root causes of out-of-memory errors in Pandas and recognising their symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Leveraging chunked processing and dtype optimisation in resource-constrained environments
• Practical Exercise: Recreate a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation techniques

Module 8: Polars as a Complementary Tool

• Positioning Polars alongside Pandas: examining performance characteristics, lazy evaluation, and memory behaviour
• Understanding where Polars fits within modern cloud data stacks and migration roadmaps, including AWS environments, alongside PySpark and Pandas
• Practical Exercise: Refactor a Pandas-intensive transformation using Polars and compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles across a realistic ETL pipeline
• Considering optimisation needs specific to machine-learning workloads running on the same cluster
• Implementing a practical tuning workflow to systematically diagnose cost and performance issues
• Practical Exercise: Complete an end-to-end mini-project involving data loading, transformation, and a basic model-training step, with participants independently tuning the Spark configuration