Batch Processing with Apache Spark

Week 6 of Data Engineering Zoomcamp by @DataTalksClub complete Just finished Module 6 - Batch Processing with Spark. Learned how to: ✅ Set up PySpark and create Spark sessions ✅ Read and process Parquet files at scale ✅ Repartition data for optimal performance ✅ Analyze millions of taxi trips with DataFrames ✅ Use Spark UI for monitoring jobs Processing 4M+ taxi trips with Spark - distributed computing is powerful Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/pyspark-homework Following along with this amazing free course - who else is learning data engineering? You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/