- Published on
A local PySpark experiment that reproduces a trusted ecommerce revenue query and makes DataFrames, laziness, jobs, stages, tasks, exchanges, and physical plans observable.
Pyspark
All Posts
- next-js (6)
- code (5)
- guide (5)
- data-engineering (4)
- java (3)
- data-engineer (3)
- stream (3)
- data-platform (3)
- spark (3)
pyspark (3)
- tailwind (3)
- data-quality (2)
- distributed-systems (2)
- feature (2)
- markdown (1)
- features (1)
- sql (1)
- duckdb (1)
- parquet (1)
- math (1)
- ols (1)
- github (1)
- holiday (1)
- canada (1)
- images (1)
- git (1)
- productivity (1)
- lazygit (1)
- multi-author (1)
- Published on
A concrete Spark experiment comparing unpartitioned and order-date-partitioned Parquet layouts, with physical files, directory pruning, pushed filters, task counts, and small-files economics.- Published on
A verified PySpark join experiment showing why join cardinality matters, when Spark broadcasts a small dimension, and how disabling broadcast exposes hash partitioning, exchanges, sorting, and shuffle.