- Published on
A verified PySpark join experiment showing why join cardinality matters, when Spark broadcasts a small dimension, and how disabling broadcast exposes hash partitioning, exchanges, sorting, and shuffle.
Data-quality
All Posts
- next-js (6)
- code (5)
- data-engineering (5)
- guide (5)
- spark (4)
- pyspark (4)
- java (3)
- data-engineer (3)
- stream (3)
- data-platform (3)
- distributed-systems (3)
- tailwind (3)
data-quality (2)
- parquet (2)
- feature (2)
- markdown (1)
- features (1)
- sql (1)
- duckdb (1)
- streamlit (1)
- math (1)
- ols (1)
- github (1)
- holiday (1)
- canada (1)
- images (1)
- git (1)
- productivity (1)
- lazygit (1)
- multi-author (1)
- Published on
A local DuckDB experiment that turns ecommerce CSV files into validated Silver data, quarantine records, and trusted Gold revenue while making grain, duplicates, nulls, and idempotency observable.