- Published on
A local PySpark experiment that reproduces a trusted ecommerce revenue query and makes DataFrames, laziness, jobs, stages, tasks, exchanges, and physical plans observable.
Data-platform
All Posts
- next-js (6)
- code (5)
- data-engineering (5)
- guide (5)
- spark (4)
- pyspark (4)
- java (3)
- data-engineer (3)
- stream (3)
data-platform (3)
- distributed-systems (3)
- tailwind (3)
- data-quality (2)
- parquet (2)
- feature (2)
- markdown (1)
- features (1)
- sql (1)
- duckdb (1)
- streamlit (1)
- math (1)
- ols (1)
- github (1)
- holiday (1)
- canada (1)
- images (1)
- git (1)
- productivity (1)
- lazygit (1)
- multi-author (1)
- Published on
A concrete Spark experiment comparing unpartitioned and order-date-partitioned Parquet layouts, with physical files, directory pruning, pushed filters, task counts, and small-files economics.- Published on
A local DuckDB experiment that turns ecommerce CSV files into validated Silver data, quarantine records, and trusted Gold revenue while making grain, duplicates, nulls, and idempotency observable.