Data Engineering · September 2026
Real-Time Transaction Streaming Pipeline
Streaming card transactions from Kafka to cloud storage, with data quality checks and lag monitoring.
- Automated tests
- 11
- Data quality rules
- 5
Overview
Containerized pipeline where a Python producer replays the Kaggle credit card dataset into Kafka, and Spark Structured Streaming validates each record against five data quality rules. Valid rows are written as Parquet to Supabase Storage through idempotent per-batch uploads, invalid rows are quarantined with reason codes instead of dropped, and an external monitor compares Kafka offsets with Spark's checkpoint to report pipeline health.
Tech stack
- Apache Kafka
- Apache Spark
- Docker
- Supabase
- Python
- pytest