data-engineering-projects

Data Warehouse Modelling

1. Sales Warehouse: Fundamental implementation of a Star Schema for sales data.

   • Focus: Star Schema, Surrogate Keys, B-Tree Indexes, Python + SQLite.

2. Subscription Warehouse: Tracking user history with SCD Type 2 and Snowflake Schema.

   • Focus: SCD (Slowly Changing Dimensions) Type 2, Snowflake Schema, DuckDB (Columnar Storage).

3. Order Fulfillment Warehouse: Tracking e-commerce orders through a multi-stage pipeline.

   • Focus: Accumulating Snapshot Fact Table, Role-Playing Dimensions, Degenerate Dimensions, Junk Dimensions, DuckDB.

4. Inventory Snapshot Warehouse: Monitoring stock levels and category hierarchies over time.

   • Focus: Periodic Snapshot Fact Table, Parent-Child Hierarchies, Recursive CTEs, Semi-Additive Measures.

5. Healthcare Bridge Warehouse: Handling Many-to-Many relationships and semantic abstractions.

   • Focus: Bridging Tables, Multi-valued Dimensions, Semantic Layer (Views), Revenue Allocation.

Data Engineering GCP

Batch Ingestion and Processing

Project	Summary
Loading Public Data into BigQuery	Downloads a public dataset with pandas and loads it into a Terraform-provisioned BigQuery table.
Dataflow Batch Analytics with Python	Runs a bounded Apache Beam Python graph that aggregates site traffic by user and by minute on Dataflow.
Serverless Spark to BigQuery	Uses Serverless for Apache Spark to enrich a Cloud Storage CSV and load the result into BigQuery.
Cloud Data Fusion Batch Pipeline	Deploys a visual GCS-to-Wrangler-to-BigQuery ETL pipeline through Cloud Data Fusion and the CDAP API.

Streaming and Change Data Capture

Project	Summary
Streaming Dataflow Dashboard	Streams live Open-Meteo events through Pub/Sub and Dataflow into dashboard-ready BigQuery views.
Dataflow Streaming with Python	Uses Python Apache Beam on Dataflow to window Pub/Sub traffic events and stream corridor aggregates into BigQuery.
E-sports Streaming Analytics	Combines Pub/Sub, Apache Beam, Bigtable enrichment, BigQuery serving views, and Streamlit monitoring in one live platform.
Datastream PostgreSQL to BigQuery	Replicates PostgreSQL inserts into BigQuery in real time using Datastream change data capture.

Climate Stream Analytics

• Focus: Cloud Run, Pub/Sub, Dataflow, BigQuery, Cloud Scheduler , Artifact Registry

Event-Driven and Serverless

Project	Summary
Cloud Run Function to BigQuery	Triggers a second-generation Cloud Run function from a storage event and performs an idempotent BigQuery load.

Analytics Engineering

Project	Summary
Dataform SQL Workflow	Publishes and executes SQLX transformations and data quality assertions over public BigQuery data.

Data Quality

Project	Summary
Serverless Spark Data Quality	Applies Spark validation rules and separates trusted records, quarantined failures, and quality metrics in BigQuery.

Lakehouse and External Data

Project	Summary
Lakehouse Qwik Start	Queries governed Cloud Storage data through BigLake external tables and column-level policy tags.
External Data and Iceberg Tables	Writes an Apache Iceberg table to Cloud Storage and queries it from BigQuery without creating a managed data copy.
Federated Query with AlloyDB	Joins live AlloyDB data with native BigQuery data through EXTERNAL_QUERY without replicating the source table.

Machine Learning and Semantic Search

Project	Summary
BigQuery ML Transaction Prediction	Trains, evaluates, and applies a BigQuery ML logistic regression model to predict ecommerce transactions.
BigQuery Vector Search	Generates Vertex AI embeddings and performs semantic retrieval with a BigQuery vector index.