04case study
High-Throughput ELT Data Platform (NewsQuant)
June 2026 - August 2026Website
24.7Mtime-series records
99.9%pipeline execution success
10+source systems
Overview
Led an ELT-over-ETL data platform scaling to 24.7M time-series records, designing a resilient 3-tier PostgreSQL warehouse with star-schema modelling across 10+ sources, building delta-load transformers, and managing a Next.js/Plotly monorepo. Reached 99.9% execution success on data-acquisition pipelines by orchestrating headless-browser automation (Selenium/Chromium) through multi-profile Docker Compose and running 19+ idempotent SQL migrations.
Stack map
The declared stack, sorted into the layers these tools belong to. It assembles as you scroll.
Computeruntime04
PythonDocker ComposeChromiumNext.js
Datastate & storage02
PostgreSQLSQL
Observabilitysignals01
Plotly
Platformcloud01
Selenium