Data Engineering Manager

Context

Crestline Commerce is a pan-European e-commerce platform (fashion vertical, €600M GMV). The data team I inherited relied entirely on a single Oracle Data Warehouse and a tangle of stored procedures that no one fully understood. ETL jobs ran nightly from 23:00 to 09:00; any failure silently nulled out the morning reports.

Challenge

Three problems demanded simultaneous attention: (1) the DW was running at 94% storage capacity with no budget to scale Oracle further; (2) the analytics team needed near-real-time order and inventory data for personalisation; (3) four engineers were planning to leave — morale was low after years of on-call firefighting.

What I did

Migration architecture: designed a phased migration from Oracle to Snowflake using Fivetran for historical backfill and custom Python connectors for Oracle sources without native Fivetran support. Ran dual-write for 3 months with automated reconciliation checks.

Transformation modernisation: replaced 800+ stored procedures with a dbt project. Introduced testing (not_null, unique, accepted_values) on every model; blocked merges that reduced test coverage below 90%.

Real-time layer: introduced Kafka for order events, feeding a Snowflake Dynamic Table for the personalisation team, reducing their data freshness from 8 hours to under 3 minutes.

Team: promoted two senior engineers to tech leads, introduced bi-weekly architecture reviews, and reduced on-call incidents by 65% in 6 months through better observability (Monte Carlo for data quality, PagerDuty).

Outcomes

• —Migration completed 2 weeks ahead of schedule with zero data loss validated by reconciliation reports.
• —Morning report availability: 09:15 → 07:00 (before business opens).
• —All four engineers at risk of leaving stayed through the end of the migration.