Member-only story
Embracing the ELT Revolution
As an engineering leader, I’ve spent over a decade creating, implementing, and supporting complex and innovative infrastructures. My journey in the data sphere, which began with traditional ETL (Extract, Transform, Load) models, has taken a fascinating turn in recent years toward the ELT (Extract, Load, Transform) paradigm. I’ve noticed the tremendous advantages ELT brings to modern data platforms, and I’d like to share my insights with you.
The Old Guard: ETL
The traditional ETL process was a great starting point, allowing us to extract data from multiple sources, transform it according to business rules, and then load it into a data warehouse. During my tenure in Data Engineering at Dow Jones, I led a number of teams responsible for building platforms that processed billions of records and terabytes of data daily. ETL served us well, offering a structured and predictable data processing methodology.
However, the ETL model often fell short of real-time data processing needs or required an ever-growing amount of capacity when updating existing or adding new data pipelines. Data transformation upfront led to delays, inflexibility, and a resource-intensive process, making it less suitable for dealing with large volumes of data.
A Paradigm Shift: ELT
While leading the data platforming operations at Dynatron Software, I spearheaded the development of a new data platform where we were able to sidestep an ETL in favor of an ELT model. This decision was fuelled by being more resource-constrained for a company of its size, the growing complexity and volume of data, the need for real-time analytics, and the evolution of cloud-based data warehouse solutions.
Unlike ETL, the ELT process loads raw data into the data warehouse before any transformations take place. Transformations are executed directly within the…
