Unnecessary Data Warehouse Workloads - Data Duplication | Architectural Simplification

Description

In this episode, Cockroach Labs Technical Evangelist Rob Reid takes a look at data warehousing, specifically around the unnecessary duplication or triplication of data. Rob uses a Before setup and an After setup to point out the benefits of a vastly simplified application architecture with fewer moving parts. With more components that can fail, more network hops involved, and separate maintenance required for each piece, setting up a data warehouse workload like the one in our Before scenario is significantly more complicated. The After scenario features fewer moving parts and less data duplication, meaning fewer network hops, less network transfer, less storage, and hence a lower data warehouse cost. 🔴 Before setup: CockroachDB publishing via CDC (Change Data Capture) to store as objects in S3 as an intermediary data store. Notifications will be published onto SQS, which will trigger a Lambda event and write it into a data warehouse. 🟢 After setup: Publishing directly to the data warehouse from CockroachDB by way of an adapter. Unnecessary Data Warehouse Workloads - Data Duplication | Architectural Simplification 00:00 Introduction 00:11 Behind Rob's demo 03:08 Testing the Before scenario 04:07 Testing the After scenario 05:23 Closing notes ******************************** Further reading: ✔️ What is CockroachDB? https://cockroa.ch/42uaZ9G ✔️ CockroachDB architecture overview: https://cockroa.ch/3RzIvrh #distributedsql #datawarehouse #datawarehousing #cockroachdb