Planning for scale: How Materialize uses CockroachDB | RoachFest 2023

Description

Operational Data Warehouse Materialize put a streaming engine in a database so teams can build real-time data products without the cost, complexity, and development time of stream processing. To deliver on Materialize’s speed and correctness guarantees, they needed a strongly consistent and scalable metadata store. CockroachDB fit the bill. At customer conference RoachFest 2023, Member of Technical Staff Parker Timmerman talked about Materialize's journey to CockroachDB. First, Parker walks through how CockroachDB allows Materialize to deliver on these correctness guarantees, then he discusses how his team leveraged CockroachDB's scalability to keep Materialize fast as they grown their user base. They also dive into some technical details like how using CockroachDB immediately sidestepped performance bottlenecks in vanilla PostgreSQL, and how they further tuned their data layout and access patterns for CockroachDB's architecture. Overall, CockroachDB solves hard distributed systems problems, so Materialize can focus on their killer feature of incremental computation. Planning for scale: How Materialize uses CockroachDB | RoachFest 2023 00:00 Welcome/Agenda 00:52 What is Materialize? 01:35 What are the key properties of Materialize? 01:57 Transitioning to a cloud-native product 03:02 Planning and development 03:40 CockroachDB vs FoundationDB, Amazon RDS, Aurora, Dynamo DB for metadata layer 05:36 What does Materialize use CockroachDB for? 08:01 Materialize's high-level cloud architecture 08:48 Design partnership with Cockroach Labs 09:30 Going into production and what the future looks like 11:01 How to optimize performance on CockroachDB 13:01 Final thoughts #distributedsql #datastreaming #datawarehouse