Internet iceberg icediagram manual#

This works great when a steady stream of people want lunch. Customers wait a little while for someone to take their order, make it, and check them out. To understand the trade-off, think of a data warehouse like a combined bakery/sandwich shop, where the sandwich shop (query layer) is the only way to get bread from the bakery (storage layer). With Iceberg, warehouses no longer need a runtime query layer to deliver safe and correct queries. Having a single way to access data is a useful assumption when building a data warehouse, but it is also an Achilles heel. This is how access controls are enforced, how queries are isolated from concurrent changes, and it enables features like result set caching and background data maintenance. But a crucial assumption is baked into this model: that the query layer handles all access to data. Warehouses are built to accept SQL queries and quickly produce manageable-sized results. These tasks are not easy to do in a data warehouse.

They expect to be able to load data into hundreds of Python containers in parallel to test model parameters, or to use a streaming framework to easily sessionize events, or to query the same tables from a BI tool as they are consumed by Spark. People want the right tool for the job and to use what they’re comfortable and productive with. I think the answer is the incredible demand for different ways to work with data. So what’s the compelling reason to build Iceberg into data warehouses? It’s easy to see how data lakes are improved by Iceberg’s warehouse features - nobody likes unreliable queries - but warehouses already have mature tables with standard SQL behavior that deliver great performance. It brings together the best capabilities of both: you can be productive and rely on the guarantees of a data warehouse using any processing tool or query engine, like a data lake. Iceberg’s appeal is that you aren’t forced to choose between data lakes and data warehouses. And there are more examples, like Flink for streaming or Python DataFrames that run on GPUs. Spark prioritized reliability and can mix Python, Scala and SQL together to handle any workload. Trino prioritizes speed and is great for ad hoc SQL queries across multiple sources. For example, Spark and Trino come from the data lake world, and are commonly used in the same architecture because they’re good at different things. The main benefit is flexibility: There is a wealth of projects and processing frameworks. The flaws can be managed and data lakes have benefits that make them attractive.

Internet iceberg icediagram manual#

On the other hand, if you’re coming from a data warehouse or modern data stack background, that probably sounds horrible! Data lakes can’t rename columns? Queries might just give wrong answers? Partitioning is a manual process that people mess up all the time? Sigh. If you’re looking at Iceberg from a data lake background, its features are impressive: queries can time travel, transactions are safe so queries never lie, partitioning (data layout) is automatic and can be updated, schema evolution is reliable - no more zombie data! - and a lot more. At Netflix, we used Iceberg to transform our data lake into a cloud-native data warehouse, by building the guarantees of SQL into data lake tables. What is Iceberg?Īpache Iceberg is a modern table format for analytic tables, created by my team at Netflix and later adopted at companies like Apple, LinkedIn, Stripe, Airbnb, Pinterest, and Expedia. Those are radical changes! In this post, I’ll take a deeper look at what Iceberg is and why it is being built into the foundation of the modern data stack. A little earlier this year, Google announced BigLake, a project that will bring together BigQuery and open standards, like Iceberg and Parquet. Last month, Snowflake announced upcoming support for native Iceberg tables. The future of the modern data stack is tightly coupled to how those warehouses evolve - and that’s where Iceberg comes in. The transition from ETL to ELT led by Fivetran was one of several trends that converged to make the modern data stack possible, but the initial spark was the arrival of cloud data warehouses: Snowflake and BigQuery. For example, Fivetran makes it easier than ever to load all of your data sources into your data warehouse, immediately ready for consumption or transformation. The modern data stack has proven to be incredibly valuable. This article is guest authored by Ryan Blue, one of Iceberg’s creators.