Building Apache Iceberg Lakehouse Storage with S3 Table Buckets

This post was originally published on graycloudarch.com . The data platform team had a deadline and a storage decision to make.\ They'd committed to Apache Iceberg as the table format --- open standard,\ time travel, schema evolution, the usual reasons. What they hadn't\ locked down was where the data was actually going to live, and whether\ the storage layer would hold up under the metadata-heavy access patterns\ Iceberg requires. The default answer is regular S3. It works. Most Iceberg deployments\ run on it. But AWS launched S3 Table Buckets in late 2024, and they're\ purpose-built for exactly this workload: Iceberg metadata operations.\ The numbers made the decision easy --- 10x faster metadata queries, 50% or\ more improvement in query planning time compared to standard S3. The\ gotcha worth knowing upfront: S3 Table Bucket support requires AWS\ Provider 5.70 or later. If your Terraform modules are pinned to an older\ provider version, that's your first upgrade. We built the stora