Google Cloud lets BigQuery query Databricks Iceberg

Google Cloud has introduced support for querying Databricks-managed Apache Iceberg tables from BigQuery without copying the underlying data. The move more closely links BigQuery and Databricks for organisations that use both systems.

BigQuery can read the same Parquet files written by Databricks and use Iceberg metadata directly, with changes appearing after the next scheduled metadata refresh, which can run as often as every five minutes. The approach is intended to avoid the data duplication, delays and extra storage costs that often come with moving information between analytics environments.

Many large companies run Apache Spark workloads in Databricks while using BigQuery for analytics at scale. Bringing the platforms together has often required separate pipelines to copy datasets between systems, a process that can also complicate security and governance because controls must be managed across more than one environment.

Under the new setup, Databricks-managed Iceberg tables generate Iceberg-compatible metadata alongside native Delta logs. BigQuery reads that metadata from storage rather than loading the dataset into its own managed tables first.

Cross-cloud reach

The same federated model can extend beyond Google Cloud Storage to data held in Amazon S3 and Azure Blob Storage. Through the Apache Iceberg REST Catalog and cross-cloud connections, BigQuery can query open-format Parquet files where they already sit instead of requiring them to be transferred into Google's own storage layer.

That means organisations with data spread across several cloud providers can analyse it through BigQuery while leaving the files in place. Remote-cloud egress fees still apply, although those charges can be reduced in some cases through AWS interconnect arrangements and caching.

Google also set out a governance model based on delegated access. Administrators can configure a BigQuery Cloud Resource Connection tied to a service account that manages identity and access rights for Google Cloud Storage buckets. Access to the Lakehouse table entity then allows BigQuery policies such as row-level security, column-level security and dynamic data masking to be applied to external Iceberg tables.

Metadata layer

The integration relies on Databricks Unity Catalog as the central metadata layer. Rather than pointing BigQuery queries to fixed metadata paths, administrators can establish a Lakehouse runtime catalog with a federated connection, allowing BigQuery to request the current table state from Unity Catalog when queries run.

The process uses short-lived tokens through credential vending to give BigQuery temporary read access to the relevant storage. That is intended to avoid long-lived credentials while keeping metadata current at query time.

Partition pruning is a key part of the performance model. BigQuery uses Iceberg metadata to determine which files need to be read and skips partitions and Parquet files that do not match a query's filters. Hidden partitioning in Iceberg means analysts can filter on logical columns such as timestamps without needing to know the physical directory layout.

For repeat access to data stored outside Google Cloud, Lakehouse caching is available in preview. Cached data blocks are kept in Google Cloud so subsequent queries can read them locally, although Iceberg manifest files are still read directly during query planning rather than being served from a metadata cache.

Operational upkeep

Google also highlighted the operational work needed to keep lakehouse environments efficient. Frequent writes, continuous ingestion and streaming jobs can create large numbers of small Parquet files, increasing the storage requests and metadata operations required during a query.

To address that, Google pointed to compaction routines in Databricks, including OPTIMIZE and Liquid Clustering, which combine smaller files into larger blocks. Larger file sizes can improve BigQuery scan efficiency by reducing the overhead of reading many fragmented objects.

Lifecycle management is another area of focus. Each transactional update or compaction step creates new Iceberg snapshots, and files that are no longer referenced can accumulate over time. Databricks VACUUM operations can remove abandoned files after the standard retention period, helping to limit storage costs while preserving enough history to avoid disrupting active analytical jobs.

The announcement reflects broader demand for open-table formats such as Apache Iceberg, which are increasingly used by companies seeking to analyse shared datasets across different compute engines and cloud platforms. By centring the link on Iceberg metadata, REST Catalog access and temporary credentials, Google is positioning BigQuery to work more directly with data managed outside its core storage estate.

Google said the model enables both read and write interoperability across engines through an open Apache Iceberg foundation, while allowing teams to run multi-statement SQL in BigQuery against the connected data.