Choosing a Lakehouse for a Contextual Agentic Enterprise

For: enterprise architects, data platform owners, AI engineers and the executives who fund them. The decision: how to compare Databricks, Snowflake, Google BigQuery and Microsoft Fabric as the data platform behind an agentic workflow, and what has to stay outside all of them.

The decision comes before the platform

A platform comparison usually starts with a product list. A contextual agentic workflow starts with a decision that a named person is accountable for. Consider an agent that helps resolve a disputed refund. It needs the right order, the policy that applies today and any earlier decision on the case. It needs permission to read each of them. And it needs a route to the one service that can actually move money.

So the first artefact is not a vendor shortlist. It is a one-page workload statement: the business outcome (a dispute resolved correctly and within policy), the decision the agent supports (propose a refund or decline), the data it needs, how fresh each item must be, who owns each source, and which system performs the action. The business owner sets freshness and recovery requirements. The platform team does not infer them.

Two earlier notes cover neighbouring ground, and this one avoids repeating them. The Contextual Agentic Graph is about how claims are structured, scoped and time-stamped, whatever store holds them. The Commit Boundary is about the final check on an exact action. This note asks a different question: where does governed enterprise data live, which promises does each platform’s own documentation let a team rely on, and what must still be tested?

A public LinkedIn discussion comparing four platforms prompted the topic. That page could not be read when this note was written, so nothing from it is quoted, reproduced or credited. Every product statement below comes from the vendor documentation linked in the sources, opened on 1 November 2026. Vendor documentation describes intended behaviour. It is not independent measurement, and this note cites no independently measured performance.

A lakehouse supplies governed context and holds no authority

A lakehouse combines data-lake storage with table, query and governance features. Products package that combination differently. Two open table formats are common to the four platforms discussed here, and they say less about governance than people tend to assume.

The Apache Iceberg specification describes how a table’s state is recorded through metadata files, manifests and snapshots, where a snapshot is the state of a table at a point in time. In the text read, it contains no authorisation section. The Delta transaction log protocol says it does not define authentication, roles or permissions for users, and leaves authorisation policy to catalog implementations. A shared format therefore makes data readable by more than one engine. Row filters, masks, lineage and audit sit in the catalog and the engine, so each has to be checked on each access path.

The CAE reading is simple. The platform supplies context with provenance and scope. It does not decide that this refund is allowed now. That decision belongs to the business service that commits the action, which rechecks the current order, current policy and delegated authority. This follows the Deterministic Shell and Probabilistic Core idea: the model proposes, and deterministic components own identity, authority and commitment. CAE is the authors’ proposed framework, not an industry standard.

Text description of Figure 1. The picture has six bands, read from top to bottom. Source systems feed a governed platform, shown as four labelled alternatives of which one is chosen. A catalog-and-controls band names the duties every option must perform: metadata and ownership, row and column rules, lineage, and retention and recovery. Authorised context is assembled for one task. The agent then proposes a refund. At the commit boundary the business service checks the current order, current policy and delegated authority, which sit outside the platform, and only then acts. The outcome and an evidence record are kept. The diagram is a proposed conceptual design and does not endorse any product.

One fictional refund, traced from source to action

Illustrative scenario: a fictional composite written to make the point concrete. It does not describe a real organisation, and its figures are not reported outcomes.

A customer of a fictional online retailer disputes a charge of 140 dollars. An agent drafts a recommendation for the support team. The trace below follows the data and shows what to record and where each step can fail. The platform appears in steps 2 and 3 only.

Step 5 is why the platform choice cannot settle the question of authority. Whichever platform sits at step 2, the service at step 5 still has to answer whether this refund is permitted now.

What each platform’s documentation says

The summaries below record what each page states, with its date where it showed one. They are not rankings. Features vary by cloud, edition, region and configuration, and the Databricks starting page is the Google Cloud variant of its documentation, with separate AWS and Azure variants that may differ.

Databricks

• Governance layer. Unity Catalog is described as the governance layer for data and AI, covering tables, views, volumes, functions and models. External tables are governed, while storage lifecycle stays with the customer. It is enabled automatically for workspaces created after 6 March 2024.
• Formats and external engines. The Iceberg documentation says an Iceberg REST Catalog API lets external engines read and write Unity Catalog Iceberg tables, with credential vending. External engines cannot read Unity Catalog views. Managed Iceberg tables need serverless compute, and managed Iceberg materialised views are in Public Preview.
• Row and column controls. Row filters and column masks are SQL functions, or attribute-based policies. External engines cannot use the Iceberg REST catalog or Unity REST APIs to read a table that has them. Time travel is not supported with table-level policies, and ABAC time travel is in Beta.
• Lineage and recovery. Lineage is captured automatically, including column-level lineage for queries run on Databricks, with a rolling one-year window in the system tables. Lineage is lost for renamed objects. Delta history keeps log entries for 30 days by default and deleted files for 7, and the page says table history is not a long-term archival backup.
• Agents. Agent authentication offers three modes. On-behalf-of-user (Public Preview) makes Unity Catalog enforce the end user’s permissions. Automatic passthrough uses a system-generated service principal. Manual credentials run as whichever identity they belong to.
• Cost categories. The pricing overview lists Databricks Units, Databricks Storage Units, separately billed cloud provider charges, and data transfer, with prices varying by region and cloud.

Snowflake

• Formats and catalog. Iceberg tables can use Snowflake as the catalog, with full platform support, or an external catalog, with limited platform support. Storage is either Snowflake-managed, where Fail-safe protects permanent tables, or a customer-managed external volume, where the customer is responsible for protection and recovery and Fail-safe does not apply.
• External engines. The same page says external engines can read Snowflake-managed Iceberg tables through Horizon Catalog, that masking and row access policies are enforced for them, and that some write operations are unsupported.
• Row and column controls. Row access policies require Enterprise Edition or higher and are evaluated using the policy owner’s role. If a provider’s policy calls the current role or user across a data share, consumers get NULL.
• Lineage and recovery. Access History needs Enterprise Edition or higher, records reads, writes and the policies referenced, and offers column lineage for write operations. It states no retention period. Time Travel is 1 day by default, up to 90 days for permanent objects on Enterprise Edition or higher, followed by 7 days of Fail-safe that only Snowflake can recover.
• Agents. Cortex Agents call tools such as Cortex Analyst and Cortex Search. Data access is governed by Snowflake privileges and each tool’s execution context. Analytical search is Public Preview, and the page says accuracy and citations are not guaranteed.
• Cost categories. Overall cost names warehouse credits, serverless compute, cloud services above a threshold, storage, and data transfer on egress. That page leaves AI features such as Cortex out of scope.

Google BigQuery

• Formats and storage. BigQuery tables for Apache Iceberg keep Parquet data in a Cloud Storage bucket the customer owns. The page lists time travel, column-level security, row-level security and data masking, and exported Iceberg metadata snapshots for other engines. Changing data files outside BigQuery can cause query failures or data loss, and several bucket features are unsupported. The text read showed no launch-stage label and no page date.
• Row controls. Row-level security filters by grantee list. Policies with subqueries are not compatible with the Storage Read API, and the feature may be unavailable with reservations created under certain editions. Policy creation is audit logged, but the filter expression is omitted from the log.
• Lineage and recovery. Lineage is enabled per project through the Data Lineage API. It is kept for 30 days, and BigQuery lineage can take up to 24 hours to appear after the job completes. The page says Dataplex Universal Catalog became Knowledge Catalog on 10 April 2026. Time travel defaults to 7 days, configurable from 2 to 7, followed by a 7-day fail-safe that is not configurable.
• Agents. The BigQuery MCP server uses OAuth with IAM. It exposes a read-only and a read-write SQL tool, and the read-write tool can be blocked with an IAM deny policy. Results are capped at 3,000 rows.
• Cost categories. Pricing lists on-demand compute by bytes processed, capacity compute by slot-hours under editions, active and long-term storage, logical or physical storage billing, ingestion, extraction and optional services. For Iceberg managed tables, table management is billed as compute.

Microsoft Fabric

• Storage and formats. OneLake is the unified data lake for Fabric. A lakehouse has Files and Tables areas, and tables are Delta. A shortcut is a pointer to data elsewhere, so the data is not copied.
• Security. OneLake security defines roles scoped to folders, tables, schemas, rows and columns, for Microsoft Entra identities. Workspace Admin, Member and Contributor roles are not affected by those roles. Authorised third-party engine enforcement is in preview.
• Row-level security. OneLake row-level security is enforced for Delta Parquet tables and virtualised Iceberg tables, and unauthorised third-party engines cannot read the table. Row and column rules combine only within a single role.
• Lineage and recovery. Lineage is item-level, and the page does not mention column-level lineage. In table maintenance, VACUUM defaults to a 7-day retention threshold, and removed files cannot be time-travelled to.
• Agents. A Fabric data agent is generally available, runs queries under the requesting user’s permissions, and is read-only. It needs a paid F2 or larger capacity, or a P1 or larger Power BI Premium capacity, and its capacity must be in the same region as the data source’s.
• Cost categories. Licences describe capacity units bought pay-as-you-go or by reservation, plus per-user licences. OneLake consumption says storage is billed per gigabyte and does not consume capacity units, while reads and writes do. OneLake row-level security is metered per million rows.

Context needs provenance, freshness and a way to correct it

Lineage tells a team how data moved. It does not say whether a policy passage was the current one, or whether a customer’s consent still stood. The NIST generative AI profile is a voluntary profile. Its suggested actions call for documenting the origin and history of training and generated data, and for documenting data lineage, sources and curation policies. This note applies that discipline by analogy to the context an agent receives at run time.

The windows on the pages read differ, and they matter for correction and replay:

Two practical rules follow. Correct the source and mark the old claim superseded, rather than patching an agent’s memory. And store the context the agent actually saw with each decision, because a platform’s time-travel window will close long before an audit or a complaint does. The Evidence-First Decision Records note describes that record.

Test every access path, not just the policy

A catalog entry is evidence of configuration. It is not evidence of enforcement. The pages read give three different answers to the same question about external engines. Databricks says external engines cannot read a table with filters or masks through its REST catalog. Snowflake says its policies are enforced for external engines on Snowflake-managed Iceberg tables. Fabric says authorised third-party engines are filtered and unauthorised ones cannot read the table, with enforcement for third-party engines in preview. None of these is better in the abstract. Each changes what a team must test.

A cross-boundary example shows why. In a Snowflake data share, a provider’s policy that calls the current role or user returns NULL to consumers, so a rule that worked inside one account behaves differently across the boundary. Other pages show path-specific behaviour too. Fabric’s workspace Admin, Member and Contributor roles are not affected by OneLake security roles, and BigQuery’s subquery-based row policies are not compatible with the Storage Read API.

Run these tests against the real paths, using real identities:

1. A person with rights, a person without, and a revoked person, through the SQL engine, a notebook, a BI tool and the agent’s own tool.
2. A cross-tenant request, and a request with the wrong purpose.
3. Every external engine, share, shortcut, export and direct-storage route that exists in the design.
4. The agent’s identity: does it run as the user, or as a service identity with its own rights? Document the answer for each tool.
5. Audit: can an auditor reconstruct who read what, and does the log omit anything that matters?

Measure cost per accepted outcome, using the same workload

The cost categories overlap but are not interchangeable, so do not set list-price units side by side. Compare what a workload costs. The categories to capture are storage, compute for queries and pipelines, background maintenance and optimisation, data movement and egress, catalog, lineage and audit features, retrieval indexes, AI orchestration and tokens, licences or capacity commitments, networking, and the human time spent on review and reconciliation.

A comparable measurement runs like this:

1. Fix one representative workload: the same data, the same questions and agent tasks, the same freshness requirement, and the same access controls switched on. Controls can carry cost. BigQuery says subquery policies can add billing, and Fabric meters OneLake row-level security.
2. Choose a period long enough to include a peak, and run the same period on each candidate.
3. Record metered line items from the platform’s own usage reporting or invoices. Label any estimate as an estimate.
4. Add what the platform does not bill: integration work, review time, reconciliation and support.
5. Divide total cost by accepted outcomes, such as refunds correctly resolved, and report rework and exceptions beside it.

This note publishes no price or measured result, because none was independently measured. Any figure in your own comparison should carry its workload, period and settings.

Know who owns recovery, and rehearse the exit

Ownership differs by design. Snowflake’s page separates Snowflake-managed storage from customer-managed volumes, where recovery is the customer’s job. BigQuery’s Iceberg tables sit in a customer-owned bucket but warn against changing files outside BigQuery. Databricks governs external tables without managing their storage lifecycle. Fabric shortcuts point at data that stays where it is. In each case, someone has to own the catalog, the storage and the recovery procedure by name.

An open format is one part of portability. Policies, lineage, audit history and operating habits are not carried by the format. A practical exit test:

1. Choose the refund workload’s data and its access rules as the sample.
2. Export tables in an open format together with metadata.
3. Restore into a separately governed environment with its own identities.
4. Recreate the access rules from documentation, and count the manual steps.
5. Rerun the access-path tests and an evidence replay for one past refund.
6. Record elapsed time, manual steps and what was lost: lineage, policies, audit history, retention.

Agree the pass criteria before the test. It measures effort. It does not prove portability.

Region, edition and preview caveats

How a platform choice goes wrong

• A lakehouse is picked on query speed, and refund quality is never measured.
• An agent runs as a broad service account, so row and column rules protect people but not the agent.
• A catalog shows a policy while a direct storage route or external engine bypasses it.
• Retrieved context is treated as approval, and the refund service never rechecks the order.
• A time-travel window closes before the complaint arrives, and nobody stored what the agent saw.
• Cost is compared as list-price units, and reconciliation work lands on operations unmeasured.
• A preview feature becomes a production dependency without anyone deciding that it should.

None of these is a claim that a particular vendor has failed. They are reasons to test a real configuration.

Executives decide; architects design; engineers test

Application requires organisation-specific architecture, security, privacy, legal, regulatory, operational, safety, accessibility, commercial and governance review, together with appropriate approvals.

What this note does not establish

• It ranks nothing and does not say any platform is best.
• It cites no independently measured performance, benchmark, price or customer result.
• It reports vendor documentation as that documentation states it. Pages change, and some showed no date.
• It did not open every related page. Snowflake’s Iceberg time travel and model availability pages are examples.
• It does not read the LinkedIn discussion, so it neither agrees nor disagrees with it.

Key takeaways

• Choose against one representative workflow and its business outcome, not a general product ranking.
• A shared table format makes data readable by more than one engine, but it does not carry access rules, lineage or audit.
• Test denied access through every real path, including external engines, shares, shortcuts and the agent’s own identity.
• Context needs provenance and a correction route, and the context an agent saw must be stored because time-travel windows are short.
• The platform supplies context. The service that commits the action must recheck current state and authority.
• Compare cost per accepted outcome on the same workload, and agree an exit test before signing.

Sources and further reading

Each product statement above comes from a vendor page opened on 1 November 2026. Where a page showed a date it is given in the references, and where it showed none that is stated. The Databricks pages are the Google Cloud variant. The NIST profile is voluntary. The LinkedIn discussion could not be read and is not a source for any statement.

Continue the work

Pick one workflow and one accountable owner, then run the access-path tests and the exit test on a small copy of its data. The Contextual Agentic Graph covers how to structure and scope the claims, and One Architecture, Four Platforms covers a different set of four platforms. The Contextual Agentic Enterprise develops the wider relationship between context, authority and evidence. None of these selects a vendor for your organisation.

This article is part of CAE Field Notes, a series accompanying the book, The Contextual Agentic Enterprise.