The One Layer of the 2026 Data Stack You Can't Buy

Teams planning their 2026 data platforms often assume that “AI readiness” means adding a fleet of AI-native tools: a vector database here, an agent framework there, perhaps a warehouse migration.

That is not how the year unfolded. The layers of the stack still have the same names they had in 2024. However, four of those layers consolidated into two vendors, the storage format war quietly ended, and one genuinely new layer appeared.

That new layer is where your attention belongs, because it is the only part of the stack you cannot buy off the shelf.

The stack consolidated (and every deal was pitched as AI)

Ten months. Four transactions.

Ten months that consolidated the transformation and orchestration layers

1. 3 Sep 2025

   Fivetran acquires Tobiko Data

   The company behind SQLMesh and SQLGlot.

2. 13 Oct 2025

   Fivetran and dbt Labs announce an all-stock merger

3. 25 Mar 2026

   Fivetran contributes SQLMesh to the Linux Foundation

   Announced at KubeCon Amsterdam.

4. 1 Jun 2026

   The merger closes, and dbt Core v2.0 ships the same day

   Core and the Rust Fusion engine move onto a shared foundation.

5. 13 Jul 2026

   Prefect acquires Dagster Labs

   The two most widely adopted Airflow successors, under one roof.

The consolidation was aggressive. Fivetran acquired Tobiko Data (the creators of SQLMesh and SQLGlot) on September 3, 2025, followed by an all-stock merger with dbt Labs that closed on June 1, 2026. In the interim, they contributed SQLMesh to the Linux Foundation. A single company now controls both major SQL transformation ecosystems.

Simultaneously, dbt shipped Core v2.0, unifying the Python Core and the Rust-based Fusion engine on a shared foundation (while keeping Core Apache 2.0).

Then came the orchestrator shakeup: on July 13, 2026, Prefect acquired Dagster Labs. The framing was telling: Prefect for execution, Dagster for definition, and FastMCP for agent governance. As The New Stack put it, “it’s not a data pipeline story.”

The pattern is clear. Fivetran and dbt branded their merger as “data infrastructure for trusted AI agents.” Prefect pivoted toward MCP. No one is selling “ETL” anymore; they are selling the plumbing for agents.

Even the incumbents shifted. Airflow 3.0 (GA April 22, 2025) introduced asset-oriented scheduling and DAG versioning—effectively adopting the asset-centric model Dagster had championed for years.

To be clear: this doesn’t change your immediate roadmap. If you use dbt and Airflow today, you will likely use them tomorrow. This consolidation is a procurement and roadmap risk, not an architectural one.

The storage argument quietly ended

The “format war” is effectively over. Iceberg v3, ratified in mid-2025, introduced deletion vectors, row lineage, and geospatial types. Adoption followed quickly: AWS shipped V3 support on November 26, 2025, Snowflake made external write support GA on October 17, 2025, and Databricks entered public preview on April 9, 2026. Meanwhile, Apache Polaris graduated to a top-level project on February 19, 2026.

One critical clarification: Delta Lake was not “donated” to Iceberg. Instead, Delta’s UniForm feature allows Iceberg-readable metadata to sit over the same Parquet files. This is interoperability, not a merger.

The takeaway: table format is no longer a bet you can lose. Pick Iceberg, implement a catalog, and spend your “argument budget” on problems that actually move the needle.

The layer that is actually new

Your 2024 stack lacked one critical component: a governed contract defining what a model is allowed to ask and exactly what it receives in return.

This isn’t a luxury; it’s a necessity. While LLMs have improved at SQL, they still struggle with raw production schemas. Academic benchmarks have shifted from “solved” single-table problems to “substantially unsolved” enterprise schemas, as seen in Spider 2.0.

Leaderboard numbers are noisy. A CIDR 2026 paper showed that correcting annotation errors in the BIRD benchmark caused performance shifts of up to 31%. The numbers vary, but the direction is constant: raw text-to-SQL is unreliable at scale.

The industry’s answer is the semantic layer. It has moved from a niche category to core infrastructure:

• dbt Labs open-sourced MetricFlow (Apache 2.0) on October 14, 2025, and joined the Open Semantic Interchange initiative.
• Snowflake turned semantic views into first-class schema objects.
• Databricks launched Genie Ontology on June 16, 2026, as an automated context layer.

Be skeptical of vendor accuracy claims. Databricks reports an 84.5% success rate for Genie on a 28-question suite they designed themselves. dbt Labs’ April 2026 comparison is more honest: raw text-to-SQL accuracy rose from 32.7% to 64.5% over three years, but the semantic layer was correct 100% of the time for questions within its defined coverage.

The structural logic is the real win: if a metric is defined once and join logic is deterministic, the model’s job shrinks from “writing complex SQL across 40 tables” to “selecting the right metric and dimension.” This drastically reduces the blast radius of a hallucination.

The access path: MCP and its security debt

The semantic layer defines what can be asked; the Model Context Protocol (MCP) is how the question is delivered.

MCP is no longer a single-vendor tool. On December 9, 2025, the Linux Foundation formed the Agentic AI Foundation, with Anthropic contributing MCP alongside AWS, Google, Microsoft, and OpenAI.

The specification is hardening. The June 18, 2025 revision classified MCP servers as OAuth Resource Servers and required RFC 8707 Resource Indicators to prevent malicious servers from over-scoping tokens.

This urgency is driven by what Simon Willison calls the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. An agent with a warehouse MCP server and a web-fetch tool possesses all three. Consequently, prompt injection now tops the OWASP Top 10 for LLM Applications.

The risks are real:

• Tool Poisoning: Invariant Labs demonstrated hidden instructions in tool descriptions that are invisible to users but followed by models.
• Supply Chain: Snyk found a malicious npm package posing as an MCP server that BCC’d outbound emails to an attacker.
• RCE: CVE-2025-49596, a critical flaw in Anthropic’s MCP Inspector, carried a CVSS 4.0 score of 9.4.

Treat an MCP server as a dependency with database credentials. To secure it, agents should have their own identities rather than inheriting human credentials. Data should be accessed through the governed semantic layer, not a general SQL tool.

Compliance is also becoming mandatory. The EU AI Act’s general application date is August 2, 2026 (Article 113). While high-risk system deadlines may have shifted to December 2027 via the Digital Omnibus package, you should verify the Official Journal before finalizing your compliance plan.

The basics still matter

As Ben Rogojan argues in his 2026 stack write-up, teams often abandon fundamentals to chase AI, only to face an operational crisis later. Ingestion must be boring, cost must be monitored, and quality expectations must be explicit.

The data suggests a gap in priority. dbt Labs’ 2026 State of Analytics Engineering report (363 respondents) found that 72% prioritize AI-assisted coding, while only 24% prioritize AI-assisted pipeline management (testing, observability, and quality). While 83% say “increasing trust in data” is important, far fewer are funding the infrastructure that actually produces that trust.

Where the investment is going, versus where trust actually comes from

Share of respondents naming each as a priority.

Item	Share of respondents
AI-assisted coding	72%
AI-assisted pipeline management testing, observability, quality controls	24%

For those building retrieval systems, two research findings are essential:

1. Context Rot: Chroma’s research across 18 models shows that performance degrades long before the context limit is reached. A larger window is not a fix.
2. The Pipeline Wins: Anthropic’s contextual retrieval results prove that the pipeline does the heavy lifting. Adding generated context to chunks cut failures by 35%; adding keyword search reached 49%; and adding reranking reached 67%. Same model, same corpus—different pipeline.

A note on the numbers

This field relies on several statistics that fall apart under scrutiny.

The claim that “87% of data science projects never reach production” stems from a 2019 VentureBeat article quoting an IBM executive on a panel. There is no published methodology. Similarly, the “85% of AI projects fail” stat is a misquoted 2018 Gartner forecast about bias-driven erroneous outcomes—a claim about a window that closed years ago.

The MIT NANDA figure—that 95% of organizations see no measurable return from GenAI—is real, but it is often misused as a data-quality statistic. The report actually attributes the gap to a learning problem, explicitly stating the barrier is “not infrastructure, regulation, or talent.”

For a more reliable analysis, see RAND’s study of why AI projects fail. While they cite an external “80%” failure estimate, their own primary research (65 practitioner interviews) identifies data and infrastructure readiness as two of the five root causes. We expanded on this in Enterprise AI runs on data engineering.

Aeolus view — The consolidation news is loud, but it requires little action from you. The quiet requirement is this: every AI feature you ship creates a new data consumer—one that cannot ask a colleague for clarification and will confidently invent an answer instead. Building the layer that constrains what an agent can ask is unglamorous work, but it offers a far better return than another round of model selection. If your metrics aren’t defined in one place yet, start there instead of an agent framework.

Where this leaves you

If you are planning your platform for the next twelve months, the reality is simple:

1. Storage and Format: These are settled. Decide quickly and move on.
2. Ingestion and Orchestration: These are consolidating. This is a contract and roadmap risk, not an engineering hurdle.
3. The Trust Layer: This is where the battle is won. Your success depends on the layer between the model and your tables: the semantic layer. If your data isn’t ready for this yet, we’ll help you get there.