How generative AI connects enterprise data systems

Last updated: July 2026

Generative AI connects enterprise data systems through three mechanisms: retrieval that assembles context at query time, tool calls that read and write live systems, and protocols such as MCP that standardise both. The connection layer, not the model, is where most programmes stall: on average, only 29% of enterprise applications are integrated (MuleSoft, 2025). This post explains how each mechanism works and which one to fix first.

Key takeaways

• MIT's NANDA initiative found that 95% of enterprise generative AI pilots deliver no measurable P&L impact, and the study blames integration that fails to adapt to workflows, not model quality.
• The average enterprise runs 897 applications, and only 29% of them are connected (MuleSoft Connectivity Benchmark, 2025). An AI system can only reason over the slice it can reach.
• Gartner predicted at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. Poor data quality is the first cause it lists.
• Three mechanisms do the connecting: retrieval-augmented generation for knowledge, tool calling for actions, and the Model Context Protocol for standardised access, replacing one custom connector per system pair with one open standard.
• Across the retrieval systems we shipped in 2025 and 2026, data preparation consumed roughly two thirds of the engineering effort. The model was never the bottleneck.

Why do AI initiatives fail without connected data?

They fail because a model can only reason over what it can reach. MIT's NANDA initiative analysed 300 public deployments and found 95% of enterprise GenAI pilots produce no measurable P&L return; the authors point at flawed integration, tools that neither learn from nor adapt to the workflows around them, rather than model capability. Gartner reached a similar verdict from the cost side, predicting that at least 30% of GenAI projects would be abandoned after proof of concept, with poor data quality first on its list of causes. A pilot that answers well from a clean demo corpus degrades on contact with the real estate: silos, stale copies, undocumented permissions.

What are the three mechanisms that connect AI to data?

Every production system we have built or audited uses some combination of three: context assembly, retrieval, and tool calling. They solve different problems, and they fail in different ways.

Context assembly

A language model is stateless. Everything it appears to know about your business in a given exchange was placed into its context window: the conversation so far, retrieved documents, schemas, instructions. Context windows are large but finite. They hold hundreds of pages, not the petabytes an enterprise stores, so the engineering problem is selection: deciding, per request, which fragments of which systems earn a place. Poor selection is invisible in demos and expensive in production, because the model answers confidently from whatever it was given.

Retrieval-augmented generation

RAG retrieves relevant records from your stores at query time, through semantic search over embeddings, keyword indexes, or both, and injects them into the context before the model answers. It is the standard pattern for grounding output in current, private data without retraining a model. Its ceiling is set upstream: if the corpus contains three conflicting versions of a policy document, retrieval will faithfully surface the conflict. Deduplication, ownership, and access control are retrieval-quality problems before they are governance problems.

Tool calling

Tool calling lets the model invoke functions against live systems: query a database, create a ticket, post a journal entry. This is the step from answering to acting, and it is the foundation of the AI and agent systems we build. It is also where connection quality becomes a safety property. A wrong retrieval produces a bad answer; a wrong write produces a bad ledger. Schema validation, retries, and a human-review path for unexpected results are part of the integration, not extras.

When does MCP replace custom integrations?

When the number of system-to-AI connections grows past a handful. Before protocols, every data source needed its own connector for every AI application, an N-by-M problem that Anthropic's Model Context Protocol, open-sourced in November 2024, collapses into N servers speaking one standard. Anthropic's framing of the underlying problem is blunt: even frontier models are "constrained by their isolation from data — trapped behind information silos and legacy systems." MCP does not fix bad data; it standardises access to whatever data you have. The economics of adopting it, and where a plain internal API is still the better call, are covered in our post on when MCP pays off for enterprise integration.

What makes data agent-readable?

An AI system can use a data source when four conditions hold: the schema is documented, identifiers are consistent across systems, permissions are machine-checkable, and access runs through an API rather than a screen. Most estates fail at least two of the four, which is why 90% of IT leaders report that data silos create business problems (MuleSoft, 2025). For older platforms, the API condition is usually the hard one. The architecture for exposing them without a rewrite is its own topic, covered in modernising legacy systems for AI agent access.

What this looks like in practice

A document-processing agent we run for a European manufacturer spent its entire first engineering month on identifier reconciliation between the ERP and the document store. None of that month went to prompts or model selection. Once supplier IDs resolved consistently, retrieval accuracy stopped being a debate and the agent went to production the following quarter. That sequencing, data first, is the default in our data and cloud engineering work because reversing it means rebuilding.

Is a data warehouse the same as connected data?

No. Warehouses and lakes copy data on a schedule for analytics; the connection layer AI needs is live, operational, and permission-aware. A nightly extract answers "what were yesterday's orders" but cannot support an agent that checks stock before confirming today's order, and it usually strips the row-level permissions that decide what a given user's assistant is allowed to see. The two investments are complementary rather than interchangeable: the warehouse serves reporting, while retrieval and tool calling need governed paths into the systems of record themselves. Enterprises that spent the last decade centralising copies still have integration work ahead, which is exactly what the 29% application-connectivity figure measures.

How much of the work is data work?

Most of it. MuleSoft's benchmark puts 39% of IT team time into designing, building, and testing custom integrations before any AI enters the picture. Our own ratio across retrieval projects shipped in 2025 and 2026 is close to two thirds data work: deduplicating corpora, mapping permissions, reconciling identifiers, setting freshness rules. Teams budget for model work because it is novel; the estimate that needs the padding is the plumbing.

Where should an enterprise start?

Start narrow and connect only what the first workflow needs.

1. Pick one workflow and inventory its systems. A quoting assistant might touch the CRM, the price book, and a contracts folder. Three connections is a project; 897 applications is a decade.
2. Make the retrieval corpus trustworthy. Deduplicate, assign ownership, and map who may see what before embedding anything, because retrieval inherits every upstream defect.
3. Add one governed tool call. Give the system a single write action with validation and a review path, and measure its failure rate for a month before adding the second.
4. Standardise access once patterns repeat. When the third team requests the same connection, that is the signal to move it behind a shared protocol server rather than copy the integration.

What separates the teams that ship?

The pattern across our 2025 and 2026 deliveries is consistent: teams that spent their first quarter on data plumbing shipped agents that survived production, and teams that spent it on prompts rebuilt in the second quarter. The next protocol cycle, with agent-to-agent communication already following the MCP playbook, will reward the same preparation, because every new layer assumes the one below it is connected.