Enterprise AI data ingestion and processing: How raw data becomes agent-consumable representations

1. Why this layer matters

The previous articles in this series covered what agents retain (memory) [1], what agents see (context engineering) [2], and what agents do (runtime) [3]. They also covered how enterprises know agents performed well (evaluation) [4], who agents are (identity and access) [5], and what agents cost (cost and capacity) [6]. Each of those layers consumes representations. None of them produces the representations in the first place.

That production responsibility belongs to data architecture. An agent that retrieves a stale product specification quotes an outdated price. An agent that grounds its response in an incomplete policy document misses an exception clause. Neither failure originates in the model. Both originate in the representation supply chain that feeds the model.

Enterprises already invest in data platforms, data lakes, and master-data management. Enterprise AI does not replace those investments. It adds a new consumer, the AI agent, whose demands differ from dashboards and reports in three ways: subsecond retrieval latency, semantically rich representations rather than raw tabular rows, and provenance metadata that traces every claim back to its source. The data architecture addresses all three demands.

2. Data versus knowledge

The distinction matters for architecture. Data is a recorded observation: a transaction log entry, a sensor reading, or a PDF contract stored in a content management system (CMS). Knowledge is a curated, contextualized representation that carries meaning. A graph triple stating that customer A holds policy B with coverage limit C is knowledge. An ontology class that defines what “coverage limit” means across three business units is knowledge.

Raw data alone does not support reliable agent reasoning. An agent asked to adjudicate a claim needs more than the raw text of a policy document. It needs the extracted coverage terms, the relationships between entities, and the business rules that govern exceptions. The data architecture defines the pipelines that transform raw data into agent-consumable representations. The knowledge architecture defines how agents retrieve, navigate, and govern those representations once they exist.

3. The representation supply chain

The representation supply chain has four stages. This article covers stages 1 and 2 in depth. Companion articles cover the data platform and governance infrastructure that underlies all four stages, the knowledge representation layer that sits between production and retrieval, and retrieval and knowledge governance (stages 3 and 4).

Stage 1: Ingestion. Source systems (enterprise resource planning, customer relationship management, CMS, and external feeds) push or expose data. The ingestion layer normalizes formats, deduplicates records, and applies access-control labels inherited from the source. Ingestion operates in two modes: batch pipelines that run on a schedule and streaming pipelines that consume events in real time (section 7). The data-platform team owns this stage. The primary operational failure mode is silent source drift. A source system changes its schema or refresh cadence and the ingestion pipeline continues to run without error but produces stale or malformed records. The primary adversarial failure mode is source-document compromise: a hostile or corrupted source document seeds false content that propagates through embedding, graph construction, and caching without reverification (section 4.1).

Stage 2: Representation production. Transformation pipelines convert ingested data into agent-consumable representations. Six production pathways operate in parallel. Unstructured-data processing pipelines parse raw documents into clean text (section 4). Embedding pipelines convert that text into dense vectors and store them in a vector database. Graph-construction pipelines extract entities and relationships and load them into a knowledge graph. Graph construction consumes an ontology defined and governed through the knowledge representation layer (covered in the companion article on knowledge representation). Structured-extraction pipelines parse documents into typed fields (dates, monetary amounts, entity names) and store them in a search index. Feature-engineering pipelines compute derived attributes and publish them to a feature store for machine-learning and agent workloads. Synthetic-data generation pipelines produce privacy-preserving representations for agent testing and development environments where restrictions bar production data (section 5). The AI-platform team owns this stage. The primary failure mode is representation mismatch: the embedding model encodes semantics that do not align with the retrieval queries agents will issue.

Stage 3: Retrieval. The retrieval infrastructure serves agent queries at inference time. The knowledge architecture articles cover retrieval in detail, including knowledge-store topologies, agentic retrieval patterns, and the Model Context Protocol.

Stage 4: Knowledge governance. Owners review, version, and retire knowledge assets on a defined cadence. A companion article covers life cycle management, governance mechanisms, and the data catalog.

4. Unstructured-data processing pipelines

Most enterprise knowledge begins as unstructured data: scanned contracts, engineering drawings, PDF manuals, spreadsheets with embedded comments, audio recordings of customer calls, and email threads. The gap between raw unstructured content and agent-consumable representations is the unstructured-data processing pipeline.

This pipeline has four stages.

Extraction. A document parser reads the source file and produces a raw content stream. For born-digital PDFs, the parser extracts text directly. For scanned documents and images, an optical character recognition (OCR) engine converts pixel data to text. For audio files, a speech-to-text (STT) model produces a transcript. For spreadsheets, the parser extracts cell values, formulas, and embedded objects. The data-platform team selects and maintains the parser suite.

Layout analysis. Enterprise documents carry structure: headers, footers, tables, columns, sidebars, and footnotes. A layout-analysis model identifies these structural elements and tags each content block with its role. Without layout analysis, a two-column regulatory filing produces interleaved text that mixes unrelated paragraphs. Table-extraction models identify row and column boundaries and convert visual tables into structured data. PubLayNet [7] provides a benchmark dataset for evaluating layout-analysis models, including table detection. Layout-analysis model selection sits with the AI-platform team.

Normalization. The normalization step standardizes encodings, strips boilerplate (headers, footers, page numbers, legal disclaimers), resolves abbreviations, and applies entity-recognition models to tag domain-specific terms. The output is a clean, structured content object that downstream pipelines chunk, embed, or feed into a graph-construction pipeline.

Quality verification. An automated quality gate compares the extracted text against the source document on a sampled basis. Metrics include character error rate for OCR output, table-structure accuracy for extracted tables, and completeness (% of source pages successfully parsed). Documents that fall below the quality threshold route to a human review queue. The data-quality team is accountable for verification.

The main risk at this stage is silent extraction error, where a corrupted PDF produces partial text, and the pipeline indexes the fragment without flagging the gap. Agents then ground responses on incomplete documents. Coverage-monitoring jobs detect this failure by comparing indexed page counts against source document page counts.

5. Synthetic data in the representation supply chain

Not every agent workload consumes production data. Privacy regulations, data-residency rules, and access-control restrictions prevent development and testing teams from using real customer records, medical histories, or financial transactions. Synthetic-data generation addresses this constraint by producing representations that preserve the statistical properties and schema structure of production data without exposing actual records.

Synthetic data serves the following three roles in the representation supply chain:

Agent testing. Quality-assurance teams use synthetic document corpora, synthetic knowledge-graph triples, and synthetic vector embeddings to test retrieval pipelines, evaluate chunking strategies, and benchmark agent accuracy before deployment. The AI-platform team generates and governs test corpora.

Training-data augmentation. Domain-adapted embedding models and extraction models require training data. In domains where labeled data is scarce (rare medical conditions, low-frequency financial instruments), synthetic augmentation expands the training set. The AI-platform team runs the augmentation pipelines.

Privacy-preserving development. Development environments replicate the structure and schema of production knowledge stores using synthetic records. Developers build and debug agent workflows against realistic data without accessing restricted content. The data-governance team defines the policies that determine which data domains require synthetic substitutes.

The primary failure mode is fidelity collapse, where the synthetic generator produces representations that satisfy schema constraints but fail to capture the semantic relationships present in production data. An agent tested on oversimplified synthetic corpora performs well in testing and fails in production. Fidelity metrics, including statistical similarity, distribution divergence, and downstream-task accuracy, measure this risk. Fidelity validation is the AI-platform team’s responsibility.

Synthetic data is a first-class data asset. It registers in the data catalog with a “synthetic” classification label. The retrieval layer never serves synthetic representations to production agents.

6. Embedding pipelines

Embedding pipelines convert clean text, images, and structured records into dense vector representations that the retrieval layer queries at inference time.

The pipeline has the following three stages:

Chunking. The pipeline splits normalized content into chunks before encoding. Chunk boundaries that break mid-sentence or mid-paragraph produce fragments that lack context. Semantic chunking, splitting at paragraph or section boundaries and preserving heading metadata, produces higher-quality retrieval results. The AI-platform team selects and tests the chunking strategy. Chunking parameters (chunk size, overlap size, boundary rules) vary by document type and domain.

Encoding. An embedding model encodes each chunk into a dense vector. A general-purpose embedding model trained on web text does not distinguish between “coverage limit” as an insurance term and “coverage limit” as a telecommunications term. Domain-adapted or fine-tuned embedding models reduce this ambiguity. Model selection rests with the AI-platform team. That choice draws on retrieval-quality tests run in the evaluation layer [4].

Indexing. The pipeline writes encoded vectors to a vector database with associated metadata: source document identifier, chunk position, modality tag, timestamp, and access-control labels. A companion article covers vector database engineering in detail: indexing algorithms, quantization, sharding, and replication.

Embedding pipelines fail primarily through embedding-query misalignment: the embedding model encodes semantics that do not align with the retrieval queries agents issue. A model trained on general-purpose text fails to place domain-specific terms near their correct semantic neighbors. Retrieval-quality tests in the evaluation layer [4] detect this failure.

7. Real-time and streaming data

Batch pipelines suit workloads where overnight freshness is acceptable, including policy documents, product catalogs, and internal knowledge bases. Streaming pipelines process events as they arrive and suit workloads where minutes or seconds of latency determine response quality, such as fraud detection, operational alerts, live customer interactions, and market-data feeds. Three architectural components support real-time data.

Event-driven ingestion. A distributed messaging platform, such as Apache Kafka or Apache Pulsar, captures events from source systems in real time [8]. Change data capture (CDC) connectors detect row-level changes in transactional databases and publish them as events. The data-platform team operates the messaging infrastructure and CDC connectors.

Streaming transformation. A stream-processing engine, such as Apache Flink or Apache Spark Structured Streaming, transforms raw events into agent-consumable representations on the fly. Streaming transformations compute rolling aggregates, detect anomalies, enrich events with reference data, and update vector embeddings or knowledge-graph triples in near real time. The AI-platform team writes and maintains the transformation logic.

Real-time context synchronization. The agent’s context store, the memory and retrieval layer described in the earlier articles, reflects streaming updates without requiring a full reindex. Incremental-update mechanisms push changed embeddings or graph triples to the retrieval index as they arrive. The AI-platform team owns the synchronization pipeline.

Streaming introduces its own failure mode: temporal inconsistency. The vector store reflects a batch refresh from last night while the relational database management system reflects a real-time balance from this moment. Agents that merge both sources without timestamp awareness produce context windows that mix past and present states. Timestamp propagation across all representation types mitigates this failure. Every representation carries a provenance timestamp, and the retrieval layer filters by recency when the use case demands it.

8. Multimodal data pipelines

Enterprise data spans text, images, audio, and video: computer-aided design drawings, medical scans, call recordings, and trading-floor video.

Multimodal data pipelines extend the representation supply chain in three ways:

Multimodal parsing. The unstructured-data processing pipeline (section 4) adds modality-specific parsers: image-classification and object-detection models for visual content, STT models for audio, and frame-extraction and scene-detection models for video. Each parser produces a text transcript, a structured annotation, or both. The AI-platform team selects and maintains multimodal parsers.

Multimodal embeddings. A multimodal embedding model encodes text and images into a shared vector space [9]. Extended architectures apply the same contrastive-learning principle to audio and video. Cross-modal embedding enables cross-modal retrieval: an agent queries with text and retrieves a relevant engineering diagram, or queries with an image and retrieves the specification document that describes it. Multimodal embedding models require more compute than text-only models. The AI-platform team accounts for this in capacity planning, covered in the cost and capacity article [6].

Multimodal vector storage. The vector database stores and indexes embeddings from multiple modalities. Each embedding carries a modality tag (text, image, audio, video) alongside its metadata. The retrieval layer filters by modality or searches across all modalities depending on the agent’s query. Lakehouse platforms and enterprise vector stores increasingly support multimodal storage natively.

The characteristic failure here is modality mismatch: an embedding model trained primarily on text produces low-quality embeddings for images, and cross-modal retrieval returns irrelevant results. Evaluation pipelines [4] measure retrieval quality per modality, in addition to aggregate quality.

9. Agent-generated content as ingestion input

In production, the pipeline is not unidirectional. Agents write: memories, generated summaries, customer relationship management updates, drafted documents, and structured reports. That output enters enterprise systems such as SharePoint libraries, databases, and CMSs, and becomes tomorrow’s ingestion input.

This loop introduces four architectural requirements:

Authorship-type tagging. Every representation carries a provenance record (section 4). That record must include an authorship-type field that distinguishes human-authored source material from agent-authored derivatives. Without this field, an agent-generated summary reenters the pipeline as if it were a primary source, and subsequent agents ground their reasoning on a derivative rather than on original evidence. The data-platform team adds the authorship-type field to the provenance schema. The AI-platform team ensures that every agent output carries the tag at the point of creation.

Feedback-loop detection. A feedback-loop detector identifies cases where an agent’s output has reentered the ingestion pipeline and the pipeline serves it back to the same or another agent as retrieved context. An agent that retrieves its own prior summary and then summarizes it again produces a progressive drift from the original source, the representation-layer equivalent of a photocopy-of-a-photocopy degradation. The detector compares the provenance chain of retrieved content against the requesting agent’s output history. The AI-platform team operates the detector.

Quality gates for agent-authored content. Agent-generated content that reenters the pipeline passes through the same quality gates as human-authored content (section 4), with an additional gate: a faithfulness check that compares the agent’s output against the source documents it cited. Content that fails the faithfulness check routes to a human review queue before it enters representation production. The data-quality team runs the faithfulness gate.

Write-path authorization. Quality gates govern what enters the pipeline. Authorization governs who may write in the first place. The identity and access architecture [5] grants agent identities least-privilege, write-scoped entitlements for each enterprise store. The ingestion layer enforces these permissions at the write boundary: an agent whose identity scope does not include write access to a given store has its write request rejected before content enters the pipeline. Without this control, an agent with SharePoint write access seeds the corpus with content that downstream agents retrieve as primary source material.

The primary failure mode is untagged reentry: agent-generated content enters the ingestion pipeline through a shared CMS without an authorship-type tag, and downstream agents treat it as primary source material. Ingestion-layer rules that classify incoming documents by authorship type, using metadata from the originating system, mitigate this risk.

10. End-to-end yield

A supply chain’s performance is the compounded result of every stage. In the representation supply chain, each stage introduces a quality loss. Consider a representative pipeline with the following stage-level quality scores. Extraction may achieve 95% completeness. Chunking may preserve 92% of semantic coherence. Embedding may capture 88% of retrieval-relevant meaning. The compounded yield across those three stages alone is roughly 77%, meaning that nearly one in four representations reaching the agent carries a quality deficiency traceable to the production pipeline.

Different teams measure each stage using different metrics. That approach serves operational ownership. System-level accountability requires a single compounded metric. The enterprise AI architect must define a compounded yield metric that multiplies stage-level quality scores across the full pipeline: extraction completeness, chunk quality, embedding fidelity, graph-triple precision, retrieval precision, and governance currency.

Two practical applications follow. First, the yield metric identifies the bottleneck. Extraction at 95% and chunking at 70% means improving extraction produces negligible end-to-end gain. The enterprise invests in chunking. Second, the yield metric informs architectural trade-offs. Choosing between hierarchical navigable small world (HNSW) and inverted file (IVF) indexing requires weighing memory cost per query alongside recall and latency. Choosing between retrieval-augmented generation and long-context prompting requires weighing cost per query alongside noise tolerance and context-window dependency. The yield metric makes cost a first-class axis in every architectural decision.

The AI-platform team is accountable for the end-to-end yield metric, and domain stewards supply the stage-level quality inputs. The enterprise AI architect performs the compounded computation and reports it to the governance board alongside the operational metrics from each stage. Two of the stage-level inputs, retrieval precision and governance currency, belong to the retrieval and knowledge governance layer. The companion article on retrieval and knowledge governance defines the full computation once all stage-level inputs are in place.

11. Reference model

Each row in Figure 1 corresponds to a stage described in this article. Reading across a row shows what the stage is built from, which team is accountable for it, and the failure most likely to go undetected there. Reading down the owner column shows where accountability concentrates. The AI-platform team appears in seven of the eight rows, which is why section 10 places the end-to-end yield metric with that team.

Figure 1. Data ingestion and processing reference model

Source: Infosys

12. What comes next

This article defined the transformation pipelines that produce agent-consumable representations, from source integrity verification through embedding, streaming, multimodal processing, and the agent write path that closes the feedback loop. A companion article covers where those representations live and who owns them: the semantic layer, the data lakehouse, vector database engineering, data security, data ownership models, and data observability. Two further companions cover the knowledge architecture: how enterprises structure knowledge in graphs, ontologies, and knowledge-store topologies, and how agents find, access, and trust representations.