Your model isn't dumb – Your data pipeline is: Why AI scaling breaks

1. The scale problem

Data preparation sits on the critical path of model outcomes. Early compute-optimal estimates from the Chinchilla study [1] suggested pairing roughly 20 training tokens with each model parameter. Frontier runs have since broken that ceiling by a wide margin: Llama 3 trained on 15 trillion tokens, and other frontier models train at comparable scale. At this volume, every preprocessing decision we make multiplies directly into GPU hours saved or wasted downstream, which is why the pipeline deserves the same rigor as the model itself.

The bottleneck is not raw parallelism but how the data is laid out, how closely computation runs to it, and how jobs land on shared hardware. Throwing more workers at any of these problems does not help; the architecture must.

2. Architecture principles

Before reaching for any specific framework, we settled on the following principles that shaped every decision that followed:

• We standardized on Apache Parquet for practical performance reasons rather than aesthetics. Its columnar storage format organizes data by column within row groups, enabling projection pushdown to read only the required columns and predicate pushdown to skip entire row groups whose min/max statistics indicate they cannot satisfy a filter condition. In addition, dictionary encoding and zstandard compression significantly reduce the amount of data read from storage, improving both efficiency and query performance.
• Columnar layout also makes enrichment additive: we append quality, domain, and toxicity signals as new columns without rewriting the source data, so a later job scores just the code subset rather than rescanning the corpus.
• Compute that runs close to data wins. Distributed Parquet scans and shared-memory object stores consistently outperform adding more nodes to an input/output (I/O)-bound problem. We optimized for locality before we optimized for parallelism.
• We deliberately matched each framework to the shape of the work: Dask for out-of-core Data Frame operations, Ray for irregular parsing and GPU coordination, high-throughput inference engines like vLLM for batched inference, and distributed inference engine like Daft for in-situ analytics. Because shard sizes and schemas stay identical across stages, switching frameworks never means reshaping data. Figure 1 shows how these stages share one storage layer.

Figure 1. The three-stage pipeline over shared storage

Note: CPU and GPU jobs enrich the same raw shards and append columns; a distributed analytics engine queries the curated dataset in place, with no separate extract, transform, and load (ETL) copy.

Source: Infosys

3. CPU preprocessing: Format and shards come first

The first stage profiles and reshapes raw text. We score quality with heuristic signals like character-to-token ratio, alphanumeric fraction, mean and maximum line length, code-to-comment ratio, and structural signals such as a universal abstract syntax tree for code. Deduplication runs in the same stage: deduplicated, quality-filtered corpora reach a target loss in materially fewer steps [2]. Exact duplicates fall out cheaply with a content hash; near-duplicates require MinHash with locality-sensitive hashing, which reduces each document to a compact signature and buckets only high-similarity pairs, turning a quadratic all-pairs comparison into a near-linear one.

We treat each shard as the atomic unit of execution, which turns fault tolerance into a property of the layout: when a malformed input fails a task, only that shard retries. In practice, 256–512 MB shards hit the sweet spot. They are large enough to amortize per-task scheduler overhead but small enough to keep the straggler tail short and stay within a worker's memory budget. Combined with checkpointing at the shard boundary, this lets pipelines resume from precise failure points rather than from the beginning.

Parallelism is not free, and this is where measurement mattered most. As worker count grows, storage I/O contention makes speedups fall below the linear ideal. Beyond a threshold, oversubscription with more workers than the per-shard memory budget drives memory and file-descriptor exhaustion; the job stops improving and starts failing outright. The rule we adopted was to size worker concurrency to the memory-per-shard budget and stop short of that cliff, rather than chasing nominal core utilization. For instance, at 512 MB shards on a 128 GB node, the ceiling was 32 concurrent workers; beyond that, file-descriptor exhaustion began terminating tasks outright. Figure 2 shows this pattern.

Figure 2. CPU parallelism scales sub linearly, then falls off a cliff

Past the optimal worker count, over-subscription causes hard failures rather than diminishing returns.

Source: Infosys

4. GPU enrichment: Three use cases, one pattern

Once data is clean and well-sharded, the bottleneck shifts to semantic enrichment, attaching domain, quality, and toxicity signals to billions of samples. Modern pre-training uses curriculum learning [3]: data moves through ordered stages and proportions, and scoring every sample comes first. The same enrichment pattern applies when building enterprise RAG pipelines: relevance ranking, content filtering, and metadata extraction all need model-driven labeling. Which execution model we reach for depends on the task.

Use case 1 — Binary and ordinal classification. Compact classifiers and reward models such as ArmoRM, a roughly 7-billion-parameter reward model [4], fit on a single GPU and fan out cleanly as data-parallel replicas. We use these for quality tiers, policy checks, and RAG relevance filtering. Running them on the CPU-filtered corpus, which is already reduced in volume, is a cost-effective intermediate step before any expensive large-model enrichment.

Use case 2 — Generative rewriting and instruction synthesis. When the work turns generative, that is, rewriting support ticket data into instruction-response pairs or generating rich metadata for RAG documents, we reach for a large model such as Qwen-72B. It no longer fits on one device, so we run it tensor-parallel: each layer splits across GPUs with an all-reduce after every block. Every run executes as batch pipeline execution rather than application programming interface serving, because at this scale sustained throughput matters, not per-request latency.

Use case 3 — Embedding and reranking for RAG. The same pattern covers enterprise retrieval pipelines: embedding generation, chunking quality scoring, and document classification. Small embedding models run data-parallel; large reranking or cross-encoder models run tensor-parallel.

Throughput is a configuration problem more often than a hardware one. We serve large models with vLLM on Ray Data in one execution graph. PagedAttention [5] manages the key-value cache in fixed-size pages, removing the fragmentation that otherwise caps concurrent sequences, and continuous batching sustains 60–80% GPU occupancy where naive static batching idles at 20–30%. Figure 3 traces successive tuning iterations of an ArmoRM scoring job on a fixed 4-GPU node: by scaling per-replica batch size from 32 to 512 and Ray actor concurrency from 4 to 16, runtime fell roughly 16× with no change in GPU allocation. This is why we tune batching before we request more GPUs.

Figure 3. Batch-inference throughput is a tuning problem, not a hardware problem

ArmoRM reward-scoring runtime across four configuration iterations on a fixed 4-GPU node. Per-replica batch size: 32 → 512; Ray actor concurrency: 4 → 16. Combined tuning yields ~16× throughput gain without adding GPUs.

Source: Infosys

Two further choices kept GPU runs economical. We checkpoint every ~1,000 samples with lineage-based recovery, so pre-emption on spot instances never forces full re-computation. And observability paid for itself: Our data center GPU manager (DCGM) instrumentation showed 40% of GPU nodes sitting idle during CPU-heavy phases, which pushed us toward heterogeneous scheduling across Kubernetes and Slurm.

5. In-situ analytics: bringing compute to the data

Producing multi-terabyte datasets is the easy part; interrogating them at that scale is where we hit friction. Validating quality-signal distributions, semantic density, and variety is a continuous part of the curation loop.

Our first approach queried local Parquet snapshots with DuckDB, an in-process vectorized engine whose morsel-driven, nonuniform memory access-aware parallelism [6] makes it excellent for medium-scale exploration with near-zero setup. In the multi-terabyte regime, it hit single-node limits: working sets outgrew memory, complex queries forced repeated materialization and joins across thousands of partitions were hard to coordinate on one machine. The problem was not querying expressiveness but execution locality: we needed to run the query where the data already lived.

We introduced Daft as a distributed analytics layer over the existing layout. Its cost-based optimizer pushes filters and column projections into the Parquet scan and decodes only the rows that survive. Because Daft runs on Ray, the same cluster that computes CPU signals and filters low-quality records can also trigger a GPU rewrite, all as one plan with no data movement between stages. A group-by aggregation over a 1.2 Terabyte (TB) dataset that took 14–18 minutes under DuckDB finished in under 5 minutes under Daft's distributed, in-situ execution (Figure 4). Analytics that had been an overnight batch became an interactive step in the curation loop.

Figure 4. In-situ execution makes multi-terabyte analytics interactive

Identical group-by aggregation on a 1.2 Terabyte (TB) Parquet dataset. Distributed, in-place execution avoids the local materialization that dominates single-node runtime.

Source: Infosys

6. The full stack: Hybrid orchestration

The three stages converge on the architecture in Figure 5. We run Ray and Dask clusters across both Kubernetes and Slurm, so the same job definitions execute on high-performance computing (HPC) nodes and cloud-native infrastructure without modification. CPU preprocessing and GPU enrichment share storage; Daft queries the Parquet datasets in place. Apache Airflow drives the inter-job directed acyclic graphs (DAGs); Prometheus, Grafana, and DCGM give us end-to-end telemetry across every stage.

A single control plane is a technical choice, not an organizational one. Because Ray schedules long-lived GPU actors, bursty CPU tasks, and vectorized transforms with one resource-aware scheduler, cross-stage dependencies are explicit in code rather than brittle glue between separate systems and that eliminates the fragmentation and idle-GPU waste that separate orchestrators produce.

Figure 5. End-to-end hybrid orchestration architecture with Ray and Dask clusters

Jobs are routed across Kubernetes and Slurm clusters that share the same dataset storage and container registry.

Source: Infosys

7. Lessons that transfer

Running this pipeline end-to-end rather than tuning each framework in isolation surfaced lessons that hold well beyond LLM data preparation:

• Layout and locality are the foundation: Columnar Parquet with stable shard sizes make fault isolation, retry cost, and query speed properties of the layout, not the framework.
• Over-subscription kills predictability: We size worker concurrency to the per-shard memory budget and stop short of the cliff; the transition from sub-linear speedup to hard failures is sharp.
• Batching beats more GPUs: Configuration tuning consistently outran additional hardware in our experiments.
• Zero-ETL analytics: Well-structured data makes analytics in-situ, removing redundant ETL and turning overnight batches into interactive queries.
• Observability drives tuning: Instrumentation exposed a hidden 40% idle-GPU window and let us fix it before it became a failure.

Closing thoughts

A scalable pipeline treats CPU preprocessing, GPU enrichment, and analytics as distinct, well-bounded stages that share one data layout, one notion of execution locality, and one observability fabric. Match each stage to the execution model that fits its access pattern, and measure the envelope of every design choice. What matters most when curating data for the next generation of models is a stack that stays predictable as data volumes grow and adaptable as workloads change.