How NVIDIA NeMo Curator Speeds Up LLM Training: Benchmarks and Results

Why Data Processing Speed Matters for LLM Training

The quality of an LLM's training data directly determines its performance. But data curation at internet scale — cleaning, deduplicating, and filtering billions of documents — is computationally expensive. CPU-based pipelines can take days or weeks to process the datasets required for modern LLM pre-training.

NVIDIA NeMo Curator is an open-source toolkit that uses GPU acceleration to dramatically speed up this process. By leveraging RAPIDS libraries (cuDF, cuML, cuGraph) for GPU-accelerated data processing, NeMo Curator transforms data curation from a bottleneck into a fast, iterative workflow.

Core Capabilities

NeMo Curator handles three critical data curation tasks:

flowchart LR
    CORPUS[("Pre-training corpus<br/>trillions of tokens")]
    FILTER["Quality filter and<br/>dedupe"]
    TOK["BPE tokenizer"]
    SHARD["Shard plus<br/>data parallel"]
    GPU{"GPU cluster<br/>FSDP or DeepSpeed"}
    CKPT[("Checkpoints<br/>every N steps")]
    LOSS["Loss curve plus<br/>eval gates"]
    SFT["SFT phase"]
    DPO["DPO or RLHF"]
    BASE([Base model])
    INSTR([Instruct model])
    CORPUS --> FILTER --> TOK --> SHARD --> GPU
    GPU --> CKPT --> LOSS
    LOSS --> BASE --> SFT --> DPO --> INSTR
    style GPU fill:#4f46e5,stroke:#4338ca,color:#fff
    style LOSS fill:#f59e0b,stroke:#d97706,color:#1f2937
    style INSTR fill:#059669,stroke:#047857,color:#fff

1. Cleaning: Removing noise, corrupted text, encoding errors, and non-linguistic content from raw datasets
2. Deduplicating: Identifying and removing exact copies, near-duplicates, and semantically redundant documents at scale
3. Filtering: Applying quality classifiers, safety filters, and domain-relevance scoring to keep only high-signal training data

The toolkit supports text, image, and multimodal data — covering the full range of modern LLM training modalities.

Additionally, NeMo Curator provides PII (Personally Identifiable Information) redaction capabilities, ensuring that sensitive information is removed from training data before it reaches the model.

Performance Benchmarks

17x Faster Fuzzy Deduplication

On the RedPajama-v2 dataset (a large-scale web-crawled corpus), NeMo Curator's GPU-accelerated fuzzy deduplication completed in 0.65 hours — compared to 11 hours using equivalent CPU-based methods.

This represents a 17x speedup, turning an overnight batch job into a process that completes in under an hour.

Near-Linear GPU Scaling

NeMo Curator demonstrates near-linear scaling across multiple H100 80GB GPU nodes:

GPU Nodes	Processing Time	Speedup
1 node	2.05 hours	1x
2 nodes	0.94 hours	2.2x
4 nodes	0.50 hours	4.1x

Processing time roughly halves with each doubling of GPU nodes. This near-linear scaling means that teams can process terabyte-scale datasets efficiently by adding hardware — without diminishing returns.

Measurable Model Accuracy Gains

The most compelling result is the downstream impact on model quality. A 357M parameter GPT base model trained on NeMo Curator-processed data showed a 3.5-point improvement (approximately 7% relative gain) on reasoning benchmarks compared to the same model trained on raw, unprocessed data.

Benchmark	Raw Data	Curated Data	Improvement
RACE	Lower	Higher	+7% relative
PiQA	Lower	Higher	+7% relative
Winogrande	Lower	Higher	+7% relative
HellaSwag	Lower	Higher	+7% relative
Average	47.5	51.0	+3.5 points

This demonstrates that data curation is not just about efficiency — it directly produces better models.

Why This Matters

NeMo Curator's performance characteristics enable a fundamentally different approach to data curation:

• Iterative experimentation: When processing takes minutes instead of hours, teams can test multiple filtering and deduplication configurations and compare downstream results
• Faster training cycles: Reducing data preparation from weeks to hours accelerates the overall model development timeline
• Cost efficiency: GPU-accelerated processing produces higher-quality data in less time, reducing both compute costs and human oversight time
• Scale independence: Near-linear GPU scaling means the same pipeline handles gigabyte and terabyte datasets with predictable performance

The toolkit transforms raw, noisy web data into clean, deduplicated, high-quality datasets — and does so fast enough to make data curation an iterative, experimental practice rather than a one-shot batch process.

Frequently Asked Questions

What is NeMo Curator?

NeMo Curator is NVIDIA's open-source toolkit for preparing large-scale datasets for LLM training. It provides GPU-accelerated tools for text cleaning, deduplication (exact, fuzzy, and semantic), quality filtering, PII redaction, and safety filtering. It uses NVIDIA RAPIDS libraries for GPU-accelerated processing and supports distributed computing across multiple GPU nodes.

What GPUs does NeMo Curator require?

NeMo Curator works with any NVIDIA GPU that supports CUDA. For optimal performance on large datasets, H100 or A100 GPUs with 40-80GB VRAM are recommended. The framework scales near-linearly across multiple GPU nodes, so adding more GPUs proportionally reduces processing time.

How does NeMo Curator compare to CPU-based data processing?

NeMo Curator achieves 10-20x speedups compared to equivalent CPU-based pipelines. On the RedPajama-v2 dataset, fuzzy deduplication completed 17x faster using GPU acceleration. Quality filtering shows approximately 20x speedup. These improvements transform multi-day batch jobs into sub-hour processes.

Does curated data actually produce better models?

Yes. Benchmark testing shows a 3.5-point improvement (7% relative gain) on reasoning benchmarks when a GPT model is trained on NeMo Curator-processed data versus raw unprocessed data. Research consistently confirms that data quality has a larger impact on model performance than model size increases.

Can NeMo Curator process multimodal data?

Yes. NeMo Curator supports text, image, and multimodal data processing. This makes it suitable for preparing training datasets for text-only LLMs, vision-language models, and multimodal AI systems.