The operations director sent us a screenshot of a CSV import that had been crashing their system for three weeks. Brand manifests with 2,000 to 10,000 SKUs: style descriptions, UPC codes, pricing, sizing, color variants. Every time they uploaded, the system either crashed on the first invalid row or timed out entirely.
Their existing system loaded the entire CSV into memory, validated every row in sequence, and tried to write everything in a single pass. On a good day, it took 20 minutes for 3,000 rows. On a bad day, it crashed and they started over.
Their operations team was spending 4+ hours before each event just getting inventory into the system. At three retail locations, with events running year-round, that's real money: $15,000 to $30,000 per year in manual reconciliation alone, plus emergency developer calls at $300 to $500 each when things broke before a launch.
We rebuilt the entire processing pipeline in 14 weeks using Laravel and queue-driven architecture. Processing time dropped to under 3 minutes for 10,000 rows. Sync accuracy went from roughly 80% (with silent data drops) to 99.7% (with flagged exceptions). Zero import failures on launch day across all three locations.
The Problem: Why Most CSV Processing Breaks at Scale
CSV processing seems simple until it isn't. The pattern most developers reach for looks like this:
// The naive approach (don't do this)
$rows = array_map('str_getcsv', file($path));
foreach ($rows as $row) {
$this->validate($row);
Item::create($row);
}
This works for 100 rows. At 2,000+ rows, three things break:
- Memory: Loading the entire file into an array, then creating model instances for every row, scales linearly with file size. A 10,000-row manifest with 15+ columns per row will exhaust memory on standard server configurations.
- Blocking execution: Synchronous validation means one bad row stalls everything. Row 1,847 has an invalid UPC? The first 1,846 valid rows wait. The user stares at a spinner with no feedback.
- No error visibility: When a row fails, the entire import either crashes or silently drops the bad data. Nobody knows what was lost until inventory doesn't match on event day.
Our client had all three problems. Plus a fourth: when rows failed validation, they were silently dropped. Nobody knew 20% of their SKU data wasn't making it into the system until inventory counts didn't match at the register.
The Architecture: Chunk, Queue, Report
We rebuilt the pipeline around three principles: chunk instead of load, queue instead of block, report instead of drop.
Chunked Processing with Laravel Horizon
Instead of reading the entire file into memory, we split it into 1,000-row chunks. Each chunk is dispatched as a queued job via Laravel Horizon (backed by Redis). The jobs run in parallel workers while the UI stays responsive.
// Simplified flow from our ProcessCsvImportJob
$chunks = array_chunk($rows, 1000);
foreach ($chunks as $chunk) {
DB::transaction(function () use ($chunk) {
foreach ($chunk as $row) {
$validated = $this->handler->validate($row);
if ($validated) {
$this->handler->upsertItem($validated);
} else {
$this->captureError($row);
}
}
});
}
Each chunk runs inside its own database transaction. If a chunk fails, only those 1,000 rows are affected. The rest continue processing. Memory usage stays constant regardless of file size.
Background Queue with Progress Tracking
When someone uploads a CSV, the API immediately returns. Processing happens in Horizon workers. The frontend shows real-time progress: how many rows processed, how many succeeded, how many failed, and estimated time remaining.
The UX difference matters
Before: Upload, stare at spinner for 20 minutes, hope it worked. After: Upload, see "Processing 10,000 rows..." with a progress bar and error count updating in real time. Same data, completely different experience. As the operations director put it: "It's okay if something takes a long time. Just tell me that it's doing something."
We used Laravel Horizon with Redis for the job queue. Each import gets its own tracked session with status (pending, running, completed, failed), progress counters, and estimated time remaining. Users can see the full history of every import: who uploaded, when, how many records created, updated, or failed.
Row-Level Error Capture
This is the critical piece most CSV processors get wrong. When a row has a problem, you have two choices: crash the entire import, or silently drop the bad row. Both are wrong.
We built a third option: capture the error with full context and keep going. Each failed row gets logged with the row number, the specific field that failed, the validation error, and the original value. After the import completes, the operations team gets a downloadable error report showing exactly what failed and why.
The import log tracks everything: file name, batch ID, created count, updated count, error count, duration. Operations staff can audit any past import and download the error details as a CSV.
The Sync Accuracy Problem Nobody Talks About
Speed was the obvious problem. But the harder problem was accuracy.
The original system reported "import successful" even when 20% of rows failed validation. A SKU with a missing UPC? Silently dropped. A duplicate item? Last one wins, no warning. A category that didn't exist in the system? Row discarded.
This meant operations staff would import a brand manifest, assume everything was fine, then discover on event day that hundreds of items weren't in the system. The point-of-sale showed inventory that the warehouse system didn't know about.
Multi-Tier Validation
We built validation in three layers:
Tier 1: Header Validation (pre-processing)
Before processing a single row, validate the file structure. Are the required columns present (UPC, SKU, description, brand, category)? Does the column mapping match a known template? This catches the "someone uploaded the wrong spreadsheet" error immediately, no wasted processing.
Tier 2: Row-Level Validation (during processing)
Each row passes through field-specific validators: UPC format and uniqueness, required fields check, numeric validation for pricing, case-insensitive location code matching. Invalid rows don't stop the batch. They get captured with their original data and the specific error, then processing continues with the next row.
Tier 3: Post-Import Reconciliation
After processing completes, the import log shows: source file row count vs. created count vs. updated count vs. error count. The numbers must add up. If they don't, something went wrong at the infrastructure level and the entire batch gets flagged.
99.7% sync accuracy means the 0.3% is visible
The difference between 80% accuracy and 99.7% isn't just the numbers. At 80%, errors are invisible. You don't know what you've lost. At 99.7%, every exception is flagged, logged, and waiting for human review. The remaining 0.3% are genuine edge cases (ambiguous data that needs a human decision), not silent failures.
Duplicate Detection: UPC-Based Matching
Brand manifests come in constantly. Same items, updated pricing. New items mixed with existing ones. The system needs to decide: is this a new SKU or an update to an existing one?
We use UPC (Universal Product Code) as the primary match key. If a row's UPC matches an existing item in the database, it's an update. If not, it's a new item. The importer gives the user a choice: skip duplicates or update them with the new data.
For categories and brands that don't exist yet, the system can auto-create them or skip those rows, depending on the import configuration. This means a single CSV upload can create new brands, new categories, and new items all in one pass without manual setup first.
The system also detects duplicate UPCs within the CSV file itself. If the same UPC appears twice in one upload, it flags it rather than letting the second row silently overwrite the first.
Two-Way POS Sync: The Coordination Problem
The inventory platform is the source of truth for product data. But the point-of-sale system (Square) handles all transactions. These two systems need to stay in sync bidirectionally.
Items created or updated in the inventory system push to Square: catalog data, pricing, categories. Sales transactions from Square pull back for reconciliation and reporting. The sync runs on a schedule with manual override when needed.
The critical design decision: selective sync. With 500,000+ SKUs in the master catalog, you can't push everything to every location. Square's performance degrades at scale. So we built event-scoped syncing: only the 10,000 to 20,000 SKUs relevant to a specific event at a specific location get pushed. This reduced the sync payload by 90% and kept Square responsive.
Items that sell at the register but don't exist in the inventory system (the "Purgatory" problem) get routed to a reconciliation queue. Operations staff can review, match, or dismiss these items rather than having them disappear into a gap between the two systems.
Sync sessions track progress in real time: items processed, items synced, items failed, estimated time remaining. Users can pause, resume, or cancel a sync mid-process.
Setup Time: From 4 Hours to 30 Minutes
Beyond the processing speed, we tackled the setup workflow. Before the rebuild, operations staff had to:
- Export the brand manifest from the vendor
- Manually reformat columns to match the import template
- Split the file by location
- Upload each file individually, waiting 20+ minutes per upload
- Cross-reference the results with the source spreadsheet to find missing items
We built a visual field mapping interface. The importer reads the CSV headers and lets you map source columns to database fields: "Style Description" maps to item name, "UPC Code" maps to UPC, "Retail" maps to price. The system remembers mappings, so the second import from the same vendor format auto-applies the same mapping.
Combined with bulk import support (one upload handles new items, updates, and new categories in a single pass), the full event setup dropped from 4+ hours to under 30 minutes.
| Metric | Before | After | Improvement |
|---|---|---|---|
| Processing time (10,000 rows) | 20+ minutes | <3 minutes | 85% faster |
| Sync accuracy | ~80% (silent drops) | 99.7% (flagged exceptions) | 99.7% accuracy |
| Event setup time | 4+ hours | Under 30 minutes | 87.5% reduction |
| Import failures on launch day | Multiple crashes | Zero failures | 100% reliability |
| Emergency developer calls | Regular | Eliminated | $0 emergency cost |
Technical Stack
The platform is a domain-driven Laravel 12 application. Here's why each piece matters for CSV processing specifically:
- Laravel Horizon + Redis for the job queue. Each CSV import dispatches chunked jobs that process in background workers. Horizon gives us real-time monitoring, job metrics, and automatic retry on failure.
- MySQL 8 with transaction batching. Each 1,000-row chunk writes inside a single transaction. If a chunk fails, only that chunk rolls back.
- Vue.js + Inertia for the frontend. Real-time import progress without WebSocket complexity. Inertia's polling handles the progress updates cleanly.
- Domain-Driven Design for the codebase structure. The import pipeline lives in
app/Domain/Inventory/Imports/with its own jobs, handlers, and validators. Completely decoupled from the sync engine and the rest of the application. - Docker (Laravel Sail) for consistent environments across development and production. Deployed to DigitalOcean App Platform with CI/CD via GitHub Actions.
We've also written about Laravel queue architecture for CSV processing if you want the deeper technical walkthrough.
The Human Result
The numbers tell one story. The operations director told a different one:
"For the first time in three years, our warehouse team didn't panic before opening day. Uploads worked. Numbers matched. The POS was in sync."
Zero import failures across 3 warehouse locations on launch day. All three retail stores running with real-time inventory sync from day one. The team went from dreading event launches to treating them as routine.
Frequently Asked Questions
How do you optimize CSV processing for large files?
Use chunked processing instead of loading entire files into memory. We process rows in 1,000-row batches through Laravel Horizon queue workers. Each chunk runs inside a database transaction with row-level validation and error capture. These changes took our client from 20 minutes to under 3 minutes for 10,000-row imports.
What's a good sync accuracy rate for CSV data imports?
Production-grade CSV sync should achieve 99.5%+ accuracy. Our platform achieves 99.7% by using row-level validation before write, UPC-based duplicate detection, and chunked transaction batching. The remaining 0.3% are flagged for manual review rather than silently dropped.
How much does poor CSV processing cost a business?
For our client running 3 retail locations with 500,000+ SKUs, manual CSV processing consumed 4+ hours of operations staff time before each event. At loaded labor costs, that's $15,000 to $30,000 per year in reconciliation alone. After optimization, event setup dropped from 4+ hours to under 30 minutes.
Should I use chunked processing or load CSV files into memory?
Always use chunked processing for files over 1,000 rows. Loading entire CSV files into memory causes out-of-memory crashes on large datasets and creates unpredictable response times. Processing in batches of 500 to 1,000 rows keeps memory usage constant regardless of file size, and a failed batch only affects that chunk rather than the entire import.
How do you handle CSV validation errors without losing data?
Multi-tier validation: header validation catches structural errors before processing starts, row-level validation flags individual bad records without stopping the batch, and post-import reconciliation verifies that source count equals created + updated + error counts. Invalid rows go to an import log with the original data preserved for manual correction.
Next Steps
If your team is spending hours on data imports that should take minutes, the problem is almost certainly architectural, not the data itself.
- Book a 30-minute architecture call and we'll review your current processing pipeline and identify the bottlenecks
- Laravel Queue Architecture for CSV Processing for the deeper technical walkthrough
- Zero-Downtime Deployment for Multi-Location SaaS covers how we launched across 3 locations simultaneously
- SaaS Development Services for our process and pricing for operations platforms