What We Learned Migrating 600,000 Files from ImageRight to SharePoint

Overview

Moving 600,000 files sounds like a straightforward migration task—until each file must be reliably pulled
from a legacy system, uploaded to SharePoint, tracked, retried, and recovered at scale.

Initially, the project was simple: read one spreadsheet row, download the file, upload it to SharePoint,
mark it complete, and move on. It proved the concept, but not the scale. At that pace, 600,000 files would
take about 22 days of continuous runtime, with limited recovery if the process failed.

With revisions, that script became a self-contained, SQL-queue-driven migration service that moved more than
8,300 files per hour, recovered from vendor outages, SharePoint throttling, and database disconnects, and
reduced the migration window from more than three weeks to under three days.

A Three-Part Approach

Below, each area is described to show how the operational issues that often stall large-scale enterprise
migrations were resolved.

1. Throughput — Parallelism and Batching

Parallelism and batching drove the entire order-of-magnitude gain. By optimizing concurrency and reducing
network idle time, the migration pipeline achieved dramatic performance improvements:

• ✓
  Concurrency Scaling with Parallel.ForEachAsync: Concurrency
  moved to Parallel.ForEachAsync, with MaxDegreeOfParallelism raised from 4
  to 10, gated by a separate semaphore on the source API so each remote system could be tuned
  independently.
• ✓
  Reducing Network Idle Wait Times: Cached OAuth tokens,
  long-lived HttpClient instances, single-PUT uploads under 4 MB, and 5 MiB upload slices
  cut idle wait time rather than the per-file cost itself.
• ✓
  8.3x Production Gain: Effective parallelism rose from 76% to
  84%, taking sustained throughput from 1,125 files/hour sequentially to 8,314 files/hour in
  production—an 8.3x gain.

2. Completion — Retry Logic, Status Tracking, and Reclaiming Abandoned Work

Retry logic, status tracking, and reclaiming abandoned work left those throughput numbers untouched, but
they decided whether a 14-hour run survived to the end:

• ✓
  SQL-Backed Queue State Machine: A SQL-backed queue table gave
  every row a distinct lifecycle state:
  
  Collected
  →
  Processing
  →
  Copied
  •
  Failed
  •
  Skipped
  
  
• ✓
  Granular Retry Tracking: The system maintained a discrete retry
  count instead of relying on a single pass-or-fail outcome.
• ✓
  Smart Error Categorization: Transient errors received
  exponential backoff, while non-retriable errors such as 401, 403, and
  404 failed fast.
• ✓
  Startup Reclaim Pass: A startup reclaim pass automatically
  requeued rows that a killed or crashed process had left stranded in Processing.
• ✓
  Exceptional Reliability: The result was an astonishing
  0.00845% failure rate across 601,064 attempted files.

3. Operability — Packaging, Scheduling, Heartbeats, and Run Caps

Packaging, scheduling, heartbeats, and run caps determined whether the process could run unattended:

• ✓
  Self-Contained Executable: A self-contained executable with a
  scheduled-task installer removed the dependency on an active interactive user session.
• ✓
  Worker Heartbeats: A per-batch heartbeat logging each worker’s
  state made a stalled batch distinguishable from a dead process.
• ✓
  Graceful Run Caps: Run caps allowed a scheduled execution to
  stop cleanly at a batch boundary.
• ✓
  Covering SQL Index: The covering index on the queue table cut
  logical reads per batch fetch by 28x as the backlog grew toward 600,000
  rows.

These changes did not increase files per hour. Instead, they made it possible to sustain that throughput for
14+ unattended hours instead of just one supervised hour.

The Architecture That Emerged

A queue table is what made every other improvement possible.

It allowed progress to live at row granularity, enabled a killed run to reclaim its orphaned work on
restart, and allowed failures to be classified as retriable versus fatal instead of simply being logged.

This transformed the migration from a basic file-processing script into a resilient, recoverable migration
service.

By the Numbers

The failure rate held at 0.00845%, and a targeted SQL index cut per-batch database reads by
28x as the queue grew toward 600,000 rows.

Key Takeaway

Before optimizing any migration or batch pipeline, classify each improvement by its purpose:

• ✓
  Increasing throughput
• ✓
  Enabling long-running jobs to complete despite failures
• ✓
  Improving visibility when the system is unattended

Speed should not come at the expense of resilience or observability. A pipeline that cannot recover from a
single failed connection is not truly performant; it is only reliable until the first failure occurs.