Back to blog

How to Handle Large File Uploads: Multipart Uploads and Presigned URLs

Uploading small files (like user avatars or PDF invoices) is straightforward: you send the file in a standard multipart/form-data request, parse it on your server using a library like Multer, and write it to disk or cloud storage.

However, if users need to upload large files (such as 1GB videos, machine learning models, or compressed archives), this naive approach fails. It blocks your application's single-threaded event loops, consumes massive server memory, and frequently triggers request timeouts.

In this guide, we will analyze strategies for uploading large files, explore multipart chunking, and implement a direct cloud upload system using AWS S3 Presigned URLs.

The Bottlenecks of Traditional Uploads

  • Memory Exhaustion: Parsing large files in memory before saving them can crash your server process if multiple users upload files concurrently.
  • Network Blocking: Exposing your backend server to long-duration upload connections occupies bandwidth, preventing the server from handling standard API requests.
  • Lack of Resilience: If a user’s connection drops at 99 percent of a 1GB upload, they must start the entire upload from scratch.

Strategy 1: Multipart Chunked Uploads

To avoid timeouts and memory bloat, you can split a large file into small, fixed-size chunks (e.g., 5MB per chunk) on the client side using JavaScript's File.slice() API.

The Chunking Workflow

  1. The frontend slices the file into an array of chunks.
  2. The client uploads each chunk sequentially or in parallel to a /upload/chunk endpoint.
  3. The server writes each chunk to a temporary directory.
  4. Once all chunks are uploaded, the client triggers a /upload/complete request, prompting the server to stitch the chunks back into a single file.

This pattern is highly resilient. If a chunk upload fails, the client only needs to retry that single 5MB chunk (resumable upload).

Strategy 2: Direct Uploads using S3 Presigned URLs (Recommended)

The most efficient way to handle large uploads is bypassing your application server entirely. You authorize the client to upload the file directly to object storage (like AWS S3 or Cloudflare R2).

To secure this direct upload without exposing your AWS root credentials:

  1. Request Permission: The client requests permission from your server, sending the file name and content type.
  2. Generate URL: The server uses its secure AWS SDK credentials to generate a Presigned URL—a temporary URL authorized to write a specific file to S3 within a short time window (e.g., 15 minutes).
  3. Direct Upload: The server returns the URL to the client. The client executes a standard HTTP PUT request, streaming the file directly to S3's servers.

Step 1: Server-Side URL Generation

Here is the Node.js implementation to generate a presigned URL using the AWS SDK v3:

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({ region: 'us-east-1' });

export async function generateUploadUrl(fileName: string, fileType: string) {
  const command = new PutObjectCommand({
    Bucket: 'my-large-uploads-bucket',
    Key: `uploads/${Date.now()}-${fileName}`,
    ContentType: fileType,
  });

  // Generate a URL valid for 15 minutes (900 seconds)
  const presignedUrl = await getSignedUrl(s3, command, { expiresIn: 900 });
  return presignedUrl;
}

Step 2: Client-Side Direct Upload

Once the frontend receives the presigned URL from the server, it uploads the file directly to S3 using a standard Axios or Fetch call:

async function uploadFileToS3(file: File) {
  // 1. Get the presigned URL from your backend proxy API
  const response = await fetch('/api/get-presigned-url', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ fileName: file.name, fileType: file.type }),
  });
  
  const { url } = await response.json();

  // 2. Upload the file directly to S3 using the URL
  const uploadResult = await fetch(url, {
    method: 'PUT',
    headers: {
      'Content-Type': file.type,
    },
    body: file, // Streams the file directly
  });

  if (uploadResult.ok) {
    console.log('Upload complete!');
  } else {
    console.error('Upload failed.');
  }
}

Conclusion

Handling large file uploads requires separating file storage streams from API compute resources. While chunked uploads provide excellent connection resilience, using S3 Presigned URLs is the most scalable architecture. By allowing clients to stream files directly to cloud storage, you eliminate server bandwidth usage and protect your backend memory pools from exhaustion.