Understanding tus.io: The Open Protocol for Resumable File Uploads

Inside the elegant solution that’s transforming the way we handle large file transfers

9 min readDec 27, 2024

Imagine you’re uploading your vacation video to a cloud storage service using your phone. You’re at 80% when you enter an elevator and lose connection. Traditional uploads would require you to start over, but with tus.io, you can resume from where you left off. This is the magic of resumable file uploads, and tus.io makes it standardized and accessible.

Tus.io represents a significant step forward in handling file uploads on the internet. By providing a standardized protocol, it helps developers implement reliable upload functionality while saving time and resources. Whether you’re building a small application or a large-scale system, understanding and implementing tus.io can significantly improve your file upload capabilities.

What is tus.io?

Tus.io is an open protocol for resumable file uploads built on HTTP. Think of it as a set of rules and guidelines that developers can follow to implement reliable file uploads in their applications. Just like how HTTP defines how web browsers and servers communicate, tus.io defines how applications should handle file uploads in a resumable way.

The protocol is like a universal language that ensures all applications implementing tus.io can understand each other, regardless of the programming language or platform they’re built with.

Why Do We Need a Protocol for Resumable Uploads?

Before tus.io, developers had to create their own solutions for handling interrupted uploads. This led to several problems:

Inconsistency: Different applications handled uploads differently, creating confusion for both users and developers.
Duplication of Effort: Every team had to solve the same problems repeatedly.
Compatibility Issues: Different upload solutions often couldn’t work together, making it difficult to switch between services.
Reliability Concerns: Many custom solutions weren’t thoroughly tested for all edge cases.

How tus.io Works

The tus protocol operates through a series of HTTP requests and specific headers that coordinate the upload process. Here’s a simplified explanation of how it works:

Creation: The client tells the server it wants to upload a file, providing details like file size and type.
Upload URL: The server responds with a unique URL where the file chunks should be sent.
Chunking: The client splits the file into smaller pieces and starts uploading them.
Progress Tracking: The server keeps track of which chunks it has received.
Resume: If the upload is interrupted, the client can ask the server which chunks it has and continue from there.

Think of it like reading a book with a bookmark. If you stop reading, you don’t start over — you continue from where you left off.

Understanding the tus.io Protocol Flow

The tus.io protocol follows a well-defined sequence of steps to ensure reliable file uploads. Let’s break down each phase of the process:

1. Initial Upload Creation

Client Initiates Upload

The client sends a POST request to /files
Includes metadata like file name, size, and type
Example header: Upload-Length: 100000

2. Server Creates Upload Record

Server generates a unique upload ID
Initializes upload tracking in storage
Returns location URL for future requests
Example response: Location: /files/2d1c6561-1c44-4a89-8710-321031224b91

2. Upload Process

Check Upload Status

Client sends HEAD request to get current upload offset
Server responds with Upload-Offset header
Example: Upload-Offset: 0 for new uploads

2. Chunk Upload

Client sends PATCH requests with file chunks
Each request includes: Current offset (Upload-Offset header), Chunk data in request body, Content-Type: application/offset+octet-stream
Server validates offset matches expected position
Server stores chunk and updates offset
Example header:

PATCH /files/2d1c6561-1c44-4a89-8710-321031224b91
Upload-Offset: 0
Content-Type: application/offset+octet-stream

3. Handling Interruptions

Connection Lost

Upload might stop due to: Network issues, Browser tab closing, Device going offline
Current progress is saved on server

2. Resume Process

Client requests current offset (HEAD request)
Server returns last successful offset
Upload continues from that point. Example:

HEAD /files/2d1c6561-1c44-4a89-8710-321031224b91
Response: Upload-Offset: 50000

4. Upload Completion

Final Chunk

Server recognizes final chunk when: Total uploaded bytes = Upload-Length
All chunks are successfully stored

2. Server Processing

Validates complete upload
Combines chunks if necessary
Updates upload status
Triggers any post-upload hooks

3. Client Notification

Final offset equals total file size
Example: Upload-Offset: 100000 (matches Upload-Length)

Key Protocol Features

Offset Tracking

Server maintains exact byte position
Prevents data duplication
Ensures upload integrity

2. Chunk Management

Flexible chunk sizes
No fixed chunk size requirement
Client can adjust based on conditions

3. Status Verification

HEAD requests provide current status
No need to maintain client-side state
Supports multiple clients uploading same file

4. Error Handling

Clear status codes for different scenarios
Automatic retry mechanisms
Consistent error reporting

Protocol Extensions

tus.io supports several optional extensions:

Creation

Enables upload creation
Required for most implementations

2. Expiration

Allows setting upload timeouts
Helps manage server resources

3. Concatenation

Enables combining multiple uploads
Useful for parallel uploads

4. Creation With Upload

Combines creation and first chunk
Improves efficiency for small files

Example HTTP Exchange

# Create Upload
POST /files HTTP/1.1
Upload-Length: 100000
Upload-Metadata: filename d29ya2Zsb3cuanBn

HTTP/1.1 201 Created
Location: /files/2d1c6561-1c44-4a89-8710-321031224b91
Tus-Resumable: 1.0.0

# Upload Chunk
PATCH /files/2d1c6561-1c44-4a89-8710-321031224b91 HTTP/1.1
Upload-Offset: 0
Content-Type: application/offset+octet-stream
Content-Length: 30000

[BINARY DATA]

HTTP/1.1 204 No Content
Upload-Offset: 30000
Tus-Resumable: 1.0.0

Real-World Applications

Tus.io is used in various scenarios:

Cloud Storage Services: Providers like Google Drive and Dropbox use similar principles for reliable uploads.
Media Platforms: Video sharing sites need reliable upload mechanisms for large files.
Enterprise Systems: Companies handling large documents benefit from resumable uploads.
Mobile Applications: Apps dealing with user-generated content need reliable upload mechanisms.

Implementing Resumable Upload Server with tusd in Go

File Structure

├── cmd
│   └── server
│       └── main.go
├── internal
│   ├── config
│   │   └── config.go
│   ├── handlers
│   │   └── upload.go
│   ├── hooks
│   │   └── hooks.go
│   └── storage
│       └── store.go
├── web
│   └── index.html
└── go.mod

Code implementation

package main

// cmd/server/main.go
import (
    "log"
    "net/http"
    
    "myapp/internal/config"
    "myapp/internal/handlers"
)

func main() {
    // Load configuration
    cfg := config.New()

    // Initialize upload handler
    uploadHandler, err := handlers.NewUploadHandler(cfg)
    if err != nil {
        log.Fatalf("Failed to create upload handler: %v", err)
    }

    // Set up routes
    mux := http.NewServeMux()
    mux.Handle("/files/", http.StripPrefix("/files/", uploadHandler))
    mux.Handle("/", http.FileServer(http.Dir("web")))

    // Start server
    log.Printf("Server starting on %s", cfg.ServerAddress)
    if err := http.ListenAndServe(cfg.ServerAddress, mux); err != nil {
        log.Fatalf("Server failed: %v", err)
    }
}

package config

// internal/config/config.go
type Config struct {
    ServerAddress string
    UploadDir     string
    MaxSize       int64
    AllowedTypes  []string
}

func New() *Config {
    return &Config{
        ServerAddress: ":8080",
        UploadDir:    "./uploads",
        MaxSize:      1024 * 1024 * 1024, // 1GB
        AllowedTypes: []string{"image/jpeg", "image/png", "application/pdf"},
    }
}

package handlers

// internal/handlers/upload.go

import (
    "myapp/internal/config"
    "myapp/internal/hooks"
    "myapp/internal/storage"
    
    tusd "github.com/tus/tusd"
)

type UploadHandler struct {
    *tusd.Handler
}

func NewUploadHandler(cfg *config.Config) (*UploadHandler, error) {
    // Initialize store
    store := storage.NewStore(cfg)
    
    // Create composer
    composer := tusd.NewStoreComposer()
    store.UseIn(composer)

    // Configure upload handler
    handler, err := tusd.NewHandler(tusd.Config{
        BasePath:              "/files/",
        StoreComposer:        composer,
        NotifyCompleteUploads: true,
        MaxSize:              cfg.MaxSize,
    })
    if err != nil {
        return nil, err
    }

    // Set up hooks
    hooks.SetupHooks(handler, cfg)

    return &UploadHandler{Handler: handler}, nil
}

package hooks

// internal/hooks/hooks.go

import (
    "log"
    "myapp/internal/config"
    
    tusd "github.com/tus/tusd"
)

func SetupHooks(handler *tusd.Handler, cfg *config.Config) {
    // Pre-create hook
    handler.BeforeCreate = func(hook tusd.HookEvent) error {
        fileType := hook.Upload.FileType
        
        // Validate file type
        allowed := false
        for _, allowedType := range cfg.AllowedTypes {
            if fileType == allowedType {
                allowed = true
                break
            }
        }
        
        if !allowed {
            return tusd.ErrUnsupportedMediaType
        }
        
        return nil
    }

    // Post-finish hook
    go func() {
        for {
            event := <-handler.CompleteUploads
            log.Printf("Upload %s completed", event.Upload.ID)
            
            // Here you can add post-processing logic
            // For example: move file, trigger notifications, etc.
        }
    }()

    // Post-terminate hook
    handler.BeforeDelete = func(hook tusd.HookEvent) error {
        log.Printf("Upload %s deleted", hook.Upload.ID)
        return nil
    }
}

package storage

// internal/storage/store.go

import (
    "myapp/internal/config"
    
    tusd "github.com/tus/tusd"
)

type Store struct {
    tusd.DataStore
}

func NewStore(cfg *config.Config) *Store {
    store := tusd.NewStoreComposer()
    
    // Use local storage
    store.UseLocalStore = true
    
    return &Store{
        DataStore: tusd.NewFileStore(cfg.UploadDir),
    }
}

func (s *Store) UseIn(composer *tusd.StoreComposer) {
    composer.UseCore(s)
    composer.UseTerminater(s)
    composer.UseLocker(s)
    composer.UseFinisher(s)
}

<!-- web/index.html -->
<!DOCTYPE html>
<html>
<head>
    <title>File Upload Example</title>
    <script src="https://cdn.jsdelivr.net/npm/tus-js-client@latest/dist/tus.min.js"></script>
</head>
<body>
    <input type="file" id="fileInput">
    <button onclick="uploadFile()">Upload</button>
    <div id="progress"></div>

    <script>
        function uploadFile() {
            const input = document.getElementById('fileInput');
            const file = input.files[0];
            const progressElement = document.getElementById('progress');

            if (!file) {
                alert('Please select a file');
                return;
            }

            const upload = new tus.Upload(file, {
                endpoint: '/files/',
                retryDelays: [0, 3000, 5000, 10000, 20000],
                metadata: {
                    filename: file.name,
                    filetype: file.type
                },
                onError: function(error) {
                    console.log('Failed:', error);
                },
                onProgress: function(bytesUploaded, bytesTotal) {
                    const percentage = ((bytesUploaded / bytesTotal) * 100).toFixed(2);
                    progressElement.textContent = percentage + '%';
                },
                onSuccess: function() {
                    progressElement.textContent = 'Upload completed!';
                }
            });

            upload.start();
        }
    </script>
</body>
</html>

Code Explanation

Let’s break down each component and understand how they work together:

1. Main Server (cmd/server/main.go)

Entry point of the application
Sets up HTTP routes
Initializes the upload handler
Serves static files from the web directory
Starts the HTTP server

Key components:

mux.Handle("/files/", http.StripPrefix("/files/", uploadHandler))

This line routes all requests starting with “/files/” to our upload handler. The StripPrefix removes "/files/" from the URL before passing it to the handler.

2. Configuration (internal/config/config.go)

Defines application settings
Server address
Upload directory
Maximum file size
Allowed file types

The configuration is centralized and easily modifiable:

MaxSize: 1024 * 1024 * 1024, // 1GB

Sets the maximum file size to 1GB.

3. Upload Handler (internal/handlers/upload.go)

Creates and configures the tusd handler
Integrates storage and hooks
Handles upload requests

Important configuration:

tusd.Config{
    BasePath:              "/files/",
    StoreComposer:        composer,
    NotifyCompleteUploads: true,
    MaxSize:              cfg.MaxSize,
}

This configures:

Base URL path
Storage composition
Upload completion notifications
Maximum file size

4. Hooks (internal/hooks/hooks.go)

Implements upload lifecycle hooks
Validates file types
Handles upload completion
Manages upload deletion

Key hooks:

Validates file type before upload starts.

handler.BeforeCreate = func(hook tusd.HookEvent) error {
    // File type validation
}

Processes completed uploads in a separate goroutine.

for {
    event := <-handler.CompleteUploads
    // Handle completed uploads
}

5. Storage (internal/storage/store.go)

Implements tusd storage interface
Configures local file storage
Sets up store composition

Storage setup:

store := tusd.NewStoreComposer()
store.UseLocalStore = true

6. Frontend (web/index.html)

Provides user interface for uploads
Implements tus-js-client
Shows upload progress
Handles errors

const upload = new tus.Upload(file, {
    endpoint: '/files/',
    retryDelays: [0, 3000, 5000, 10000, 20000],
    // ...
});

Key client features:

Configures the tus client
Sets up retry delays
Handles upload events

Using the Application

Start the server:

go run cmd/server/main.go

2. Open a browser and navigate to http://localhost:8080

3. Select a file and click Upload

4. The upload will:

Start automatically
Show progress
Handle interruptions
Resume automatically if needed

Conclusion

The tus.io protocol represents a significant advancement in handling file uploads on the modern web, offering a elegant solution to the age-old problem of interrupted file transfers. By providing a standardized, HTTP-based approach with features like automatic resumption, chunk-based uploads, and cross-platform compatibility, tus.io makes it possible to build reliable upload systems without reinventing the wheel.

Whether you’re dealing with large video files, important documents, or any other type of data transfer, tus.io’s robust protocol and growing ecosystem of client/server implementations make it an invaluable tool in any developer’s arsenal. As file sizes continue to grow and reliable file transfer becomes increasingly critical in our interconnected world, the principles and practices established by tus.io will only become more relevant and essential.

If you’re interested in diving deeper into system design and backend development, be sure to follow me for more insights, tips, and practical examples. Together, we can explore the intricacies of creating efficient systems, optimizing database performance, and mastering the tools that drive modern applications. Join me on this journey to enhance your skills and stay updated on the latest trends in the tech world! 🚀

Read the design system in bahasa on iniakunhuda.com