Skip to content

A high-performance, simple backup tool written in Go - designed as a modern alternative to Duplicity. This is a project I will get to work on someday. Leaving it as is for now.

Notifications You must be signed in to change notification settings

Rubix982/gosync

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 

Repository files navigation

GoSync

A high-performance, simple backup tool written in Go - designed as a modern alternative to Duplicity.

The Problem

Duplicity is reliable but slow (Python-based, single-threaded)
Restic is fast but complex (snapshot-based, steep learning curve)
GoSync aims to be fast AND simple (Go-based, file-path oriented)

Goals

  • 10-50x faster than Duplicity through Go's concurrency and performance
  • Drop-in replacement with familiar CLI interface
  • Simple file-path based restore (no snapshot IDs or repository concepts)
  • Modern cloud integration with progress bars and resumption
  • Single binary deployment with no dependencies

Planned Features

Core Backup

# Simple backup syntax (like Duplicity)
GoSync /home/user s3://mybucket/backup
GoSync /important/data file:///backup/local

# Incremental backups by default
GoSync --full /home/user s3://mybucket/backup

Simple Restore

# Restore everything
GoSync --restore s3://mybucket/backup /home/user

# Restore specific file (intuitive paths)
GoSync --restore-file documents/important.txt s3://mybucket/backup /tmp/

# List backup contents
GoSync --list s3://mybucket/backup

Modern Features

# Progress and resumption
GoSync --progress --resume-on-failure /data s3://bucket

# Flexible exclusions
GoSync --exclude-patterns "*.tmp,.git,node_modules" /project s3://bucket

# Dry run
GoSync --dry-run /home/user s3://bucket

Technical Architecture

Performance Design

  • Concurrent chunk processing using goroutines
  • Parallel uploads/downloads to cloud storage
  • Efficient deduplication with rolling hash algorithms
  • Memory-mapped file I/O for large files

Storage Backends

  • Amazon S3 and S3-compatible (MinIO, DigitalOcean Spaces)
  • Google Cloud Storage
  • Azure Blob Storage
  • Local filesystem and network drives
  • SFTP/SSH remote servers

Security

  • AES-256 encryption with user-provided passwords
  • GPG integration for key management
  • Secure key derivation (PBKDF2/Argon2)
  • Integrity verification with cryptographic checksums

Development Roadmap

Phase 1: MVP (Core Functionality)

  • Basic backup/restore with local storage
  • File-level incremental backups
  • Simple AES encryption
  • Duplicity-compatible CLI interface

Phase 2: Cloud Integration

  • S3 backend implementation
  • Google Cloud Storage support
  • Progress bars and resumption
  • Compression (zstd/gzip)

Phase 3: Advanced Features

  • GPG encryption compatibility
  • Multiple storage backends
  • Backup verification and repair
  • Configuration file support

Phase 4: Enterprise Features

  • Backup scheduling daemon
  • Web UI for management
  • Monitoring and alerting
  • Backup retention policies

Why Go?

  • Performance: 10-50x faster than Python for I/O operations
  • Concurrency: Built-in goroutines for parallel processing
  • Deployment: Single binary, no dependencies
  • Cloud APIs: Excellent SDK support for all major providers
  • Memory efficiency: Better garbage collection than Python
  • Cross-platform: Easy compilation for all platforms

Target Users

  • Existing Duplicity users frustrated with performance
  • System administrators wanting reliable, fast backups
  • Developers needing simple CI/CD backup integration
  • Small businesses requiring efficient data protection
  • Power users who find Restic overly complex

Competitive Advantage

Feature Duplicity Restic GoSync
Performance ❌ Slow ✅ Fast ✅ Fast
Simplicity ✅ Simple ❌ Complex ✅ Simple
File Paths ✅ Direct ❌ Snapshots ✅ Direct
Cloud Support ⚠️ Limited ✅ Good ✅ Excellent
Dependencies ❌ Python ✅ Single Binary ✅ Single Binary

Getting Started (Future)

# Install
curl -sSL https://get.GoSync.dev | sh

# First backup
GoSync --encrypt-password mysecret /home/user s3://mybucket/backup

# Daily incremental
GoSync /home/user s3://mybucket/backup

# Restore when needed
GoSync --restore s3://mybucket/backup /home/user

Implementation Notes

Key Libraries

  • Storage: AWS SDK, Google Cloud SDK, Azure SDK
  • Encryption: Go crypto/aes, golang.org/x/crypto
  • Compression: klauspost/compress (zstd)
  • Hashing: Rolling hash for deduplication
  • CLI: spf13/cobra for command interface

Performance Targets

  • Backup speed: >100MB/s on modern hardware
  • Memory usage: <512MB for large datasets
  • CPU efficiency: Utilize all available cores
  • Network optimization: Concurrent uploads with retry logic

About

A high-performance, simple backup tool written in Go - designed as a modern alternative to Duplicity. This is a project I will get to work on someday. Leaving it as is for now.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published