BSync

A high-performance, modular synchronization library for Elixir inspired by rsync.

BSync is an efficient file synchronization library featuring delta compression, parallel processing, and a modern functional lens-based architecture. It supports multiple data sources including filesystems, memory storage, and network protocols.

🚀 Key Features

• Delta Compression: Efficient binary diff/patch using rolling checksums and block-based comparison (inspired by rsync algorithm)
• Parallel Processing: Concurrent synchronization with configurable worker pools
• Multiple Data Sources: FileSystem, Memory, TCP, and extensible DataSource interface
• Lens Architecture: Composable functional pipeline for flexible data transformations
• Rate Limiting: Built-in bandwidth throttling and progress reporting
• Dry Run Support: Preview changes before applying them
• CLI Tools: Complete command-line utilities including BCP (BSync Copy) for testing and file operations

📦 Installation

Add bsync to your list of dependencies in mix.exs:

def deps do
  [
    {:bsync, "~> 0.1.0"}
  ]
end

🔧 Quick Start

Basic File Synchronization

config = %{
  source_module: BSync.DataSource.FileSystem,
  source_uri: "file:///home/user/photos",
  dest_module: BSync.DataSource.FileSystem,  
  dest_uri: "file:///backup/photos",
  mode: :execute,
  pool_size: 4
}

{:ok, result} = BSyncLens.sync(config)
IO.puts "Synced #{result.session.progress.completed} files"

Custom Pipeline with Filtering

# Create a custom sync pipeline
pipeline = [
  {BSync.Lenses.Discovery, %{}},
  {BSync.Lenses.Comparison, %{
    strategy: :checksum,  # Use checksums instead of mtime
    delete: false         # Don't delete files not in source
  }},
  BSyncLens.filter(fn resource ->
    # Only sync .txt files
    String.ends_with?(resource.identifier, ".txt")
  end),
  BSyncLens.progress_reporter(fn session ->
    completed = length(session.tasks) - length(get_in(session, [:metadata, :pending_tasks], []))
    total = length(session.tasks)
    IO.puts "Processing: #{completed}/#{total}"
  end),
  {BSync.Lenses.Worker, %{pool_size: 6, rate_limit_mb_sec: 5.0}}
]

config = %{
  source_module: BSync.DataSource.FileSystem,
  source_uri: "file:///documents",
  dest_module: BSync.DataSource.FileSystem,
  dest_uri: "file:///backup/documents"
}

{:ok, result} = BSyncLens.sync_with_pipeline(config, pipeline)

Memory-Based Synchronization (Testing)

# Useful for testing or temporary data
BSync.DataSource.Memory.setup_test_data("memory://source", [
  %{identifier: "file1.txt", metadata: %{size: 100, mtime: DateTime.utc_now()}}
], %{"file1.txt" => "Hello, world!"})

{:ok, result} = BSyncLens.sync(%{
  source_module: BSync.DataSource.Memory,
  source_uri: "memory://source",
  dest_module: BSync.DataSource.Memory,
  dest_uri: "memory://dest",
  mode: :execute
})

🏗️ Architecture Overview

BSync uses a modern functional architecture with composable components:

Core Components

• BSyncLens - Primary API with lens-based pipeline composition
• BSync - Core library documentation and version information
• BSync.Pipeline - Lens execution engine and session management
• BSync.LensSession - Immutable session state and metadata tracking

Data Processing Lenses

• BSync.Lenses.Discovery - Enumerates resources from source and destination
• BSync.Lenses.Comparison - Compares resources to determine required operations
• BSync.Lenses.Worker - Executes file transfers with delta compression

Data Sources and Transport

• BSync.DataSource.FileSystem - Local and network filesystem access
• BSync.DataSource.Memory - In-memory storage for testing and caching
• BSync.DataSource.TCP - Network-based resource access (extensible)

Algorithms and Utilities

• BSync.Delta - Rolling checksum and binary diff implementation
• BSync.Filter - Resource filtering based on patterns and predicates
• BSync.RateLimiter - Bandwidth throttling and traffic shaping
• BSync.Roller - Rolling checksum implementation for delta compression

📋 Command Line Tools

BCP (BSync Copy) - File Copy Utility

BCP is a comprehensive file copy utility that demonstrates BSync's capabilities:

# Build the BCP executable
mix escript.build

# Copy files with delta compression
./bcp file1.txt file2.txt /backup/

# Copy directories with progress
./bcp --verbose ~/Documents/ /backup/docs/

# Preview operations (dry run)
./bcp --dry-run --verbose ~/Photos/ /backup/photos/

# Concurrent operations
./bcp --pool-size 8 ~/large_dataset/ /backup/

BCP Features

• Full BSync Integration: Uses BSyncLens.sync for all operations
• Delta Compression: Automatically applies delta sync for file updates
• Multiple Source Support: Copy files, directories, or mixed sources
• Structure Preservation: Maintains directory hierarchies
• Progress Reporting: Verbose and non-verbose modes with compression statistics
• Error Handling: Graceful error reporting and recovery

🔍 Understanding Lenses

Lenses are the core building blocks of BSync - composable units that transform session state:

Creating Custom Lenses

# Simple logging lens
logging_lens = BSyncLens.custom_lens(fn session ->
  resource_count = map_size(session.resources)
  task_count = length(session.tasks)
  IO.puts "Session: #{resource_count} resources, #{task_count} tasks"
  session
end)

# File size filter
size_filter = BSyncLens.filter_resources(fn _identifier, resource ->
  case resource do
    %{source: %{size: size}} when size > 1024 -> true
    _ -> false
  end
end)

# Progress reporter
progress_lens = BSyncLens.progress_reporter(fn session ->
  completed = get_in(session, [:progress, :completed], 0)
  total = get_in(session, [:progress, :total], 0)
  IO.puts "Progress: #{completed}/#{total}"
end)

Building Custom Pipelines

custom_pipeline = [
  {BSync.Lenses.Discovery, %{}},
  
  # Log discovery results
  BSyncLens.custom_lens(fn session ->
    IO.puts "Discovered #{map_size(session.resources)} resources"
    session
  end),
  
  # Filter by file type
  BSyncLens.filter(fn resource ->
    Path.extname(resource.identifier) in [".jpg", ".png", ".gif"]
  end),
  
  # Checksum comparison for accuracy
  {BSync.Lenses.Comparison, %{strategy: :checksum, delete: false}},
  
  # Execute with progress reporting
  BSyncLens.progress_reporter(fn session ->
    IO.puts "Processing #{length(session.tasks)} image files"
  end),
  
  {BSync.Lenses.Worker, %{pool_size: 4, rate_limit_mb_sec: 5.0}}
]

{:ok, result} = BSyncLens.sync_with_pipeline(config, custom_pipeline)

⚡ Performance Features

• Block-based Delta Compression: Only transfers changed data blocks using rolling checksums
• Parallel Processing: Configurable worker pools for concurrent transfers
• Memory Efficient: Streaming data processing with minimal memory footprint
• Progress Reporting: Real-time feedback on transfer progress and compression ratios
• Rate Limiting: Built-in bandwidth controls and traffic shaping

🔧 Configuration Options

The lens-based API supports comprehensive configuration:

• :mode - :execute (default) or :dry_run
• :pool_size - Number of concurrent workers (default: 4)
• :rate_limit_mb_sec - Bandwidth limit in MB/s (default: no limit)
• :progress_callback - Function called with progress updates
• :delete - Whether to delete destination files not in source (default: false)
• :comparison_strategy - :mtime (default) or :checksum
• :transport_opts - Options passed to transport layer
• :delta_opts - Options for delta compression algorithm

📚 Use Cases

• Backup Systems: Efficient incremental backups with delta compression
• Content Distribution: Distribute large datasets with minimal bandwidth usage
• Development Tools: Sync code and assets between environments
• Data Migration: Move data between different storage systems
• File Replication: Keep multiple locations in sync
• Testing Utilities: Mock data sources for integration testing

🧪 Testing and Development

Running Tests

# Run all tests
mix test

# Run with coverage
mix test --cover

# Run specific test files
mix test test/bsync/delta_test.exs
mix test test/integration/

# Run tests with verbose output
mix test --trace

Demo and Development Tools

# Run interactive demo
mix bsync.demo

# Dry run demo (preview operations)
mix bsync.demo.dry_run

# Live demo with real operations
mix bsync.demo.live_run

Building Documentation

# Generate documentation
mix docs

# View documentation
open doc/index.html

📖 Learning Resources

• Tutorial - Comprehensive tutorial with practical examples
• Architecture Diagram - Visual overview of module relationships
• Module Specification - Detailed technical specifications

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Write tests for your changes
4. Ensure all tests pass (mix test)
5. Commit your changes (git commit -am 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Inspired by the rsync algorithm and its efficient delta compression approach
• Built with modern Elixir/OTP patterns for fault tolerance and concurrency
• Uses functional lens architecture for composable data transformations

🔗 Related Projects

• rsync - The original inspiration for delta compression algorithms
• Elixir - The language and runtime that makes BSync possible
• OTP - Fault-tolerant patterns used throughout the architecture